Beats: How to create a custom Elastic Beat

Beats Elastic Stack as an important part of the family. It can be easily and let us send our data into Elasticsearch or Logstash. If we want to generate your own Beat, Beat generator use GitHub repository of beats provided. In today's article, we will describe in detail how step by step to create a beat ourselves want.

Set up your development environment

Installation go environment

Beats actually go program. We can install their own development environment golang language reference link "Go get started" (https://golang.org/doc/install). Once we installed our go, we can enter the following command in the terminal:

    $ which go
    /usr/local/go/bin/go

So we need to set the following variables in our environment:

    export GOROOT=/usr/local/go
    export PATH=$GOPATH/bin:$GOROOT/bin:$PATH
    export GOPATH=$HOME/go/beats

Here, I also set up to GOPATH. You can set your own path. For my case, I created in my home directory go up a directory and catalog production go under a directory named beats. In it, we will generate our custom beat in this directory.

Download Elastic beats Source

In this step we Elastic beats download the source code. Into the following command in termnial:

    mkdir -p ${GOPATH}/src/github.com/elastic
    git clone https://github.com/elastic/beats ${GOPATH}/src/github.com/elastic/beats

Installing Python

Python2 applicable only to the current generator, so we need to install Python2. We can refer to page https://www.python.org/downloads/ install our python2.
Installation virtualenv

We must be installed so that the generator virtualenv to work properly. You can install reference link https://virtualenv.pypa.io/en/latest/installation/. If your own computer while python3 already installed, then we need to set variables such as writing:

    export PYTHON_EXE='python2.7'
    export VIRTUALENV_PARAMS='-p python2.7'
    export VIRTUALENV_PYTHON='/usr/bin/python2.7'
     
    export VIRTUALENV_PYTHON='/usr/local/bin/python' (for Mac)

Please note: python here is 2.x version of python, rather than python3. We need to ensure that VIRTUALENV_PYTHON pointing to the executable file of our Python2.

Installation mage

We need to download the source code in the address https://github.com/magefile/mage, and compile:

    go get -u -d github.com/magefile/mage
    cd $GOPATH/src/github.com/magefile/mage
    go run bootstrap.go

After the implementation of the above command, etc., we can find the compiled executable file in the following directory mage:

    liuxg-2:bin liuxg$ ls $GOPATH/bin
    mage

Create a custom beat

First, create a directory under $ GOPATH, and enter the directory.

    mkdir ${GOPATH}/src/github.com/{user}
    cd ${GOPATH}/src/github.com/{user}

Note that the user refers to the same username on the github. For example, for my case is liu-xiao-guo. I enter the following command to write:

    mkdir ${GOPATH}/src/github.com/liu-xiao-guo
    cd  $GOPATH/src/github.com/elastic/beats/

Next, we run the following command:

mage GenerateCustomBeat

Results of the:

    $ mage GenerateCustomBeat
    2019/11/13 15:24:01 Found Elastic Beats dir at /Users/liuxg/go/beats/src/github.com/elastic/beats
    Enter the beat name [examplebeat]: Countbeat
    Enter your github name [your-github-name]: liu-xiao-guo
    Enter the beat path [github.com/liu-xiao-guo/countbeat]: 
    Enter your full name [Firstname Lastname]: Xiaoguo Liu
    Enter the beat type [beat]: 
    DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
    DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
     
    WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='pypi.tuna.tsinghua.edu.cn', port=443): Read timed out. (read timeout=15)",)': /simple/semver/
    2019/11/13 15:25:50 Found Elastic Beats dir at /Users/liuxg/go/beats/src/github.com/liu-xiao-guo/countbeat/vendor/github.com/elastic/beats
    Generated fields.yml for countbeat to /Users/liuxg/go/beats/src/github.com/liu-xiao-guo/countbeat/fields.yml
    2019/11/13 15:25:52 Found Elastic Beats dir at /Users/liuxg/go/beats/src/github.com/liu-xiao-guo/countbeat/vendor/github.com/elastic/beats
    Auto packing the repository in background for optimum performance.
    See "git help gc" for manual housekeeping.
    =======================
    Your custom beat is now available as /Users/liuxg/go/beats/src/github.com/liu-xiao-guo/countbeat
    =======================

In this way, we basically produced a basic framework of the beat.

Next, we enter into our beat directory, and compile:

cd ${GOPATH}/src/github.com/{user}/countbeat

For my situation:

cd ${GOPATH}/src/github.com/liu-xiao-guo/countbeat

We can look inside the most basic documents:

    $ pwd
    /Users/liuxg/go/beats/src/github.com/liu-xiao-guo/countbeat
    liuxg-2:countbeat liuxg$ ls
    CONTRIBUTING.md     cmd         magefile.go
    LICENSE.txt     config          main.go
    Makefile        countbeat.docker.yml    main_test.go
    NOTICE.txt      countbeat.reference.yml make.bat
    README.md       countbeat.yml       tests
    _meta           docs            vendor
    beater          fields.yml
    build           include

Here are the basic framework document. Which contains a countbeat.yml profile and some standard template file called. We directly into the following command at the command line:

make

    $ make
    go build -i -ldflags "-X github.com/liu-xiao-guo/countbeat/vendor/github.com/elastic/beats/libbeat/version.buildTime=2019-11-13T07:33:25Z -X github.com/liu-xiao-guo/countbeat/vendor/github.com/elastic/beats/libbeat/version.commit=501bd87da668346f78398676c78b4a39394a3640"

After compiling the above, we can find in the current directory, there is a good countbeat compiled executable file:

We run the executable file directly in the current directory:

./countbeat -e -d "*"

We can see in the terminal:

So in our Kibana you can also see the following information:

Obviously the data has been successfully uploaded to the Elasticsearch in the.

The contents of each document are as follows:

    {
      "@timestamp": "2019-11-13T07:38:57.095Z",
      "agent": {
        "version": "8.0.0",
        "type": "countbeat",
        "ephemeral_id": "d3f0638e-ee58-45ff-92cc-74f188fd66a4",
        "hostname": "liuxg-2.local",
        "id": "1d35220e-7f75-442a-88eb-43ec1e97f0d0"
      },
      "counter": 5,
      "ecs": {
        "version": "1.2.0"
      },
      "host": {
        "hostname": "liuxg-2.local",
        "architecture": "x86_64",
        "os": {
          "build": "19B88",
          "platform": "darwin",
          "version": "10.15.1",
          "family": "darwin",
          "name": "Mac OS X",
          "kernel": "19.0.0"
        },
        "id": "E51545F1-4BDC-5890-B194-83D23620325A",
        "name": "liuxg-2.local"
      },
      "type": "liuxg-2.local"
    }

Inside it contains an integer value of a counter.

All of the code on the beat of the design can be found in the directory $ {GOPATH} /beater/CountBeat.go file under /src/github.com/liu-xiao-guo/countbeat's implementation. Design is relatively straightforward. You can look at the code should be able to understand.

Read JSON file beat

In the above we are already familiar with how to create a template of the beat. It is a basic beat, and no special features. In this section, we then do the same to create a slightly beat it uses. This beat is called our readjson beat. Its source can be obtained according to the following method:

git clone https://github.com/liu-xiao-guo/beats-readjson

First, we can prepare a json file we want, for example:

users.json

    {
      "users": [
        {
          "name": "Elliot",
          "type": "Reader",
          "age": 23,
          "social": {
            "facebook": "https://facebook.com",
            "twitter": "https://twitter.com"
          }
        },
        {
          "name": "Fraser",
          "type": "Author",
          "age": 17,
          "social": {
            "facebook": "https://facebook.com",
            "twitter": "https://twitter.com"
          }
        }
      ]
    }

We can put this document into how we liked the location. For my case, I put it in my computer's location as follows:

/Users/liuxg/data/beats/users.json

我们可以在readjson.yml文件中进行配置：

readjson.yml

我们的readjson.go设计也相当简单：

readjson.go

    package beater
     
    import (
        "fmt"
        "os"
        "io/ioutil"
        "encoding/json"
        "strconv"
        "time"
        "os/signal"
        "syscall"
     
        "github.com/elastic/beats/libbeat/beat"
        "github.com/elastic/beats/libbeat/common"
        "github.com/elastic/beats/libbeat/logp"
     
        "github.com/liu-xiao-guo/readjson/config"
    )
     
    type Users struct {
        Users []User `json:"users"`
    }
     
    // User struct which contains a name
    // a type and a list of social links
    type User struct {
        Name   string `json:"name"`
        Type   string `json:"type"`
        Age    int    `json:"Age"`
        Social Social `json:"social"`
    }
     
    // Social struct which contains a
    // list of links
    type Social struct {
        Facebook string `json:"facebook"`
        Twitter  string `json:"twitter"`
    }
     
    // readjson configuration.
    type readjson struct {
        done   chan struct{}
        config config.Config
        client beat.Client
    }
     
    // New creates an instance of readjson.
    func New(b *beat.Beat, cfg *common.Config) (beat.Beater, error) {
        c := config.DefaultConfig
        if err := cfg.Unpack(&c); err != nil {
            return nil, fmt.Errorf("Error reading config file: %v", err)
        }
     
        bt := &readjson{
            done:   make(chan struct{}),
            config: c,
        }
        return bt, nil
    }
     
    // Run starts readjson.
    func (bt *readjson) Run(b *beat.Beat) error {
        logp.Info("readjson is running! Hit CTRL-C to stop it.")
        var err error
        bt.client, err = b.Publisher.Connect()
        if err != nil {
            return err
        }
     
        fmt.Println("Path: ", bt.config.Path)
        fmt.Println("Period: ", bt.config.Period)
     
        
        // Open our jsonFile
        jsonFile, err := os.Open(bt.config.Path)
        // if we os.Open returns an error then handle it
        if err != nil {
            fmt.Println(err)
        }
     
        fmt.Println("Successfully Opened users.json")
        // defer the closing of our jsonFile so that we can parse it later on
        defer jsonFile.Close()
        
     
        byteValue, _ := ioutil.ReadAll(jsonFile)
     
        // we initialize our Users array
        var users Users
     
        json.Unmarshal(byteValue, &users)
     
        // we iterate through every user within our users array and
        // print out the user Type, their name, and their facebook url
        // as just an example
        for i := 0; i < len(users.Users); i++ {
            fmt.Println("User Type: " + users.Users[i].Type)
            fmt.Println("User Age: " + strconv.Itoa(users.Users[i].Age))
            fmt.Println("User Name: " + users.Users[i].Name)
            fmt.Println("Facebook Url: " + users.Users[i].Social.Facebook)
     
            event := beat.Event{
                Timestamp: time.Now(),
                Fields: common.MapStr {
                    "ostype":       b.Info.Name,
                    "name":     users.Users[i].Name,
                    "type":     users.Users[i].Type,
                    "age":      users.Users[i].Age,
                    "social":   users.Users[i].Social,
                },
            }
     
            bt.client.Publish(event)
        }
     
        c := make(chan os.Signal)
        signal.Notify(c, os.Interrupt, syscall.SIGTERM)
        go func() {
            <-c
            os.Exit(1)
        }()
     
        for {
            fmt.Println("sleeping...")
            time.Sleep(10 * time.Second)
        }   
    }
     
    // Stop stops readjson.
    func (bt *readjson) Stop() {
        bt.client.Close()
        close(bt.done)
    }

它在run method里把json文件读入，并把它们分别发送出去到我们的Elasticsearch中。

我们按照上面的步骤进行编译，并最终运行我们的readjson beat。

./readjson -e

我们可以在Kibana中看到我们已经发送上来的beat信息：

参考：
【1】https://www.elastic.co/guide/en/beats/devguide/7.5/newbeat-generate.html