Miss Sister teaches you to customize a Logstash Java Filter ~

Logstash is a processing engine used to collect data, parse and process the data, and finally output the data to the storage component. The data processing flow is:

Logstash Java Filter is to 基于Logstash的Filter扩展APIdevelop a 用Java语言实现的Filter, and then will Filter代码打包构建到自己服务器上的Logstash Filter libbe. You can 数据流转配置文件中(也就是logstash -f 指定的配置文件)use this customized Logstash Java Filter.

The customization steps include the following five steps:

1. Prepare the Logstash environment

Because Logstash Java Filter depends on Logstash API, we need to download and build Logstash source code

1.1. Download logstash source code

git clone --branch <branch_name> --single-branch https://github.com/elastic/logstash.git <target_folder>
复制代码

Which <branch_name>needs to be replaced with the logstash version you want to use, you can use the GA version after 7.1. <target_folder> needs to be replaced with the parent directory of the logstash code you want to download, if not specified, it will be downloaded to the logstash folder of the current directory. I am using version 7.6 here:

git clone --branch 7.6  --single-branch https://github.com/elastic/logstash.git 
复制代码

1.2. Build logstash source code

Go to the logstash directory of the current directory (that is, logstash source directory, hereinafter referred to as:) $LS_HOMEand execute

./gradlew assemble
复制代码

If it is a Windows system, executegradlew.bat assemble

This step has to wait for a long time, if you can't download it, you can try to add gradle's domestic mirror. vim $LS_HOME/build.gradleAnd then add to the file

    repositories {
           maven { url 'https://maven.aliyun.com/repository/google/' }
           maven { url 'https://maven.aliyun.com/repository/jcenter/'}
           mavenCentral()
   
           maven {
               url 'https://plugins.gradle.org/m2/'
           }
      }
复制代码

After successful construction, check $LS_HOME/logstash-core/build/libs/whether it is generated in the directory logstash-core-x.y.z.jar. Where x, y, z are the logstash version numbers you downloaded. Mine is

/Users/xx/corprepo/logstash/logstash-core/build/libs/logstash-core-7.6.3.jar
复制代码

2. Write Logstash Java Filter code

2.1. Download the official demo

The official provides a demo , which we can download and modify based on this demo .

2.2. Specify LOGSTASH_CORE_PATH

After downloading the demo, create a gradle.propertiesfile in the root directory of the project and add a line of data:

LOGSTASH_CORE_PATH=<target_folder>/logstash-core
复制代码

2.3. Development of Filter code

We need to inherit Logstash's Filter API to implement our own Java Filter function. A good Filter is as follows:

import co.elastic.logstash.api.Configuration;
import co.elastic.logstash.api.Context;
import co.elastic.logstash.api.Event;
import co.elastic.logstash.api.Filter;
import co.elastic.logstash.api.FilterMatchListener;
import co.elastic.logstash.api.LogstashPlugin;
import co.elastic.logstash.api.PluginConfigSpec;
import org.apache.commons.lang3.StringUtils;

import java.util.Collection;
import java.util.Collections;
//类名必须按照驼峰命名匹配这个下划线注解名，JavaFilterExample -> java_filter_example
@LogstashPlugin(name = "java_filter_example")
public class JavaFilterExample implements Filter {
    //定义一个该Filter支持的setting配置。名字是source，默认值为message
    //可从filter方法中看出是拿 SOURCE_CONFIG 的value值做field 的名称使用的
    public static final PluginConfigSpec<String> SOURCE_CONFIG =
            PluginConfigSpec.stringSetting("source", "message");

    private String id;
    private String sourceField;

    public JavaFilterExample(String id, Configuration config, Context context) {
        // constructors should validate configuration options
        this.id = id;
        this.sourceField = config.get(SOURCE_CONFIG);
    }

    /**
     * 该Filter的过滤逻辑，可以对输入的event数据做各种CRUD操作
     * @param events
     * @param matchListener
     * @return 最终流转到下一个pipeline的数据，如果有符合条件的event必须返回
     */
    @Override
    public Collection<Event> filter(Collection<Event> events, FilterMatchListener matchListener) {
        for (Event e : events) {
            Object f = e.getField(sourceField);
            if (f instanceof String) {
                e.setField(sourceField, StringUtils.reverse((String)f));
                matchListener.filterMatched(e);
            }
        }
        return events;
    }
    /**
     *
     * @return 返回该Filter支持的所有setting配置
     */
    @Override
    public Collection<PluginConfigSpec<?>> configSchema() {
        // should return a list of all configuration options for this plugin
        return Collections.singletonList(SOURCE_CONFIG);
    }

    /**
     *
     * @return 该Filter的ID，Logstash会帮我们赋值
     */
    @Override
    public String getId() {
        return this.id;
    }
}
复制代码

There are two points to note:

• @LogstashPluginThe annotation namemust be highly consistent with the class name. Such as java_filter_example-> JavaFilterExample (I'm hacked anyway ...)

• The implementation co.elastic.logstash.api.Filterclass is required . If your import is unsuccessful, it means that the gradle.propertiesconfiguration is unsuccessful or the logstash source code is unsuccessful. Rewrite its three methods:

getId方法

Return the ID of the Filter, and Logstash will assign values ​​for us. We only need to define one 成员变量, and that's 构造方法中赋值进去it.

configSchema方法

Returns all the setting configuration sets supported by this Filter. PluginConfigSpecDefined setting配置time that we use the Filter in logstash configuration file, you can pass parameters, such as when using the pass in grok Filter patterns_dirand match.

filter {
      grok {
        patterns_dir => ["./patterns"]
        match => { "message" => "%{SYSLOGBASE} %{POSTFIX_QUEUEID:queue_id}: %{GREEDYDATA:syslog_message}" }
      }
    }
复制代码

The PluginConfigSpecconfiguration parameters supported by this setting are name, type, deprecation status, required status, 和 default value:

Filter in our class, we define PluginConfigSpec<String> SOURCE_CONFIG = PluginConfigSpec.stringSetting("source", "message");whichname=source, default value= message

filter方法

The filter must of course do the filtering. Collection<Event> eventsThe input parameter is the input data that we want to process. We can do some CURD operations on the logic. FilterMatchListener matchListenerThe input parameter is that the Filter notifies matchListener of event data that satisfies its own logic. For example, the implementation of matchListener in Logstash is DecoratingFilterMatchListener. The operations it can do, for example, ADD_FIELDalso require us to define PluginConfigSpec first, and then configure the add_fieldparameters when using the Filter . For example, grok Filter supports this parameter and the DecoratingFilterMatchListener

 filter {
      grok {
        add_field => { "foo_%{somefield}" => "Hello world, from %{host}" }
      }
    }
复制代码

There is no need to call when there is no need to notify matchListener matchListener.filterMatched(e).

3. Unit Testing

There are also test classes in the demo, and it will be finished after a run. .

4. Package and deploy Filter

We need to use gradle to get our Filter project to the ruby ​​gem package, so it is best to modify it based on the gradle configuration file in the demo project.

4.1. Configure gradle packaging task Edit the build.gradlefile under the project root path

plugin infoPart of it is the information of our Filter. I have TODOmarked the special points that need to be modified . 4.2. Run the gradle packaging task and execute it in the root directory of the project

./gradlew gem
复制代码

Windows system executiongradlew.bat gem

After successful execution, you will see a logstash-{plugintype}-<pluginName>-<version>.gemfile generated in the root directory of the project

4.3. Install the filter gem package in Logstash and execute it in the logstash directory ($ LS_HOME)

bin/logstash-plugin install --no-verify --local /path/to/javaPlugin.gem
复制代码

Which /path/to/javaPlugin.gemis an absolute gem path we generated in step 4.2.

5. Use our Java Filter to run Logstash

5.1. Create logstash running configuration file in $ LS_HOME / config directoryjava_filter.conf

input {
  generator { message => "Hello world!" count => 1 }
}
filter {
# java_filter_example：我们的filter中@LogstashPlugin注解的name
  java_filter_example {}
}
output {
  stdout { codec => rubydebug }
}
复制代码

5.2. Start Logstash and run it in $ LS_HOME

bin/logstash -f  config/java_filter.conf
复制代码

That's it ~

{
       "message" => "!dlrow olleH",
      "sequence" => 0,
      "@version" => "1",
          "host" => "xxdeMacBook-Pro.local",
    "@timestamp" => 2020-04-12T13:15:30.376Z
}
复制代码

Thank you for reading, my name is Monica23334 || Monica2333. Ladies who write an original article flag every week, follow me and look forward to a face-slap ~

Refer to the official documentation: www.elastic.co/guide/en/lo…