logstash-jdbc-input sync with mysql database

In most cases, our data is stored in the database, but elasticsearch has its own index library, so if we are doing searches, we need to synchronize the data in the database to elasticsearch. Here we use logstash's The plugin of logstash-jdbc-input synchronizes with the database. For the synchronization of logstash and the database, we can set the synchronization time between elasticsearch and the database. It is very convenient to use this method to synchronize.

1. Download and install logstash

 Note that the downloaded version should be the same as your elasticsearch version number , my version elasticsearch6.2.2

Logstash download address: https://www.elastic.co/downloads/logstash

After downloading, just unzip it directly

(For the environment construction of elasticsearch, please refer to http://www.cnblogs.com/xuwenjin/p/8745624.html )

 

2. Configure logstash

For versions above logstash 5.x, it has integrated this plug-in itself, so we don't need to install it separately, just use it directly. Let me talk about a simple configuration for synchronizing with mysql here

In the logstash file directory, create a new folder (name it arbitrarily). Such as: mysql

2.1 First put a jdbc driver in this folder to connect to the mysql database

　 For maven projects, you can add the following dependencies to the pom file, and then copy the jar package.

　<dependency>
        <groupId>mysql</groupId>
        <artifactId>mysql-connector-java</artifactId>
        <version>5.1.40</version>
  </dependency>

2.2 Create a .conf configuration file (named at will) to associate es with the database. Such as mysql.conf

input {
    jdbc {
      # mysql database link, shop is the database name
      jdbc_connection_string => "jdbc:mysql://dev.yonyouccs.com:3001/test"
      # user name and password
      jdbc_user => "root"
      jdbc_password => "root"
      # drive
      jdbc_driver_library => "D:/software/logstash-6.2.2/logstash-6.2.2/mysqletc/mysql-connector-java-5.1.40.jar"
      # driver class name
      jdbc_driver_class => " com.mysql.jdbc.Driver " 
      jdbc_paging_enabled => " true " 
      jdbc_page_size => " 50000 " 
      # Executed sql file path + name
      statement_filepath => " D:/software/logstash-6.2.2/logstash-6.2.2/mysql/jdbc.sql " 
      # Set the meaning of each field of the listening interval (from left to right) minutes, hours, days, months, years, All * Default means update every minute
      schedule => "* * * * *"
    }
}

filter {
    json {
        source => "message"
        remove_field => ["message"]
    }
}

output {
    elasticsearch {
        # ES IP address and port
        hosts => ["localhost:9200"]
        # index name
        index => "synces"
        # There is an id field in the database that needs to be associated, corresponding to the id in the type
        document_id => "%{id}"
        # index name
        document_type => "xwjUser"
    }
    stdout {
        # JSON format output
        codec => json_lines
    }
}

2.3 Create a sql file, which I named jdbc.sql and the configuration file above

SELECT id, last_name lastName, age, email FROM xwj_user

Note: sql cannot have a terminator, otherwise it will report an error when running (as for the reason, it will be mentioned later)

 

3. Start logstash

In the bin directory of logstash, use cmd to execute the command: logstash -f ../mysql/mysql.conf

You will see the following startup information:

It can be seen that during the synchronization process, our script is executed and wrapped, so the sql script cannot have a terminator.

It will also print out the synchronized data as a json string ( the field names are all lowercase by default )

 

4. View data through elasticsearch-head

You can see that all data has been synchronized to elasticsearch. However, under this index, two fields @version and @timestamp are added by default.

 

5. Stepped pit

1. The configuration file of logstash must be UTF-8, not GBK, otherwise an error will be reported

2. The logstash startup report cannot find the main class solution, please refer to https://www.cnblogs.com/sbj-dawn/p/8549369.html

 

 For more usage of logstash, please refer to elastic's official website: Jdbc input plugin