注:按顺序运行,不要一次性把配置文件全都配置好,然后直接启动,这个因为需要安装插件,所以有可能会有问题
docker run -d --restart=always -p 5044:5044 --name logstash logstash:7.3.2
拷贝配置文件:
docker mkdir -p /usr/local/docker/logstash
docker cp logstash:/usr/share/logstash/config /usr/local/docker/logstash/
docker cp logstash:/usr/share/logstash/pipeline /usr/local/docker/logstash/
docker cp logstash:/usr/share/logstash/Gemfile /usr/local/docker/logstash/Gemfile
启动:
#先删除旧容器
docker rm -f logstash
#以挂载方式启动
docker run -d --restart=always -p 5044:5044 --name logstash \
-v /usr/local/docker/logstash/config:/usr/share/logstash/config \
-v /usr/local/docker/logstash/pipeline:/usr/share/logstash/pipeline \
-v /usr/local/docker/logstash/Gemfile:/usr/share/logstash/Gemfile \
logstash:7.3.2
修改Gemfile,并安装输入输出插件
!注意:因为https://rubygems.org/已经停止维护了,所以需要将gem source替换为https://gems.ruby-china.com/
bin/logstash-plugin install logstash-input-jdbc
bin/logstash-plugin install logstash-output-elasticsearch
vi mysql.conf
=============================================================
修改如下配置文件:
/usr/local/docker/logstash/config/logstash.yml
http.host: "0.0.0.0"
xpack.monitoring.elasticsearch.hosts: [ "http://101.201.150.23:9200" ]
/usr/local/docker/logstash/config/pipelines.yml
- pipeline.id: goods
path.config: "/usr/share/logstash/pipeline/goods.conf"
/usr/local/docker/logstash/pipeline/goods.conf:
input {
jdbc {
jdbc_driver_library => "/usr/share/logstash/pipeline/mysql-connector-java-5.1.48.jar"
jdbc_driver_class => "com.mysql.jdbc.Driver"
jdbc_connection_string => "jdbc:mysql://xxxxx:3306/mytest"
jdbc_user => "xxxxx"
jdbc_password => "xxxxx"
schedule => "* * * * *"
lowercase_column_names => false
use_column_value => true
tracking_column_type => "timestamp"
tracking_column => "update_time"
statement => "SELECT id,name FROM goods t WHERE t.update_time >= :sql_last_value"
}
}
output {
elasticsearch {
action => "update"
doc_as_upsert => true
# ES的IP地址及端口
hosts => ["xxxxx:9200"]
# 索引名称 可自定义
index => "vc_base"
# 需要关联的数据库中有有一个id字段,对应类型中的id
document_id => "%{id}"
}
stdout {
# JSON格式输出
codec => json_lines
}
}