一般模式 FileBeat(更轻量化) --> Logstash
1、Logstash是什么？
Logstash是一个日志收集、过滤、转发的中间件，主要负责将各条业务线的各类日志统一收集、过滤后，转发给 Elasticsearch 进行下一步处理
2、安装Logstash
logstash无需安装，直接解压就行。收集日志需要写符合业务需求的配置文件。下面从最基本的控制台输入输出开始，一步步配置符合内心期望的收集、过滤等配置。
1）简单安装
# cd /usr/local/src
# tar -zxvf logstash-5.2.2.tar.gz
# mv logstash-5.2.2 /usr/local/logstash
2）为logstash安装x-pack
# cd /usr/local/logstash/bin
# ./logstash-plugin install x-pack
[root@test bin]# ./logstash-plugin install x-pack
Downloading file: https://artifacts.elastic.co/downloads/logstash-plugins/x-pack/x-pack-5.2.2.zip
Downloading [=============================================================] 100%
Installing file: /tmp/studtmp-e97bf0800211a6de985aa19c225dd2a5dce0d369da9f5eb975df52c0adee/x-pack-5.2.2.zip
Install successful
[root@test bin]#
3）创建配置文件，指定数据流向，在目录下创建一个simple.conf，内容如下
input { beats { port => "5044" } } # 数据过滤 filter { grok { match => { "message" => "%{COMBINEDAPACHELOG}" } } geoip { source => "clientip" } } # 输出到本机的 ES output { elasticsearch { hosts => [ "127.0.0.1:9200" ] } }
3）上述配置输出日志到ES，启动命令如下：
# cd /usr/local/logstash/bin
# ./logstash -f /usr/local/logstash/config/simple.conf --config.reload.automatic
启动时报错：LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :error=>"Got response code '401' contacting Elasticsearch at URL 'http://127.0.0.1:9200/'"}
原因：之前给ES 安装了x-pack插件，访问需要登录账号和密码
解决：调整logstash配置文件，在output中 访问es时增加 user、password
# cd /usr/local/logstash/config
# vi + simple.conf output { elasticsearch { hosts => [ "127.0.0.1:9200" ]
user => elastic
password=>changeme } }
# cd /usr/local/logstash/bin
# ./logstash -f /usr/local/logstash/config/simple.conf --config.reload.automatic 重新启动
...
[2018-08-03T19:08:53,852][INFO ][logstash.filters.geoip ] Using geoip database {:path=>"/usr/local/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-geoip-4.0.4-java/vendor/GeoLite2-City.mmdb"}
[2018-08-03T19:08:53,879][INFO ][logstash.pipeline ] Starting pipeline {"id"=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>500}
[2018-08-03T19:08:54,307][INFO ][logstash.inputs.beats ] Beats inputs: Starting input listener {:address=>"0.0.0.0:5044"}
[2018-08-03T19:08:54,343][INFO ][logstash.pipeline ] Pipeline main started
[2018-08-03T19:08:54,408][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
参考地址https://blog.csdn.net/Ahri_J/article/details/79609444
3、安装FileBeats
下载地址： https://www.elastic.co/downloads/beats/filebeat
安装文件：filebeat-5.2.2-linux-x86_64.tar.gz
1）简单安装FileBeats
# cd /usr/local/src
# tar -zxvf filebeat-5.2.2-linux-x86_64.tar.gz
# mv filebeat-5.2.2-linux-x86_64 /usr/local/filebeat
2) 配置 filebeats.yml ，指定读取日志目录，指定输出到本机logstash
- input_type: log
# Paths that should be crawled and fetched. Glob based paths.
paths:
- /home/daxiang/logs/mobile/*.log
#----------------------------- Logstash output --------------------------------
output.logstash:
# The Logstash hosts
hosts: ["localhost:5044"]
3）启动
测试启动命令
# ./filebeat -e -c filebeat.yml -d "publish"
正常后台运行启动，日志开始采集
# nohup ./filebeat -e -c filebeat.yml >/dev/null 2>&1 &
4、Logstash基本学习
1)运行最基本的logstash管道
# cd /usr/local/logstash/bin
# ./logstash -e 'input { stdin {} } output { stdout {}}'
运行效果如下
[root@test bin]# ./logstash -e 'input { stdin {} } output { stdout {}}'
Sending Logstash's logs to /usr/local/logstash/logs which is now configured via log4j2.properties
[2018-07-25T16:18:00,220][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.queue", :path=>"/usr/local/logstash/data/queue"}
[2018-07-25T16:18:00,238][INFO ][logstash.agent ] No persistent UUID file found. Generating new UUID {:uuid=>"1850a7c6-e6b8-43ae-9079-6e8d27be4173", :path=>"/usr/local/logstash/data/uuid"}
[2018-07-25T16:18:00,410][INFO ][logstash.pipeline ] Starting pipeline {"id"=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>500}
[2018-07-25T16:18:00,433][INFO ][logstash.pipeline ] Pipeline main started
The stdin plugin is now waiting for input:
[2018-07-25T16:18:00,506][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
hello (在控制台输入hello，可看到如下输出)
2018-07-25T08:18:34.363Z test hello
hello world
2018-07-25T08:19:15.051Z test hello world (当前输出格式 时间戳 主机名 输入内容)
2)配置输出时，指定使用codec
创建一个logstash-simple.conf
# cd /usr/local/
# touch logstash-simple.conf
# vi logstash-simple.conf
#输入来源
input { stdin {} }
#指定输出格式
output {
stdout{
codec=>rubydebug
}
}
# cd /usr/local/logstash/bin
# ./logstash -f /usr/local/logstash/config/logstash-simple.conf
执行结果如下
[root@test bin]# ./logstash -f /usr/local/logstash/config/logstash-simple.conf
Sending Logstash's logs to /usr/local/logstash/logs which is now configured via log4j2.properties
[2018-07-25T16:53:33,755][INFO ][logstash.pipeline ] Starting pipeline {"id"=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>500}
[2018-07-25T16:53:33,794][INFO ][logstash.pipeline ] Pipeline main started
The stdin plugin is now waiting for input:
[2018-07-25T16:53:33,849][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
hello 输入hello，返回了JSON
{
"@timestamp" => 2018-07-25T08:53:45.928Z,
"@version" => "1",
"host" => "test",
"message" => "hello"
}
5、Logstash配置指定日志的切割正则，需用到ruby做正则，待后续完善。