EFK-nginx日志收集详细步骤+解析+图像化配置!!!

在上一篇的基础上 【153端】操作

1.安装epel和ab压测

yum -y install epel-release  httpd-tools

2.安装nginx并启动

yum -y install nginx
nginx

3.验证端口

ss -nlpt |grep 80

4.编辑filebeat配置文件

vim /etc/filebeat/filebeat.yml

内容如下:

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/messages
  fields:
    log_topics: msg007
###nginx##
- type: log
  enabled: true
  paths:
    - /var/log/nginx/access.log
  fields:
    log_topics: nginx007

output.kafka:
  enabled: true
  hosts: ["192.168.112.153:9092","192.168.112.154:9092","192.168.112.155:9092"]
  topic: '%{[fields][log_topics]}'

5. 启动filebeat

systemctl  restart filebeat

6.添加nginx配置文件【154端】

vim /etc/logstash/conf.d/nginx.conf

内容如下:

input{
    
    
        kafka{
    
    
                bootstrap_servers => ["192.168.10.130:9092,192.168.10.131:9092,192.168.10.132:9092"]
                group_id => "logstash"
                topics => "nginx007"
                consumer_threads => 5
        }

}
filter {
    
    
        json{
    
    
                source => "message"
        }

        mutate {
    
    
                remove_field => ["host","prospector","fields","input","log"]
        }
        grok {
    
    
                match => {
    
     "message" => "%{NGX}" }
        }

}

output{
    
    
        elasticsearch {
    
    
                hosts => "192.168.10.130:9200"
                index => "nginx-%{+YYYY.MM.dd}"
            }      
}

7.修改管道【化一为二】

vim /etc/logstash/pipelines.yml

8.编写正则匹配

路径

vim /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-patterns-core-4.1.2/patterns/nginx

内容

NGX %{
    
    IPORHOST:client_ip} (%{
    
    USER:ident}|- ) (%{
    
    USER:auth}|-) \[%{
    
    HTTPDATE:timestamp}\] "(?:%{WORD:verb} (%{NOTSPACE:request}|-)(?: HTTP/%{NUMBER:http_version})?|-)" %{
    
    NUMBER:status} (?:%{
    
    NUMBER:bytes}|-) "(?:%{URI:referrer}|-)" "%{GREEDYDATA:agent}"

9.重启/验证

systemctl restart logstash
ss -nlpt|grep 9600

10.【153端】

触发生成数据

chmod -R  777 /var/log/*                              【加权限】
 ab -n 100 -c 100 http://1923168.112.153/index.html  【压测数据】

11.登录测试

返回上一层