ELK收集Nginx日志

重点是通过Logstash正则提取Nginx日志

一、Nginx日志格式配置

  log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '

                      '$status $body_bytes_sent "$http_referer" '

                      '"$http_user_agent" "$http_x_forwarded_for"';

二、Nginx日志格式 

192.168.20.7 - - [18/Nov/2020:21:42:26 +0800] "GET / HTTP/1.1" 200 612 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; 

x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.198 Safari/537.36" "-"

192.168.20.7 - - [18/Nov/2020:21:42:26 +0800] "GET /favicon.ico HTTP/1.1" 404 555 "http://192.168.20.41/" "Mozil

la/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.198 Safari/537.36" "

-"

三、内置正则提取语法：%{内置正则表达式:字段名}

%{IP:remote_addr} - (%{WORD:remote_user}|-) \[%{HTTPDATE:time_local}\] "%{WORD:method} %{NOTSPACE:request} HTTP/%{NUMBER}" %{NUMBER:status} %{NUMBER:body_bytes_sent} %{QS} %{QS:http_user_agent}

四、Logstash正则提取Nginx写入ES

1、使用kibana自身的Grok提取，需要掌握正则表达式

2、使用正式提取语法

3、把正行的nginx日志分段提取出来

4、配置logstash

5、默认logstash收集nginx日志

[root@master nginx]# more /etc/logstash/conf.d/logstash.conf 

input {

  file {

    path => "/var/log/nginx/access.log"

  }

}

output {

  elasticsearch {

    hosts => ["http://192.168.20.41:9200", "http://192.168.20.42:9200"]

    user => "elastic"

    password => "hahashen"

    index => "sjgnginx-%{+YYYY.MM.dd}"

  }

}

6、kibana创建Create index pattern

下一步

7、kibana显示nginx日志

8、最后正则拆分nginx日志

 请大家扫描关注，共同学习