本例介绍如何使用Filebeat收集Nginx日志，在【Beats】 Filebeat介绍及使用（十六）中，介绍了如何抓入日志，
前面要想实现日志数据的读取以及处理都是自己手动配置的，其实，在Filebeat中，有大量的Module，可以简化我 们的配置，直接就可以使用，如下:
Filebeat Nginx Module使用
Nginx日志格式如下：
1 log_format upstreaminfo 2 '$remote_addr - $remote_user [$time_local] "$request" ' 3 '$status $body_bytes_sent "$http_referer" "$http_user_agent" ' 4 '$request_length $request_time [$proxy_upstream_name] [$proxy_alternative_upstream_name] $upstream_addr ' 5 '$upstream_response_length $upstream_response_time $upstream_status $req_id';
1、查看Filebeat支持模块
命令：./filebeat modules list
2、启动Nginx模块
启动命令：./filebeat modules enable nginx
禁用命令：./filebeat modules disable nginx
3、查看modules.d目录中的文件，可以看到nginx.yml配置文件，证明nginx module模块已开启
命令：ls modules.d/
4、配置modules.d/nginx.yml文件，指定access.log 和 error.log 的地址
命令：vim modules.d/nginx.yml
1 # Module: nginx 2 # Docs: https://www.elastic.co/guide/en/beats/filebeat/7.6/filebeat-module-nginx.html 3 4 - module: nginx 5 # Access logs 6 access: 7 enabled: true 8 var.paths: ["/data/logs/nginx-1.16.1/access.log"] 9 10 # Set custom paths for the log files. If left empty, 11 # Filebeat will choose the paths depending on your OS. 12 #var.paths: 13 14 # Error logs 15 error: 16 enabled: true 17 var.paths: ["/data/logs/nginx-1.16.1/error.log"] 18 19 # Set custom paths for the log files. If left empty, 20 # Filebeat will choose the paths depending on your OS. 21 #var.paths:
5、配置test-nginx.yml文件
1 # 自定义测试配置文件test-nginx.yml 2 3 # 输入 4 filebeat.inputs: 5 6 # 指定索引的分区数 7 setup.template.settings: 8 index.number_of_shards: 3 9 10 filebeat.config.modules: 11 path: ${path.config}/modules.d/*.yml 12 reload.enabled: false 13 14 # 输出到指定ES的配置 15 output.elasticsearch: 16 hosts: ["127.0.0.1:9200"] 17 username: "elastic" 18 password: "123456"
6、运行filebeat：
命令：./filebeat -e -c test-nginx.yml -d "publish"
7、查看ES数据如下：
1 { 2 "_index": "filebeat-7.6.1-2020.06.21-000001", 3 "_type": "_doc", 4 "_id": "3gsc3XIBeBo_vUMFUe7F", 5 "_version": 1, 6 "_score": null, 7 "_source": { 8 "agent": { 9 "hostname": "H__D", 10 "id": "9f14c4db-2f85-4740-8183-36f475ffdfed", 11 "type": "filebeat", 12 "ephemeral_id": "52fb4acc-a216-4d5e-9e66-aff57694c4c2", 13 "version": "7.6.1" 14 }, 15 "nginx": { 16 "access": { 17 "remote_ip_list": [ 18 "127.0.0.1" 19 ] 20 } 21 }, 22 "log": { 23 "file": { 24 "path": "/data/logs/nginx-1.16.1/access.log" 25 }, 26 "offset": 41957019 27 }, 28 "source": { 29 "geo": { 30 "continent_name": "Asia", 31 "region_iso_code": "CN-ZJ", 32 "city_name": "Hangzhou", 33 "country_iso_code": "CN", 34 "region_name": "Zhejiang", 35 "location": { 36 "lon": 120.1619, 37 "lat": 30.294 38 } 39 }, 40 "as": { 41 "number": 37963, 42 "organization": { 43 "name": "Hangzhou Alibaba Advertising Co.,Ltd." 44 } 45 }, 46 "address": "127.0.0.1", 47 "ip": "127.0.0.1" 48 }, 49 "fileset": { 50 "name": "access" 51 }, 52 "url": { 53 "original": "/.reporting-*/_search" 54 }, 55 "input": { 56 "type": "log" 57 }, 58 "@timestamp": "2020-06-22T17:38:37.000Z", 59 "ecs": { 60 "version": "1.4.0" 61 }, 62 "service": { 63 "type": "nginx" 64 }, 65 "host": { 66 "name": "H__D" 67 }, 68 "http": { 69 "request": { 70 "referrer": "-", 71 "method": "POST" 72 }, 73 "response": { 74 "status_code": 200, 75 "body": { 76 "bytes": 159 77 } 78 }, 79 "version": "1.1" 80 }, 81 "event": { 82 "timezone": "+08:00", 83 "created": "2020-06-22T17:38:39.155Z", 84 "module": "nginx", 85 "dataset": "nginx.access" 86 }, 87 "user": { 88 "name": "kibana" 89 }, 90 "user_agent": { 91 "original": "-", 92 "name": "Other", 93 "device": { 94 "name": "Other" 95 } 96 } 97 }, 98 "fields": { 99 "event.created": [ 100 "2020-06-22T17:38:39.155Z" 101 ], 102 "suricata.eve.timestamp": [ 103 "2020-06-22T17:38:37.000Z" 104 ], 105 "@timestamp": [ 106 "2020-06-22T17:38:37.000Z" 107 ] 108 }, 109 "sort": [ 110 1592847517000 111 ] 112 }