在EFK部署–zookeeper+kafka服务环境下
1,在elasticsearch服务的主机B下,安装nginx,并启动
[root@kafka02 ~] yum -y install gcc pcre-devel zlib-devel
[root@kafka02 ~] tar -zxf nginx-1.12.2.tar.gz
[root@kafka02 ~] cd nginx-1.12.2/
[root@kafka02 ~] ./configure
[root@kafka02 ~] make && make install
[root@kafka02 ~] /usr/local/nginx/sbin/nginx
[root@kafka02 ~] curl -I http://127.0.0.1 #测试一下
#可以在主机A上安装httpd-tools进行压测,增加nginx的日志量
[root@kafka01 ~] yum -y install httpd-tools
[root@kafka01 ~] ab -n 100 -c 100 http://192.168.59.111/index.html
2,在主机B上修改filebeat配置文件
[root@kafka02 ~] vim /etc/filebeat/filebeat.yml
[root@kafka02 ~] cat /etc/filebeat/filebeat.yml
filebeat.inputs:
- type: log
enabled: true
paths:
- /var/log/messages
fields: #新增字段
log_topic: msg #新增索引
#新增nginx日志收集项
- type: log
enabled: true
paths:
- /usr/local/nginx/logs/access.log
fields:
log_topics: nginx
output.kafka:
enabled: true
hosts: ["192.168.59.110:9092","192.168.59.111:9092","192.168.59.112:9092"]
# topic: msg
topic: '%{[fields][log_topics]}' #新增引入字段
3,查看主机B的当前topic,和创建nginx的topic,并启动filebeat
[root@kafka02 ~] /usr/local/kafka/bin/kafka-topics.sh --list --zookeeper 192.168.59.111:2181
__consumer_offsets
msg
[root@kafka02 ~] /usr/local/kafka/bin/kafka-topics.sh --create --zookeeper 192.168.59.111:2181 --replication-factor 2 --partitions 3 --topic nginx
Created topic nginx.
[root@kafka02 ~] systemctl restart filebeat
[root@kafka02 ~] tailf /var/log/filebeat/filebeat #查看有误报错
#模拟消费
[root@kafka02 ~] /usr/local/kafka/bin/kafka-console-consumer.sh --bootstrap-server 192.168.59.111:9092 --topic nginx --from-beginning
4,在logstash服务主机A下,修改配置nginx.conf,并重启logstash
[root@kafka01 ~] vim /etc/logstash/conf.d/nginx.conf
[root@kafka01 ~] cat /etc/logstash/conf.d/nginx.conf
input {
kafka {
bootstrap_servers => ["192.168.59.110:9092,192.168.59.111:9092,192.168.59.112:9092"]
group_id => "logstash"
topics => "nginx"
consumer_threads => 5
}
}
filter {
json {
source => "message" #过滤
}
mutate { #删除多余字段,因为日志字段量大而导致启动失败
remove_field => ["@version","fields","prospector","source","host","beat","input","offset","log"]
}
grok { #按照正则分将索引片
match => { "message" => "%{NGINXACCESS}" }
}
}
output {
elasticsearch {
hosts => "192.168.59.111:9200"
index => "nginx-%{+YYYY.MM.dd}"
}
}
#添加正则
[root@kafka01 ~] cd /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-patterns-core-4.1.2/patterns/
[root@kafka01 patterns] vim nginx_access
[root@kafka01 patterns] cat nginx_access
NGINXACCESS %{IPORHOST:client_ip} (%{USER:ident}|- ) (%{USER:auth}|-) \[%{HTTPDATE:timestamp}\] "(?:%{WORD:verb} (%{NOTSPACE:request}|-)(?: HTTP/%{NUMBER:http_version})?|-)" %{NUMBER:status} (?:%{NUMBER:bytes}|-) "(?:%{URI:referrer}|-)" "%{GREEDYDATA:agent}"
#增加日志收集管道
[root@kafka01 ~] vim /etc/logstash/pipelines.yml
- pipeline.id: msg
path.config: "/etc/logstash/conf.d/messages.conf"
- pipeline.id: nginx
path.config: "/etc/logstash/conf.d/nginx.conf"
#重启logstash
[root@kafka01 ~] systemctl restart logstash
#可以压测一下,再查看日志
[root@kafka01 ~] ab -n 100 -c 100 http://192.168.59.111/index.html
[root@kafka01 ~] tailf /var/log/logstash/logstash-plain.log
5,在浏览器上测试,创建索引
6,创建nginx日志视图
7,添加访问监控
8, 添加ip访问量前10的ip
9,访问量趋势
10,创建仪表盘
没有nginx 索引需注意以下几点:
1:用talif 查看日志,可以看出服务是否生效
2: pipiline.yml 文件
3:nginx log 里要有内容 可以用ab压测用具生成一些日志