ELK是Elasticsearch、Logstash、Kibana的简称
Elasticsearch是实时全文搜索和分析引擎，提供搜集、分析、存储数据三大功能
Logstash是一个用来搜集、分析、过滤日志的工具。
Kibana是一个基于Web的图形界面，用于搜索、分析和可视化存储在 Elasticsearch指标中的日志数据。
测试环境为 192.168.0.101 部署Elasticsearch、Logstash、Kibana、nginx ，收集nginx 日志
192.168.0.63 部署Logstash 、httpd 收集httpd 日志
下载ELK ，官网地址为https://www.elastic.co/downloads/past-releases
1、 192.168.0.101主机安装 java, yum install java-1.8.0-openjdk-devel -y，参照官网安装ELK ,https://www.elastic.co/guide/en/elasticsearch/reference/5.x/rpm.html
[root@els ~]# cat /etc/elasticsearch/jvm.options
-Xms1g
-Xmx1g
按主机大小修改下内存
[root@els ~]# cat /etc/elasticsearch/elasticsearch.yml |grep -v "#"
cluster.name: my-application
node.name: els
path.data: /els/data
path.logs: /els/logs
network.host: 192.168.0.101
http.cors.enabled: true
http.cors.allow-origin: "*"
修改下elasticsearch 配置文件，启动服务systemctl start elasticsearch.service
2、安装head 插件 ，官网有安装步骤https://github.com/mobz/elasticsearch-head
[root@els elasticsearch-head]# npm run start& 启动插件
3、下载对应logstash rpm 包 https://www.elastic.co/cn/downloads/logstash
rpm -ivh logstash-5.*.*.rpm 安装对应的包
[root@els conf.d]# cd /etc/logstash/conf.d/
[root@els conf.d]# cat full.conf
input {
file {
path => "/var/log/messages"
type => "system"
start_position => "beginning"
}
file {
path => "/var/log/secure"
type => "secure"
start_position => "beginning"
}
file {
path => "/var/log/nginx/elk.access.log"
type => "nginx"
start_position => "beginning"
}
}
output {
if [type] == "system" {
elasticsearch {
hosts => ["192.168.0.101:9200"]
index => "ma-system-%{+YYYY.MM.dd}"
}
}
if [type] == "secure" {
elasticsearch {
hosts => ["192.168.0.101:9200"]
index => "ma-secure-%{+YYYY.MM.dd}"
}
}
if [type] == "nginx" {
elasticsearch {
hosts => ["192.168.0.101:9200"]
index => "ma-nginx-%{+YYYY.MM.dd}"
}
}
}
修改nginx 日志格式
http {
log_format json '{"@timestamp":"$time_iso8601",'
'"@version":"1",'
'"client":"$remote_addr",'
'"url":"$uri",'
'"status":"$status",'
'"domian":"$host",'
'"host":"$server_addr",'
'"size":"$body_bytes_sent",'
'"responsetime":"$request_time",'
'"referer":"$http_referer",'
'"ua":"$http_user_agent"'
'}';
access_log /var/log/nginx/elk.access.log json;
[root@els conf.d]# logstash -f full.conf 运行logstash
4、官网下载kibana包, rpm -ivh kibana-5.5.1-x86_64.rpm
[root@els conf.d]# cat /etc/kibana/kibana.yml | grep -v "#"
server.port: 5601server.host: "0.0.0.0"
elasticsearch.url: "http://192.168.0.101:9200"
kibana.index: ".kibana"
systemctl start kibana.service,
5、192.168.0.63上同上面步骤安装java,logstash ，httpd
[root@filebeat conf.d]# cd /etc/logstash/conf.d/
[root@filebeat conf.d]# cat httpd.conf
input {
file {
path => "/var/log/httpd/access_log"
type => "http"
start_position => "beginning"
}
}
output {
if [type] == "http" {
elasticsearch {
hosts => ["192.168.0.101:9200"]
index => "ma-http-%{+YYYY.MM.dd}"
}
}
}
修改httpd 日志格式
LogFormat "{ \
\"@timestamp\": \"%{%Y-%m-%dT%H:%M:%S%z}t\", \
\"@version\": \"1\", \
\"tags\":[\"apache\"], \
\"message\": \"%h %l %u %t \\\"%r\\\" %>s %b\", \
\"clientip\": \"%a\", \
\"duration\": %D, \
\"status\": %>s, \
\"request\": \"%U%q\", \
\"urlpath\": \"%U\", \
\"urlquery\": \"%q\", \
\"bytes\": %B, \
\"method\": \"%m\", \
\"site\": \"%{Host}i\", \
\"referer\": \"%{Referer}i\", \
\"useragent\": \"%{User-agent}i\" \
}" apache_json
修改输出格式为上面定义的json格式
CustomLog logs/access_log apache_json
查看httpd日志已经输出到Elasticsearch
