Linux——ELK日志收集（Apache）

简介（Elasticsearch）

​ Elasticsearch是开源分布式搜索引擎，它的特点有：分布式，零配置，自动发现，索引自动分片，索引副本机制，restful 风格接口，多数据源，自动搜索负载等！

官方网站： https://www.elastic.co

官方参考文档：https://www.elastic.co/guide/en/elasticsearch/reference/6.6/setup-configuration-memory.html

下载地址： https://mirrors.tuna.tsinghua.edu.cn/elasticstack/6.x/yum/6.6.0/

环境

主机IP	主机名
192.168.1.20	locathost
192.168.1.19	httpd

服务搭建（elasticsearch）

1.下载并安装elasticsearch包

[root@localhost ~]# yum -y install   https://mirrors.tuna.tsinghua.edu.cn/elasticstack/6.x/yum/6.6.0/elasticsearch-6.6.0.rpm
已加载插件：fastestmirror, langpacks
elasticsearch-6.6.0.rpm              | 109 MB     00:14     
......

2.创建数据目录，并修改权限

[root@localhost ~]# mkdir -p /data/elasticsearch
[root@localhost ~]# chown -R elasticsearch.elasticsearch /data/elasticsearch/

3.修改服务配置文件

[root@localhost ~]# vim /etc/elasticsearch/elasticsearch.yml
更改：
node.name: node-1				#群集中本机节点名
path.data: /data/elasticsearch		#数据目录
path.logs: /var/log/elasticsearch	#日志目录
bootstrap.memory_lock: true	#锁定内存，需要和/etc/elasticsearch/jvm.options关联
network.host: 192.168.1.20,127.0.0.1  #监听的ip地址
http.port: 9200	 #端口号
如果是节点服务器的话，请把以下俩项也配置：
discovery.zen.ping.unicast.hosts: ["主服务器的IP", "本机的IP"]
discovery.zen.minimum_master_nodes: 2   

4.分配锁定内存

[root@localhost ~]# vim /etc/elasticsearch/jvm.options 
查找并修改
-Xms1g		#分配最小内存	
-Xmx1g		#分配最大内存,官方推荐为物理内存的一半，但最大为32G

5.重启服务

[root@localhost ~]# systemctl  restart elasticsearch.service 
[root@localhost ~]# systemctl status  elasticsearch.service 
● elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; disabled; vendor preset: disabled)
   Active: active (running) since 四 2020-07-30 14:55:27 CST; 17s ago
     Docs: http://www.elastic.co
 Main PID: 57470 (java)
   CGroup: /system.slice/elasticsearch.service

7月 30 14:55:27 localhost.localdomain systemd[1]: Starte...
7月 30 14:55:27 localhost.localdomain systemd[1]: Starti...
Hint: Some lines were ellipsized, use -l to show in full.

修改锁定内存后，有时候可能无法重启，解决方法如下：

[root@localhost ~]# systemctl edit elasticsearch.service 
添加：
[Service]
LimitMEMLOCK=infinity
最后按F2保存退出
[root@localhost ~]# systemctl daemon-reload 
[root@localhost ~]# systemctl restart elasticsearch.service

6.检查

（1）查看单主机

http://192.168.1.20:9200/

（2）查看群集健康状态

http://192.168.1.20:9200/_cluster/health?pretty

（3）查看整个群集状态信息

http://192.168.1.20:9200/_cluster/state?pretty

常见群集管理监控命令

（1）创建索引

选项说明：

• XPUT 创建
• XDELETE 删除

[root@node-1 ~]# curl -XPUT '192.168.1.20:9200/vipinfo/users/1?pretty&pretty' -H 'Content-Type: application/json' -d '{"name": "guofucheng","age": "45","job": "mingxing"}'
{
  "_index" : "vipinfo",
  "_type" : "users",
  "_id" : "1",
  "_version" : 1,
  "result" : "created",
  "_shards" : {
    "total" : 2,
    "successful" : 1,
    "failed" : 0
  },
  "_seq_no" : 0,
  "_primary_term" : 1
}

（2）查看索引信息

[root@node-1 ~]# curl -XGET '192.168.1.20:9200/_cat/indices?pretty'
green open index1  Nl2se1FqQAW_UIeK7M4yGQ 3 2 0 0    2kb  690b
green open vipinfo mzPDLS0ZSm-862uJ-JWEvg 5 1 1 0 10.9kb 5.4kb

（3）查看群集健康状态

[root@node-1 ~]# curl -XGET '192.168.1.20:9200/_cluster/health?pretty'
{
  "cluster_name" : "elasticsearch",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 3,
  "number_of_data_nodes" : 3,
  "active_primary_shards" : 0,
  "active_shards" : 0,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

（4）统计群集节点

[root@node-1 ~]# curl -XGET '192.168.1.20:9200/_cat/nodes?human&pretty'
192.168.1.20 13 96 1 0.00 0.05 0.13 mdi * node-1
192.168.1.19 21 96 0 0.00 0.01 0.12 mdi - node-2
192.168.1.11 19 96 1 0.00 0.04 0.05 mdi - node-3

（5）查看群集所有节点详细信息

[root@node-1 ~]# curl -XGET '192.168.1.20:9200/_nodes/_all/info/jvm.process?human&pretty'
{
  "_nodes" : {
    "total" : 3,
    "successful" : 3,
    "failed" : 0
  },
  "cluster_name" : "elasticsearch",
  "nodes" : {
    "8crfmI3QSB6p9e9GL7rBNA" : {
      "name" : "node-3",
      "transport_address" : "192.168.1.11:9300",
      "host" : "192.168.1.11",
      "ip" : "192.168.1.11",
      "version" : "6.6.0",
      "build_flavor" : "default",
      "build_type" : "rpm",
      "build_hash" : "a9861f4",
      "roles" : [
        "master",
        "data",
        "ingest"
      ],
      "attributes" : {
        "ml.machine_memory" : "1907970048",
        "ml.max_open_jobs" : "20",
        "xpack.installed" : "true",
        "ml.enabled" : "true"
      }
    },
    "aqvOxpEnQ7Koe9Q62Q9o9w" : {
      "name" : "node-1",
      "transport_address" : "192.168.1.20:9300",
      "host" : "192.168.1.20",
      "ip" : "192.168.1.20",
      "version" : "6.6.0",
      "build_flavor" : "default",
      "build_type" : "rpm",
      "build_hash" : "a9861f4",
      "roles" : [
        "master",
        "data",
        "ingest"
      ],
      "attributes" : {
        "ml.machine_memory" : "1912107008",
        "xpack.installed" : "true",
        "ml.max_open_jobs" : "20",
        "ml.enabled" : "true"
      }
    },
    "pkZM81gYS2up4J3fpQkjVQ" : {
      "name" : "node-2",
      "transport_address" : "192.168.1.19:9300",
      "host" : "192.168.1.19",
      "ip" : "192.168.1.19",
      "version" : "6.6.0",
      "build_flavor" : "default",
      "build_type" : "rpm",
      "build_hash" : "a9861f4",
      "roles" : [
        "master",
        "data",
        "ingest"
      ],
      "attributes" : {
        "ml.machine_memory" : "1907970048",
        "ml.max_open_jobs" : "20",
        "xpack.installed" : "true",
        "ml.enabled" : "true"
      }
    }
  }

安装插件（elasticsearch-head）

elasticsearch-head插件提供 一个索引树预览和创建索引树工具，还可以 运行一些 命令 ，如定义 命令的别名和删除 索引。

elasticsearch群集：

状态颜色：

颜色	解释
灰色	未连接
绿色	数据完整态
黄色	副本不完整
红色	数据分片不完整
紫色	数据分片复制过程

群集主机角色：

• 主节点master：负责管理调度
• 工作节点： 负责处理数据
• 默认情况，所有节点都是工作节点，即主节点也处理数据

1.安装node环境

官网: https://nodejs.org/en/download/

（1）下载node包

[root@localhost ~]# wget https://nodejs.org/dist/v8.11.3/node-v8.11.3-linux-x64.tar.xz
--2020-07-30 15:14:23--  https://nodejs.org/dist/v8.11.3/node-v8.11.3-linux-x64.tar.xz
正在解析主机 nodejs.org (nodejs.org)... 104.20.22.46, 104.20.23.46, 2606:4700:10::6814:162e, ...
正在连接 nodejs.org (nodejs.org)|104.20.22.46|:443... 已连接。
已发出 HTTP 请求，正在等待回应... 200 OK
长度：11351132 (11M) [application/x-xz]
正在保存至: “node-v8.11.3-linux-x64.tar.xz”

100%[==================>] 11,351,132  37.7KB/s 用时 84s    
2020-07-30 15:15:54 (133 KB/s) - 已保存 “node-v8.11.3-linux-x64.tar.xz” [11351132/11351132])
[root@localhost ~]# tar -xvf node-v8.11.3-linux-x64.tar.xz 

（2）进入解压后的目录 bin 目录下，执行 ls 会看到两个文件 node,npm. 然后执行./node -v ，如果显示出 版本号说明我们下载的程序包是没有问题的。

[root@localhost ~]# cd node-v8.11.3-linux-x64/bin/
[root@localhost bin]# ls
node  npm  npx
[root@localhost bin]# ./node -v
v8.11.3

（3）配置环境变量

[root@localhost ~]# vim /etc/profile
添加：
export PATH=/usr/local/node/node-v8.11.3-linux-x64/bin:$PATH
[root@localhost ~]# source  /etc/profile    //立即生效变量

2.安装grunt

PS： 运行head需要借助grunt命令！

在Node安装目录执行以下命令即可安装grunt工具

[root@localhost bin]# npm install -g grunt-cli
npm WARN deprecated [email protected]: https://github.com/lydell/resolve-url#deprecated
npm WARN deprecated [email protected]: Please see https://github.com/lydell/urix#deprecated
/usr/bin/grunt -> /usr/lib/node_modules/grunt-cli/bin/grunt
/usr/lib
└─┬ [email protected] 
  ├── [email protected] 
  ├── [email protected] 
  ├─┬ [email protected] 
  │ ├── [email protected] 
......

3.安装head插件

插件github地址 ：https://github.com/mobz/elasticsearch-head

（1）下载包

[root@localhost ~]# git clone https://github.com/mobz/elasticsearch-head
正克隆到 'elasticsearch-head'...
remote: Enumerating objects: 10, done.
remote: Counting objects: 100% (10/10), done.
remote: Compressing objects: 100% (10/10), done.
remote: Total 4347 (delta 0), reused 3 (delta 0), pack-reused 4337
接收对象中: 100% (4347/4347), 2.49 MiB | 24.00 KiB/s, done.
处理 delta 中: 100% (2417/2417), done.
[root@localhost ~]# cd elasticsearch-head/
[root@localhost elasticsearch-head]# npm install 
......
此段时间过长 请耐心等待！

（2） 修改Elasticsearch主配置文件

[root@localhost elasticsearch-head]# vim /etc/elasticsearch/elasticsearch.yml 
http.cors.enabled: true     
#开启跨域访问支持
http.cors.allow-origin: "*"
#跨域访问允许的域名地址

（3）重启服务

[root@localhost elasticsearch-head]# systemctl restart elasticsearch.service  
[root@localhost elasticsearch-head]# npm  run  start &   // 前台启动，一旦关闭终端。，服务器自动关闭 ，所有此刻选择后台开启
[1] 9436
[root@localhost elasticsearch-head]# [..................] - : info using node@v6
> [email protected] start /root/elasticsearch-head
> grunt server

>> Local Npm module "grunt-contrib-jasmine" not found. Is it installed?

Running "connect:server" (connect) task
Waiting forever...
Started connect web server on http://localhost:9100

[root@localhost elasticsearch-head]# netstat -lnupt | grep  9100
tcp        0      0 0.0.0.0:9100            0.0.0.0:*               LISTEN      9449/grunt          
[root@localhost elasticsearch-head]# netstat -lnupt | grep  9200
tcp6       0      0 192.168.1.20:9200       :::*                    LISTEN      9232/java           
tcp6       0      0 127.0.0.1:9200          :::*                    LISTEN      9232/java           

（4）检查

4.验证

（1）插入一条索引

[root@localhost ~]# curl -X PUT 192.168.1.20:9200/index1  -H 'Content-Type: application/json' -d '{
>     "settings" : {
>         "index" : {
>             "number_of_shards" : 3, 
>             "number_of_replicas" : 2 
>         }
>     }
> }'
{"acknowledged":true,"shards_acknowledged":true,"index":"index1"}

（2）浏览器查看

安装插件（kibana）

（1）下载kibana包。

[root@localhost ~]# yum -y install https://mirrors.tuna.tsinghua.edu.cn/elasticstack/6.x/yum/6.6.0/kibana-6.6.0-x86_64.rpm
已加载插件：fastestmirror, langpacks
kibana-6.6.0-x86_64.rpm                                  | 177 MB     00:07     
正在检查 /var/tmp/yum-root-1aqueq/kibana-6.6.0-x86_64.rpm: kibana-6.6.0-1.x86_64
/var/tmp/yum-root-1aqueq/kibana-6.6.0-x86_64.rpm 将被安装
正在解决依赖关系
--> 正在检查事务
---> 软件包 kibana.x86_64.0.6.6.0-1 将被 安装
--> 解决依赖关系完成
......

（2）修改kibana的配置文件

[root@localhost ~]# vim /etc/kibana/kibana.yml 
[root@localhost ~]# grep -v '#' /etc/kibana/kibana.yml | grep -v '^$' 
server.port: 5601
#Kibana打开的端口
server.host: "0.0.0.0"
#Kibana侦听的地址
elasticsearch.url: "http://192.168.1.20:9200"
#和Elasticsearch建立连接
kibana.index: ".kibana"
#在Elasticsearch中添加。Kibana索引
[root@localhost ~]# systemctl start kibana.service //开启服务

（3）验证

安装插件（logstash）

（1）下载logstash包。

[root@localhost ~]# yum -y install https://mirrors.tuna.tsinghua.edu.cn/elasticstack/6.x/yum/6.6.0/logstash-6.6.0.rpm
已加载插件：fastestmirror, langpacks
logstash-6.6.0.rpm                                       | 163 MB     00:23     
正在检查 /var/tmp/yum-root-1aqueq/logstash-6.6.0.rpm: 1:logstash-6.6.0-1.noarch
/var/tmp/yum-root-1aqueq/logstash-6.6.0.rpm 将被安装
正在解决依赖关系
--> 正在检查事务
---> 软件包 logstash.noarch.1.6.6.0-1 将被 安装
--> 解决依赖关系完成
......
[root@localhost ~]# systemctl daemon-reload 
[root@localhost ~]# systemctl start  logstash.service 
[root@localhost ~]# systemctl enable  logstash.service 
Created symlink from /etc/systemd/system/multi-user.target.wants/logstash.service to /etc/systemd/system/logstash.service.

（2）安装httpd服务

[root@localhost ~]# yum -y install httpd
已加载插件：fastestmirror, langpacks
Loading mirror speeds from cached hostfile
 * base: mirrors.bfsu.edu.cn
 * extras: mirrors.bfsu.edu.cn
 * updates: mirrors.njupt.edu.cn
......
[root@localhost ~]# systemctl start httpd
[root@localhost ~]# systemctl enable  httpd
Created symlink from /etc/systemd/system/multi-user.target.wants/httpd.service to /usr/lib/systemd/system/httpd.service.

(3)编写logstash配置文件

[root@efk html]# cd /etc/logstash/conf.d/
[root@efk conf.d]# touch apache_log.conf
[root@efk conf.d]# vim apache_log.conf
input{
	file{
		path => "/etc/httpd/logs/access_log" #收集APache访问日志
		type => "access" #类型指定为acces
		start_position => "beginning" #从开始处收集
}
	file{
		path => "/etc/httpd/logs/error_log" #收集Apache错误日志
		type => "error"  #类型指定为error
		start_position => "beginning"  #从开始处收集
}
}

output{
	if[type] == "access"{  # 如果类型为access，即Apace访问日志输出到elasticsearch
		elasticsearch{
			hosts => ["192.168.1.19:9200"]  #elasticsearch监听地址及端口
			index => "apache_access-%{+YYYY.MM.dd}" #指定索引格式
}
}
	if[type] == "error"{ # 如果类型为error，即Apace访问日志输出到elasticsearch
		elasticsearch{ 
			hosts => ["192.168.1.19:9200"] #elasticsearch监听地址及端口
             index => "apache_access-%{+YYYY.MM.dd}" #指定索引格式
}
}
}
[root@efk conf]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/apache_log.conf 

WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[INFO ] 2020-08-03 17:50:10.596 [main] writabledirectory - Creating directory {:setting=>"path.queue", :path=>"/usr/share/logstash/data/queue"}
[INFO ] 2020-08-03 17:50:10.605 [main] writabledirectory - Creating directory {:setting=>"path.dead_letter_queue", :path=>"/usr/share/logstash/data/dead_letter_queue"}
.......

（4）在es页面查看索引

（5）在kinbana页面创建apache索引