一、ELKStack简介
中文指南:https://www.gitbook.com/book/chenryn/elk-stack-guide-cn/details
ELK Stack包含:ElasticSearch、Logstash、Kibana
ElasticSearch是一个搜索引擎,用来搜索、分析、存储日志。它是分布式的,也就是说可以横向扩容,可以自动发现,索引自动分片,总之很强大。
文档介绍: https://www.elastic.co/guide/cn/elasticsearch/guide/current/index.html
Logstash用来采集日志,把日志解析为json格式交给ElasticSearch。
kibana 也是一个开源和免费的工具,Kibana可以为 Logstash 和 ElasticSearch 提供的日志分析友好的 Web 界面,可以帮助您汇总、分析和搜索重要数据日志。
Beats在这里是一个轻量级日志采集器,其实Beats家族有5个成员
早期的ELK架构中使用Logstash收集、解析日志,但是Logstash对内存、cpu、io等资源消耗比较高。相比 Logstash,Beats所占系统的CPU和内存几乎可以忽略不计
x-pack对Elastic Stack提供了安全、警报、监控、报表、图表于一身的扩展包,是收费的。
2、ELK架构:
二、Elasticsearch部署
2.1、安装JDK
2.1.1 yum安装JDK [root@linux-node1 ~]# yum install -y java [root@linux-node1 ~]# java -version openjdk version "1.8.0_151" OpenJDK Runtime Environment (build 1.8.0_151-b12) OpenJDK 64-Bit Server VM (build 25.151-b12, mixed mode) 2.1.2 源码安装JDK 下载 [root@linux-node1 ~]# wget http://download.oracle.com/otn-pub/java/jdk/8u151-b12/e758a0de34e24606bca991d704f6dcbf/jdk-8u151-linux-x64.tar.gz 配置Java环境 [root@linux-node1 ~]# tar zxf jdk-8u151-linux-x64.tar.gz -C /usr/local/ [root@linux-node1 ~]# ln –s /usr/local/jdk1.8.0_91 /usr/local/jdk [root@linux-node1 ~]# vim /etc/profile export JAVA_HOME=/usr/local/jdk export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar export PATH=$PATH:$JAVA_HOME/bin [root@linux-node1 ~]# source /etc/profile [root@linux-node1 ~]# java -version 注:linux-node2节点上也安装JDK
2.2、安装Elasticsearch
(安装版本为为elasticsearch6.X)
(1)源码安装elasticsearch:
(1)源码安装elasticsearch: 安装elasticsearch [root@linux-node1 ~]# wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.0.0.rpm [root@linux-node1 ~]# yum install -y elasticsearch-6.0.0.rpm 配置elasticsearch,linux-node2配置一个相同的节点,通过组播进行通信,会通过cluster进行查找,如果无法通过组播查询,修改成单播即可。 [root@linux-node1 ~]# vim /etc/elasticsearch/elasticsearch.yml cluster.name:elk-cluster #集群名称 node.name:elk-node1 #节点名称,一个集群之内节点的名称不能重复 path.data:/data/elkdata #数据路径 path.logs:/data/logs #日志路径 #bootstrap.memory_lock:true #锁住es内存,保证内存不分配至交换分区,这里暂时不配置。注释掉 network.host:192.168.56.11 #网络监听地址 http.port:9200 #用户访问查看的端口,9300是组件访问使用 discovery.zen,ping.unicast.hosts:["192.168.56.11","192.168.56.12"] #单播(配置一台即可,生产可以使用组播方式) [root@linux-node1 ~]# mkdir /data/{elkdata,logs} [root@linux-node1 ~]# chown elasticsearch.elasticsearch /data -R [root@linux-node1 ~]# systemctl start elasticsearch.service [root@linux-node1 ~]# netstat -tulnp |grep java tcp6 0 0 192.168.56.11:9200 :::* LISTEN 26866/java tcp6 0 0 192.168.56.11:9300 :::* LISTEN 26866/java 将配置文件拷贝到linux-node2 [root@linux-node1 ~]# scp /etc/elasticsearch/elasticsearch.yml 192.168.56.12:/etc/elasticsearch/ [root@linux-node2 ~]# vim /etc/elasticsearch/elasticsearch.yml 修改: node.name=elk-node2 network.host=192.168.56.12 [root@linux-node2 ~]# mkdir /data/{elkdata,logs} [root@linux-node2 ~]# chown elasticsearch.elasticsearch /data -R [root@linux-node2 ~]# systemctl start elasticsearch.service [root@linux-node2 ~]# netstat -tulnp |grep java tcp6 0 0 192.168.56.12:9200 :::* LISTEN 16346/java tcp6 0 0 192.168.56.12:9300 :::* LISTEN 16346/java
(2)yum安装elasticsearch
1.下载并安装GPG key [root@linux-node1 ~]# rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch 2.添加yum仓库 [root@linux-node1 ~]# vim /etc/yum.repos.d/es.repo [elasticsearch-6.x] name=Elasticsearch repository for 6.x packages baseurl=https://artifacts.elastic.co/packages/6.x/yum gpgcheck=1 gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch enabled=1 autorefresh=1 type=rpm-md 3.安装elasticsearch [root@hadoop-node1 ~]# yum install -y elasticsearch
三、Elasticsearch的集群配置和监控
可以使用命令来查看elasticsearch的状态内容
[root@linux-node1 ~]# curl http://192.168.56.11:9200/_cluster/health?pretty=true { "cluster_name" : "elk-cluster", "status" : "green", "timed_out" : false, "number_of_nodes" : 2, "number_of_data_nodes" : 2, "active_primary_shards" : 0, "active_shards" : 0, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 0, "delayed_unassigned_shards" : 0, "number_of_pending_tasks" : 0, "number_of_in_flight_fetch" : 0, "task_max_waiting_in_queue_millis" : 0, "active_shards_percent_as_number" : 100.0 } [root@linux-node2 ~]# curl http://192.168.56.12:9200/_cluster/health?pretty=true { "cluster_name" : "elk-cluster", "status" : "green", "timed_out" : false, "number_of_nodes" : 2, "number_of_data_nodes" : 2, "active_primary_shards" : 0, "active_shards" : 0, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 0, "delayed_unassigned_shards" : 0, "number_of_pending_tasks" : 0, "number_of_in_flight_fetch" : 0, "task_max_waiting_in_queue_millis" : 0, "active_shards_percent_as_number" : 100.0 }