搭建ELK-流水账-只记思路

环境准备：

操作系统：CentOS Linux release 7.4.1708 (Core) 虚拟机即可

兼容矩阵

最近软件更新换代很快啊，到底搭建哪一款ELK适合呢。来官网看下兼容矩阵：

支持矩阵 | Elastic

根据兼容矩阵，选择一款适合自己的，由于小编操作系统centos自带了openjdk1.8.因此还需要考虑到是否兼容java1.8的环境。

从兼容矩阵看到，ES8已经放弃了JDK1.8。因此小编选择了ES7.10来作为实验品。

软件列表如下：

elasticsearch-7.10.1-linux-x86_64.tar.gz

kibana-7.10.0-linux-x86_64.tar.gz

扫描二维码关注公众号，回复： 17064588 查看本文章

logstash-7.10.0-linux-x86_64.tar.gz

就选择这3个来搭建ELK环境。

下载地址

从开源社区下载：下载中心 - Elastic 中文社区

ElasticSearch安装

先安装elastic search，赋权给非root用户。我喜欢visudo。简单快捷方便。

ES 有点注意的是，它不能用root用户启动。所以提前建好非root用户。

如下命令暂时记下，需要再ES启动报错执行。偷懒了，记得是需要改些系统配置的。

操作命令

301 2022-03-23 11:37:28 ulimit -S -n
302 2022-03-23 11:37:31 ls /etc/security/limits.conf
303 2022-03-23 11:37:34 vim /etc/security/limits.conf
304 2022-03-23 11:39:14 echo "fs.file-max = 1000000">> /etc/sysctl.conf
305 2022-03-23 11:39:14 echo "net.core.somaxconn = 32768">> /etc/sysctl.conf
306 2022-03-23 11:39:14 echo "net.ipv4.tcp_tw_recycle = 0">> /etc/sysctl.conf
307 2022-03-23 11:39:14 echo "net.ipv4.tcp_syncookies = 0">> /etc/sysctl.conf
308 2022-03-23 11:39:14 echo "vm.overcommit_memory = 1">> /etc/sysctl.conf
309 2022-03-23 11:39:15 sysctl -p
310 2022-03-23 11:39:39 cat >> /etc/systemd/system/cpupower.service << EOF
311 2022-03-26 16:41:00 [Unit]
312 2022-03-26 16:41:00 Description=CPU performance
313 2022-03-26 16:41:00 [Service]
314 2022-03-26 16:41:00 Type=oneshot
315 2022-03-26 16:41:00 ExecStart=/usr/bin/cpupower frequency-set --governor performance
316 2022-03-26 16:41:00 [Install]
317 2022-03-26 16:41:00 WantedBy=multi-user.target
318 2022-03-26 16:41:00 EOF
319 2022-03-23 11:39:44 systemctl daemon-reload
320 2022-03-23 11:39:48 systemctl enable cpupower.service
321 2022-03-23 11:39:54 systemctl start cpupower.service
322 2022-03-23 11:40:14 echo never > /sys/kernel/mm/transparent_hugepage/enabled

323 2022-03-23 11:40:15 echo never > /sys/kernel/mm/transparent_hugepage/defrag
324 2022-03-23 11:40:23 vi /etc/udev/rules.d/60-tidb-schedulers.rules
325 2022-03-23 11:40:42 udevadm control --reload-rules
326 2022-03-23 11:40:43 udevadm trigger --type=devices --action=change
327 2022-03-23 11:40:46 cat >> /etc/systemd/system/cpupower.service << EOF
328 2022-03-26 16:41:00 [Unit]
329 2022-03-26 16:41:00 Description=CPU performance
330 2022-03-26 16:41:00 [Service]
331 2022-03-26 16:41:00 Type=oneshot
332 2022-03-26 16:41:00 ExecStart=/usr/bin/cpupower frequency-set --governor performance
333 2022-03-26 16:41:00 [Install]
334 2022-03-26 16:41:00 WantedBy=multi-user.target
335 2022-03-26 16:41:00 EOF
336 2022-03-23 11:40:50 systemctl daemon-reload
337 2022-03-23 11:40:50 systemctl enable cpupower.service
338 2022-03-23 11:40:51 systemctl start cpupower.service
339 2022-03-23 11:40:55 cat /sys/kernel/mm/transparent_hugepage/enabled
340 2022-03-23 11:40:59 cat /sys/block/sd[bc]/queue/scheduler
341 2022-03-23 11:41:24 vim /etc/security/limits.conf

ES配置

ES配置文件更改：

config文件夹下的jvm.options更改。

-Xms10g
-Xmx10g
来它10个G的内存，否则会很慢。

config文件夹下的elasticsearch.yml,有效配置为：

cluster.name: my-application

node.name: node-1

path.logs: /data/path/to/logs

network.host: 0.0.0.0

http.port: 9200

cluster.initial_master_nodes: ["node-1"]

其他配置暂且搁置，不影响Es使用。启动即可使用。

保证高可用

为了保证一个高可用，用脚本实现下单机版本的高可用

#!/bin/bash
nnn1=$(ps -ef | grep elasticsearch | grep -v grep | grep -v monitor | wc -l)
echo $nnn1

for i in $(ps -ef | grep elasticsearch | grep -v grep | grep -v monitor |awk '{print $2}');do echo $i;done;
ps -ef | grep elasticsearch | grep -v grep | grep -v monitor

if [ $nnn1 -ne 2 ]; then
echo 'wrong,pease restart the elasticsearch'
for i in $(ps -ef | grep elasticsearch | grep -v grep |grep -v monitor |awk '{print $2}');do kill -9 $i;done;
echo "restart now ...."
/data/software/elasticsearch/bin/elasticsearch -d
echo "restart over ....."
fi

在系统定时器里加上

[sysadm@tidb196 config]$ crontab -l
*/2 * * * * sh /data/software/elasticsearch/bin/monitor.sh &

只要ES有变动，两分钟监控一次，然后启动，可以保证ES不会挂掉吧。。。。

哈哈。这样，只要发现es服务器挂，就启动。优点low。

kibana安装

这个更简单了，直接把压缩包解压即可。

有效配置

server.port: 5601

server.host: "0.0.0.0"

server.name: "your-hostname"

elasticsearch.requestTimeout: 60000

i18n.locale: "zh-CN"

然后sbin目录下kiban运行即可。

也用脚本和监控来了下。

#!/bin/bash
kn=$(ss -antlup | grep 5601 | wc -l)
echo $kn

for i in $(ss -antlup | grep 5601 | awk '{print $7}' | awk -F"," '{print $2}' | awk -F"=" '{print $2}');do echo $i;done;

if [ $kn -ne 2 ]; then
echo 'wrong,pease restart the kibana'
for i in $(ss -antlup | grep 5601 | awk '{print $7}' | awk -F"," '{print $2}' | awk -F"=" '{print $2}');do kill -9 $i;done;
echo "kibana restart now ...."
/data/software/kibana/bin/kibana &
echo "kibana restart over ....."
fi

[sysadm@tidb196 bin]$ crontab -l
*/2 * * * * sh /data/software/elasticsearch/bin/monitor.sh &
*/2 * * * * sh /data/software/kibana/bin/monitor.sh &
*/2 * * * * sh /data/software/logstash/bin/monitor.sh &

logstash安装和配置

input {
gelf {
port => 1560
use_tcp => true
}
}

filter {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}" }
}
geoip {
source => "clientip"
}
}

output {
elasticsearch {
hosts => ["http://11.10.32.196:9200"]
index => "logstash-%{service_name}-%{+YYYY-MM-dd}"
}
}

#!/bin/bash
ln=$(ps -ef | grep logstash | grep -v grep | grep -v monitor | wc -l)
echo $ln
if [ $ln -ne 1 ]; then
echo 'wrong,pease restart the logstash'
for i in $(ps -ef | grep logstash | grep -v grep |grep -v monitor |awk '{print $2}');do kill -9 $i;done;
echo "logstash restart now ...."
/data/software/logstash/bin/logstash -f /data/software/logstash/config/logstash.conf --config.reload.automatic &
echo "logstash restart over ....."
fi