架构：
APP Server：
IP:192.168.1.2
安装软件:Kibana &logstash
存储服务器1：
IP:192.168.1.3
安装软件:KElasticsearch
存储服务器2：
IP:192.168.1.4
安装软件:KElasticsearch
存储服务器3：
IP:192.168.1.5
安装软件:KElasticsearch
1.安装Java #所有服务器安装
yum -y install java-1.8.0-openjdk*
vim /etc/profile
export JAVA_HOME=/usr/lib/jvm/jre-1.8.0-openjdk.x86_64
export CLASSPATH=.:$JAVA_HOME/jre/lib/rt.jar:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
export PATH=$PATH:$JAVA_HOME/bin
2.修改主机名
hostnamectl set-hostname elk
3.elasticsearch #存储服务器安装
rpm --import
https://artifacts.elastic.co/GPG-KEY-elasticsearch
elasticsearch client端 192.168.1.3 （用作负载均衡，不是必须）
elasticsearch.yml配置
# cat /etc/elasticsearch/elasticsearch.yml |grep -v "#"
cluster.name: elk-group
node.name: elk-1
node.master: true
node.data: false
node.attr.rack: r1
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 0.0.0.0
http.port: 9200
transport.tcp.port: 9300
http.cors.enabled: true
discovery.zen.ping.unicast.hosts: ["192.168.1.4"]
discovery.zen.minimum_master_nodes: 2
cluster.routing.allocation.same_shard.host: true
elasticsearch master端 192.168.1.4
elasticsearch.yml配置
cluster.name: elk-group
node.name: elk-2
node.master: true
node.data: true
node.attr.rack: r1
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 0.0.0.0
http.port: 9200
transport.tcp.port: 9300
http.cors.enabled: true
discovery.zen.ping.unicast.hosts: ["
192.168.1.4"]
discovery.zen.minimum_master_nodes: 2
cluster.routing.allocation.same_shard.host: true
elasticsearch salve端 192.168.1.5 （可多台）
elasticsearch.yml配置
cluster.name: elk-group
node.name: elk-3
node.master: false
node.data: true
node.attr.rack: r1
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 0.0.0.0
http.port: 9200
transport.tcp.port: 9300
http.cors.enabled: true
discovery.zen.ping.unicast.hosts: ["192.168.1.4"]
discovery.zen.minimum_master_nodes: 2
cluster.routing.allocation.same_shard.host: true
设置
ulimit数量和线程
vi /etc/security/limits.conf
elasticsearch
-
nofile
65536
elasticsearch - nproc 2048
Ubuntu与
limits.conf
Ubuntu在
init.d
启动时忽略了
limits.conf
文件。要开启
limits.conf
文件，需要编辑
/etc/pam.d/su
，删除下面行的注释：
# session required pam_limits.so
禁止
swap
vi /etc/fstab
#/dev/mapper/cryptswap1 none swap sw 0 0
设置JVM堆大小
vi /etc/elasticsearch/jvm.options
-Xms2g
-Xmx2g
4Kibana
rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
yum install -y kibana
service kibana start
chkconfig --add kibana
5.安装logstash
vi /etc/yum.repos.d/logstash.repo
[logstash-6.x]
name=Elastic repository for 6.x packages
baseurl=https://artifacts.elastic.co/packages/6.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
yum install logstash -y
[root@elk logstash]# cat /etc/logstash/logstash.yml |grep -v "#"
node.name: elk
path.data: /var/lib/logstash
path.config: /etc/logstash/conf.d/*.conf
http.host: "0.0.0.0"
http.port: 9600
path.logs: /var/log/logstash
[root@elk logstash]# cat /etc/logstash/conf.d/filter.conf
input {
beats {
type => log
port => 5044
}
}
output {
elasticsearch {
hosts =>["192.168.1.3:9200"]
}
stdout { codec =>rubydebug }
}
service logstash start
6。安装x-pack
下载
https://artifacts.elastic.co/downloads/packs/x-pack/x-pack-6.0.0.zip
/usr/share/elasticsearch/bin/elasticsearch-plugin install file:///root/x-pack-6.0.0.zip
6.安装filebeat
curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.0.0-x86_64.rpm
rpm -vi filebeat-6.0.0-x86_64.rpm
cat /etc/filebeat/filebeat.yml|grep -v "#"
filebeat.prospectors:
- type: log
enabled: true
paths:
- /var/log/*.log
- /var/log/zabbix/zabbix_server.log
filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: false
setup.template.settings:
index.number_of_shards: 3
output.logstash:
hosts: ["192.168.1.2:5044"]
/etc/init.d/filebeat start
7.安装winlogbeat
1.解压winlogbeat
2.放置到C:\winlogbeat
3.进入powershell
Set-ExecutionPolicy Unrestricted
PS C:\Windows\system32> cd C:\winlogbeat
PS C:\winlogbeat> .\install-service-winlogbeat.ps1
问题解决：
1.log出现 Caused by: java.lang.IllegalStateException: Failed to create node environment
目录没权限
cd /var/lib/elasticsearch/
chown elasticsearch:elasticsearch -R elasticsearch
2.问题2
[2017-11-23T22:51:17,333][ERROR][o.e.b.Bootstrap ] Exception
java.lang.NullPointerException: null
at sun.nio.fs.UnixFileSystem.getPath(UnixFileSystem.java:273) ~[?:?]
at org.elasticsearch.common.io.PathUtils.get(PathUtils.java:60) ~[elasticsearch-6.0.0.jar:6.0.0]
mount -t cgroup -o rw,nosuid,nodev,noexec,relatime,cpu,cpuacct cgroup /sys/fs/cgroup/cpu,cpuacct
vi /etc/rc.local
mount -t cgroup -o rw,nosuid,nodev,noexec,relatime,cpu,cpuacct cgroup /sys/fs/cgroup/cpu,cpuacct
9.x-pack
cd /usr/share/elasticsearch/bin/
./elasticsearch-plugin install file:///root/x-pack-6.0.0.zip
cd /usr/share/kibana/bin/
./kibana-plugin install file:///root/x-pack-6.0.0.zip
cd /usr/share/logstash/bin/
./logstash-plugin install file:///root/x-pack-6.0.0.zip
service logstash restart
service elasticsearch restart
service kibana restart
/usr/share/elasticsearch/bin/x-pack/setup-passwords interactive 重置所有密码
xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.url:
http://127.0.0.1:920
0
vi /etc/logstash/logstash.yml
10.优化
vi /etc/sysctl.conf
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_tw_recycle = 1
net.ipv4.tcp_fin_timeout = 30
net.ipv4.tcp_keepalive_time = 360
net.ipv4.ip_local_port_range = 1024 65000
net.ipv4.tcp_max_syn_backlog = 8192
net.ipv4.tcp_max_tw_buckets = 5000
/sbin/sysctl -p
11.安装head