破解x-pack+ELK优化

6.2.3破解：https://www.jianshu.com/p/6acfeabb44f8

主要方法：更改LicenseVerifier.class和 XPackBuild.class（6.0版本不需要）

1. 修改java证书代码 ---- 在Windows本地上操作

Windows需要工具：luyten-0.5.0（java代码编辑器）

/opt/apps/elk/elasticsearch/plugins/x-pack/x-pack-core/x-pack-core-6.2.3.jar  

将x-pack-core-6.2.3.jar包下载到Windows本地，用luyten打开，导出LicenseVerifier.class和 XPackBuild.class分别更改代码

1. LicenseVerifier.class

org.elasticresearch.license----》LicenseVerifier.class : 将此文件另外到桌面（另外后文件名自动为LicenseVerifier.java）

用notepad 编辑LicenseVerifier.java，内容如下：

package org.elasticsearch.license;

import java.nio.*;

import java.util.*;

import java.security.*;

import org.elasticsearch.common.xcontent.*;

import org.apache.lucene.util.*;

import org.elasticsearch.common.io.*;

import java.io.*;

public class LicenseVerifier

{

    public static boolean verifyLicense(final License license, final byte[] encryptedPublicKeyData) {

        return true;

    }

    public static boolean verifyLicense(final License license) {

        return true;

    }

}

1. org.elasticsearch.xpack.core----》XPackBuild.class： 将此文件另外到桌面（另外后文件名自动为XPackBuild.java）

用notepad 编辑XPackBuild.java，内容如下：

package org.elasticsearch.xpack.core;

 

import org.elasticsearch.common.io.*;

 

import java.net.*;

 

import org.elasticsearch.common.*;

import java.nio.file.*;

 

import java.io.*;

 

import java.util.jar.*;

 

public class XPackBuild

 

{

 

    public static final XPackBuild CURRENT;

 

    private String shortHash;

 

    private String date;

 

    @SuppressForbidden(reason = "looks up path of xpack.jar directly")

 

    static Path getElasticsearchCodebase() {

        final URL url = XPackBuild.class.getProtectionDomain().getCodeSource().getLocation();

        try {

            return PathUtils.get(url.toURI());

        }

        catch (URISyntaxException bogus) {

            throw new RuntimeException(bogus);

        }

    }

    XPackBuild(final String shortHash, final String date) {

 

        this.shortHash = shortHash;

 

        this.date = date;

    }

    public String shortHash() {

 

        return this.shortHash;

 

    }

    public String date() {

 

        return this.date;

    }

    static {

 

        final Path path = getElasticsearchCodebase();

 

        String shortHash = null;

 

        String date = null;

 

        Label_0157: {

 

            shortHash = "Unknown";

 

            date = "Unknown";

 

        }

        CURRENT = new XPackBuild(shortHash, date);

 

    }

}

 

1. 更新java证书代码    ----- 在elk服务器上操作

将刚才在Windows本地修改的LicenseVerifier.java和XPackBuild.java上传到服务器/opt/src/bianyi，然后进行编译，并重新打包jar包

1. 编译LicenseVerifier.java和XPackBuild.java

[swadmin@localhost ~]$ cd /opt/src/bianyi/ && ls

 LicenseVerifier.java  XPackBuild.java

[swadmin@localhost bianyi]$  javac -cp "/opt/apps/elk/elasticsearch/lib/elasticsearch-6.2.3.jar:/opt/apps/elk/elasticsearch/lib/lucene-core-7.2.1.jar:/opt/apps/elk/elasticsearch/plugins/x-pack/x-pack-core/x-pack-core-6.2.3.jar"  LicenseVerifier.java

[swadmin@localhost bianyi]$ javac -cp "/opt/apps/elk/elasticsearch/lib/elasticsearch-core-6.2.3.jar:/opt/apps/elk/elasticsearch/lib/lucene-core-7.2.1.jar:/opt/apps/elk/elasticsearch/plugins/x-pack/x-pack-core/x-pack-core-6.2.3.jar"  XPackBuild.jav

1. 重新打包jar包

[swadmin@localhost bianyi]$ mkdir /opt/src/test/  &&  cd /opt/src/test/

[swadmin@localhost test]$ cp /opt/apps/elk/elasticsearch/plugins/x-pack/x-pack-core/x-pack-core-6.2.3.jar /opt/src/test/

[swadmin@localhost test]$ jar -xvf x-pack-core-6.2.3.jar &&  rm x-pack-core-6.2.3.jar

[swadmin@localhost test]$ cp /opt/src/bianyi/LicenseVerifier.class  org/elasticsearch/license/

[swadmin@localhost test]$ cp /opt/src/bianyi/XPackBuild.class org/elasticsearch/xpack/core/

[swadmin@localhost test]$ jar -cvf  x-pack-core-6.2.3.jar ./*

1. 将重新打包的jar包替换之前的旧x-pack-core-6.2.3.jar包

[swadmin@localhost test]$ cp x-pack-core-6.2.3.jar /opt/apps/elk/elasticsearch/plugins/x-pack/x-pack-core/

1. 重新认证

1. 申请一个免费license

下载链接：https://license.elastic.co/registration/download/96d3e8ef-49e7-4e6b-930a-47f695440def

将下载好的文件max-mac-e8e90718-9333-49b6-817a-9098c35c0920-v5.json传至elk服务器/opt/src/bianyi

1. 修改license文件

[swadmin@localhost bianyi]$ cd /opt/src/bianyi/  &&  vim max-mac-e8e90718-9333-49b6-817a-9098c35c0920-v5.json

"type":"platinum"

"expiry_date_in_millis":2524579200999

"max_nodes":1000

1. 禁用x-pack 配置license有效期

更新License文件，因为elasticsearch 6.0.0，现在更新license文件时，要么配置SSL\TLS，要么就禁用security。

更新之前先配置elasticsearch.yml，加入：

[swadmin@localhost ~]$ vim /opt/apps/elk/elasticsearch/config/elasticsearch.yml

xpack.security.enabled: false

1. 启动elasticsearch，更新认证

即更新前面编辑的max-mac-e8e90718-9333-49b6-817a-9098c35c0920-v5.json认证文件）

[swadmin@localhost ~]$ elasticsearch -d

[swadmin@localhost test]$ cd /opt/src/bianyi/ && ll

max-mac-e8e90718-9333-49b6-817a-9098c35c0920-v5.json

[swadmin@localhost bianyi]$  curl -XPUT -u elastic 'http://127.0.0.1:9200/_xpack/license?acknowledge=true' -H "Content-Type: application/json" -d  @max-mac-e8e90718-9333-49b6-817a-9098c35c0920-v5.json

出现acknowledge字符就代表成功

1. 重启elasticsearch，检查更新

[swadmin@MyCloudServer ~]$ curl -XGET -u elastic "http://127.0.0.1:9200/_license"

{

  "license" : {

    "status" : "active",

    "uid" : "e8e90718-9333-49b6-817a-9098c35c0920",

    "type" : "platinum",

    "issue_date" : "2018-02-24T00:00:00.000Z",

    "issue_date_in_millis" : 1519430400000,

    "expiry_date" : "2049-12-31T16:00:00.999Z",

    "expiry_date_in_millis" : 2524579200999,

    "max_nodes" : 1000,

    "issued_to" : "max mac (SOLAIRE)",

    "issuer" : "Web Form",

    "start_date_in_millis" : 1519430400000

  }

}

 

 

• Elk优化

优化指南：

删除时间范围的数据：https://juejin.im/post/58e5de06ac502e006c254145

ELK平台性能优化 http://www.th7.cn/db/nosql/201708/250381.shtml

http://blog.csdn.net/jiao_fuyou/article/details/49783861

Logstash优化 http://blog.csdn.net/ypc123ypc/article/details/78033142   https://yq.aliyun.com/articles/413002 

elasticsearch优化 https://www.jianshu.com/p/29ffce0850af 

安装pv (pipe views） http://blog.csdn.net/u011478909/article/details/52584935 

优化： https://zhuanlan.zhihu.com/p/30888923  有具体参考方法

• elasticsearch优化

1. 索引的curl管理

1. 查看所有indices  

[swadmin@MyCloudServer ~]$ curl -XGET http://127.0.0.1:9200/_cat/indices -u elastic

1. 删除指定indices

[swadmin@MyCloudServer ~]$ curl -XDELETE "http://127.0.0.1:9200/hbet_tomcat_9002"  -u elastic

备注：

获取7天前的日期：[swadmin@MyCloudServer ~]$ date +%Y.%m.%d --date="-7 day"

Curator  ---删除indices    https://zhuanlan.zhihu.com/p/30888923     

删除索引脚本

#!/bin/bash

valite_date=$(date +%Y.%m.%d --date="-1 day")

elastic_ip=192.168.175.241

elastic_port=9200

elastic_user=elastic

elastic_pass=Passw0rd\!\*\*yibo

function delete_index(){

expect << EOF

 set timeout 2

 spawn curl -XDELETE http://$elastic_ip:$elastic_port/$1 -u $elastic_user ;

 expect {

     "elastic\'\:" { send "$elastic_pass\r" }

     }

expect eof

EOF

}

##### 删除指定索引

delete_index gameplat-$valite_date

delete_index tomcat-$valite_date

delete_index logstash-nginx-$valite_date

1. 查看threadpool

curl -XGET 'http://localhost:9200/_nodes/stats?pretty'  -u elastic

1. 查看集群信息

curl 'http://127.0.0.1:9200/_cluster/health?pretty'  -u elastic

1. 定期删除索引数据

参考： https://juejin.im/post/58e5de06ac502e006c254145

1. elasticsearch集群搭建

参考资料：

https://www.felayman.com/articles/2017/12/12/1513085668561.html

http://cwiki.apachecn.org/pages/viewpage.action?pageId=4882617

https://www.zybuluo.com/tinadu/note/516453

节点1 --- master  

[swadmin@MyCloudServer ~]$ cd /opt/apps/elk/elasticsearch/

[swadmin@MyCloudServer ~]$ cp config/elasticsearch.yml config/elasticsearch.yml.bak

[swadmin@MyCloudServer elasticsearch]$ grep -v \# config/elasticsearch.yml

cluster.name: my-application

node.name: node-1

path.data: /opt/apps/elk/elasticsearch/data/

path.logs: /opt/apps/elk/elasticsearch/logs/

network.host: 103.68.110.227

http.port: 9200

transport.tcp.port: 9600

node.master: true                    

discovery.zen.ping.unicast.hosts: ["103.68.110.227:9600", "103.68.110.242:9601"]

discovery.zen.minimum_master_nodes: 1

生成x-pack证书

103.68.110.223,103.68.110.225,103.68.110.227,103.68.110.242

参考资料：https://segmentfault.com/a/1190000012789290

[swadmin@MyCloudServer es_crt]$ cd /opt/apps/elk/elasticsearch/config/

[swadmin@MyCloudServer elk]$ /opt/apps/elk/elasticsearch/bin/x-pack/certgen

依次输入：my_cluster.zip ---> my_cluster ---> my_cluster ---> 103.68.110.242 ---> enter  ---> enter

[swadmin@MyCloudServer config]$ unzip my_cluster.zip

[swadmin@MyCloudServer ~]$ vim /opt/apps/elk/elasticsearch/config/elasticsearch.yml   ##末行追加

################x-pack相关配置

######停用x-pack

##xpack.security.enabled: false

######所有节点进行如下配置，证书秘钥路径一定要正确

xpack.ssl.key: my_cluster/my_cluster.key

xpack.ssl.certificate: my_cluster/my_cluster.crt

xpack.ssl.certificate_authorities: ca/ca.crt

xpack.security.transport.ssl.enabled: true

[swadmin@es_01 config]$ ../bin/x-pack/setup-passwords interactive  ###重新配置x-pack密码

节点2

[swadmin@MyCloudServer ~]$ cd /opt/apps/elk/elasticsearch/

[swadmin@MyCloudServer ~]$ cp config/elasticsearch.yml config/elasticsearch.yml.bak

[swadmin@MyCloudServer elasticsearch]$ grep -v \# config/elasticsearch.yml

cluster.name: my-application

node.name: node-2

path.data: /opt/apps/elk/elasticsearch/data/

path.logs: /opt/apps/elk/elasticsearch/logs/

network.host: 103.68.110.242

http.port: 9201

transport.tcp.port: 9601

discovery.zen.ping.unicast.hosts: ["103.68.110.227:9600", "103.68.110.242:9601"]

discovery.zen.minimum_master_nodes: 1

从节点1将my_cluster 和ca目录拉过来放到/opt/apps/elk/elasticsearch/config/下

1. 其他优化

1. 日志优化

[swadmin@MyCloudServer]$ vim log4j2.properties

logger.index_search_slowlog_rolling.level = info      ##默认为trace

1. 启动多个elasticsearch

https://my.oschina.net/u/3470972/blog/1586637

http://knktc.com/2016/06/10/elasticsearch-multiple-instances/

elasticsearch -Epath.conf=/opt/apps/elk/elasticsearch/config/my_cluster/  -Ecluster.name=my_cluster -Enode.name=node_2

1. 开放elasticsearch给外网

[swadmin@MyCloudServer ~]$ vim /opt/apps/elk/elasticsearch/config/elasticsearch.yml

transport.host: localhost

network.host: 0.0.0.0

[swadmin@MyCloudServer]$ vim /opt/apps/elk/elasticsearch/config/jvm.options

-Xms5g

-Xmx5g

备注：内存分配物理内存1/2或1/4

[swadmin@MyCloudServer]$ vim /opt/apps/elk/elasticsearch/bin/elasticsearch

ES_JAVA_OPTS="-Xms6g -Xmx6g"

完整优化

https://www.cnblogs.com/ningskyer/articles/5788667.html

1. 禁用x-pack功能   ---有必要的时候

[swadmin@vhost-elk ~]$ vim /opt/apps/elk/elasticsearch-01/config/elasticsearch.yml

xpack.security.enabled: false

• Logstash优化

1. 利用Pipeline启动多个管道

参考：

http://blog.csdn.net/ypc123ypc/article/details/78033142 

http://blog.csdn.net/ypc123ypc/article/details/69945031

http://blog.51niux.com/?id=205

[swadmin@MyCloudServer]$ vim /opt/apps/elk/logstash/config/pipelines.yml

- pipeline.id: tomcat_log

  queue.type: persisted

  path.config: "/opt/apps/elk/logstash/conf.d/elk_tomcat.conf"

  pipeline.workers: 6

  pipeline.batch.size: 1000

  pipeline.batch.delay: 5

- pipeline.id: nginx_log

  path.config: "/opt/apps/elk/logstash/conf.d/elk_nginx.conf"

  pipeline.workers: 4

  pipeline.batch.size: 800

  pipeline.batch.delay: 5

- pipeline.id: gameplat _log

  queue.type: persisted

  path.config: "/opt/apps/elk/logstash/conf.d/elk_gameplat.conf"

  pipeline.batch.size: 1000

  pipeline.batch.delay: 5

1. 内存和线程优化

[swadmin@MyCloudServer]$ vim /opt/apps/elk/logstash/config/jvm.options

-Xms6g

-Xmx6g

Logstash参数表

	参数	类别	说明
1	LS_HEAP_SIZE	LS	Logstash堆内存大小，默认1g
2	-w	LS启动	logstash线程数，默认与cpu数相同
3	-b	LS启动	Batch数，即logstash取多少数据进行一次filter，默认125
4	redis.threads	LS input	Redis线程数，默认1
5	redis.batch_count	LS input	Redis每次pop的数量，默认1
6	es.workers	LS output	Es提交线程，默认1
7	es.flush_size	LS output	ESbulk提
8	-l	LS启动	产生日志文件

备注：

带pv启动logstash

logstash -f logstash_dots_zzm1.conf  -l ./logstash_zzm1.log -b 8000 | pv -abt >/dev/null 

#####

再次优化索引

用一个

If ([fields][source] =~ "gameplat")

• x-pack常见参数优化

参考资料：https://www.felayman.com/articles/2017/12/12/1513085668561.html

• elk安全告警机制

参考：

https://xizhibei.github.io/2017/11/19/alerting-with-elastalert/   

https://github.com/xuyaoqiang/elastalert-dingtalk-plugin  钉钉报警插件

http://ksowo.com/2018/02/01/ELK%E6%8E%A5%E6%94%B6paloalto%E6%97%A5%E5%BF%97%E5%B9%B6%E7%94%A8%E9%92%89%E9%92%89%E5%91%8A%E8%AD%A6/

钉钉添加群---》群设置----》群机器人----》添加机器人---》编辑

创建的钉钉群链接：https://oapi.dingtalk.com/robot/send?access_token=db5c6b508ee0ffb30dfa9dc88589582f9fe5f0904def7ec8bcb4fb1c597cb436

sudo pip install setuptools --upgrade

• GeoIP地图库

Logstash下载GeoData

[swadmin@MyCloudServer]$ cd /opt/apps/elk/logstash/ && mkdir geoData/ 

[swadmin@MyCloudServer logstash]$ cd geoData && wget http://geolite.maxmind.com/download/geoip/database/GeoLite2-City.tar.gz 

[swadmin@logstash logstash]$ vim  /opt/apps/elk/logstash/conf.d/elk_nginx.conf

.....

     geoip {

       source => "clientip"

       target => "geoip"

       database => "/opt/apps/elk/logstash/geoData/GeoLite2-City_20180306/GeoLite2-City.mmdb"

       add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]

       add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ]

     }

....

        index => "logstash-nginx-%{+YYYY.MM.dd}"

...

备注：index必须以logstash开头

• 常见报错

1. 删掉tags的_grokparsefailure

    grok {

      match => { "message" => "%{COMBINEDAPACHELOG}" }

      tag_on_failure => []

      }