ELK安装

安装Logstash依赖包JDK

Logstash的运行依赖于Java运行环境， Logstash 1.5以上版本不低于java 7推荐使用最新版本的Java。由于我们只是运行Java程序，而不是开发，下载JRE即可。首先，在Oracle官方下载新版jre，下载地址：http://www.oracle.com/technetwork/java/javase/downloads/jre8-downloads-2133155.html

$ mkdir /usr/local/java

$ tar -zxf jdk-8u45-linux-x64.tar.gz -C /usr/local/java/

设置JDK的环境变量，如下：

$ vim ~/.bash_profile

export JAVA_HOME=/usr/local/java/jdk1.8.0_161

export PATH=$PATH:$JAVA_HOME/bin

exportCLASSPATH=.:$JAVA_HOME/lib/tools.jar:$JAVA_HOME/lib/dt.jar:$CLASSPATH

$ java -version (注意是-version 而非 --version)

java version "1.8.0_161"

Java(TM) SE Runtime Environment (build 1.8.0_161-b12)

Java HotSpot(TM) 64-Bit Server VM (build 25.161-b12, mixed mode)

安装Logstash

$ https://download.elastic.co/logstash/logstash/logstash-6.2.3.tar.gz

$ tar –zxf logstash-6.2.3.tar.gz -C /usr/local/

安装完成后运行如下命令：

$ /usr/local/logstash-6.2.3/bin/logstash -e 'input { stdin { } } output { stdout {} }'

输出：

Sending Logstash's logs to /usr/local/logstash-6.2.3/logs which is now configured via log4j2.properties

[2018-04-09T01:54:36,236][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/usr/local/logstash-6.2.3/modules/netflow/configuration"}

[2018-04-09T01:54:36,267][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/usr/local/logstash-6.2.3/modules/fb_apache/configuration"}

[2018-04-09T01:54:36,953][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified

[2018-04-09T01:54:37,799][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.2.3"}

[2018-04-09T01:54:38,385][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}

[2018-04-09T01:54:40,664][INFO ][logstash.pipeline ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}

[2018-04-09T01:54:40,871][INFO ][logstash.pipeline ] Pipeline started succesfully {:pipeline_id=>"main", :thread=>"#<Thread:0x72564250 run>"}

The stdin plugin is now waiting for input:

[2018-04-09T01:54:40,981][INFO ][logstash.agent ] Pipelines running {:count=>1, :pipelines=>["main"]}

hello world

我们可以看到，我们输入什么内容logstash按照某种格式输出，其中-e参数参数允许Logstash直接通过命令行接受设置。这点可以帮助我们反复的测试配置是否正确而不用写配置文件。使用CTRL-C命令可以退出之前运行的Logstash。

使用-e参数在命令行中指定配置是很常用的方式，不过如果需要配置更多设置则需要很长的内容。这种情况，我们首先创建一个简单的配置文件，并且指定logstash使用这个配置文件。

例如：在logstash安装目录下创建一个“基本配置”测试文件logstash-simple.conf，文件内容如下：

$ cat logstash-simple.conf

input { stdin { } }

output {

stdout { codec=> rubydebug }

}

Logstash使用input和output定义收集日志时的输入和输出的相关配置，本例中input定义了一个叫"stdin"的input，output定义一个叫"stdout"的output。无论我们输入什么字符，Logstash都会按照某种格式来返回我们输入的字符，其中output被定义为"stdout"并使用了codec参数来指定logstash输出格式。

使用logstash的-f参数来读取配置文件，执行如下开始进行测试：

$ echo "`date` hello World"

Mon Apr 9 02:08:45 UTC 2018 hello World

$ /usr/local/logstash-6.2.3/bin/logstash -f logstash-simple.conf

Logstash startup completed

Mon Apr 9 02:08:45 UTC 2018 hello Worl #该行是执行echo “`date`hello World” 后输出的结果，直接粘贴到该位置

{

"@timestamp" => 2018-04-09T02:17:15.064Z,

"message" => "Mon Apr 9 02:08:45 UTC 2018 hello Worl",

"@version" => "1",

"host" => "5ef8026aa3bf"

}

安装Elasticsearch

$ tar -zxf elasticsearch-6.2.3.tar.gz -C /usr/local/

启动Elasticsearch

$ /usr/local/elasticsearch-6.2.3/bin/elasticsearch

OR后台执行

$ nohup /usr/local/elasticsearch-6.2.3/bin/elasticsearch &

这时有可能会直接被Killed掉，因为内存溢出（OOM），elastisearch占用的内存非常大，所以在内存比较小的服务器上运行要先修改jvm的内存大小

$ vim config/jvm.options

将22和23行的栈堆大小改为512M

-Xms512M

-Xmx512M

注：如果在运行过程中还出现Killed，继续将以上的值调小。

[2018-04-09T03:06:34,552][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [] uncaught exception in thread [main]

org.elasticsearch.bootstrap.StartupException: java.lang.RuntimeException: can not run elasticsearch as root

at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:125) ~[elasticsearch-6.2.3.jar:6.2.3]

at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:112) ~[elasticsearch-6.2.3.jar:6.2.3]

at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.2.3.jar:6.2.3]

at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.2.3.jar:6.2.3]

at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.2.3.jar:6.2.3]

at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-6.2.3.jar:6.2.3]

at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:85) ~[elasticsearch-6.2.3.jar:6.2.3]

Caused by: java.lang.RuntimeException: can not run elasticsearch as root

at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:105) ~[elasticsearch-6.2.3.jar:6.2.3]

at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:172) ~[elasticsearch-6.2.3.jar:6.2.3]

at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:323) ~[elasticsearch-6.2.3.jar:6.2.3]

at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:121) ~[elasticsearch-6.2.3.jar:6.2.3]

... 6 more

因为安全问题elasticsearch 不让用root用户直接运行，所以要创建新用户

$ groupadd tzhennan

$ useradd tzhennan -g tzhennan

Exception in thread "main" java.nio.file.AccessDeniedException: /usr/local/elasticsearch-6.2.3/config/jvm.options

at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84)

at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)

at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)

at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214)

at java.nio.file.Files.newByteChannel(Files.java:361)

at java.nio.file.Files.newByteChannel(Files.java:407)

at java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:384)

at java.nio.file.Files.newInputStream(Files.java:152)

at org.elasticsearch.tools.launchers.JvmOptionsParser.main(JvmOptionsParser.java:58)

$ chown -R elasticsearch:elasticsea elasticsearch-6.2.3/

查看elasticsearch的9200端口是否已监听

$ netstat -anp | grep :9200

tcp 0 0 127.0.0.1:9200 0.0.0.0:* LISTEN 1483/java

接下来我们在logstash安装目录下创建一个用于测试logstash使用elasticsearch作为logstash的后端的测试文件logstash-es-simple.conf，该文件中定义了stdout和elasticsearch作为output，这样的“多重输出”即保证输出结果显示到屏幕上，同时也输出到elastisearch中

input { stdin { } }

output {

elasticsearch {

hosts => "localhost:9200"

user => "tzhennan"

password => "xxx"

}

stdout {

codec=> rubydebug

}

执行命令

$ /usr/local/logstash-6.2.3/bin/logstash -f logstash-es-simple.conf

[2018-04-09T07:33:50,581][INFO ][logstash.outputs.elasticsearch] Installing elasticsearch template to _template/logstash

[2018-04-09T07:33:51,886][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost:9200"]}

[2018-04-09T07:33:52,078][INFO ][logstash.pipeline ] Pipeline started succesfully {:pipeline_id=>"main", :thread=>"#<Thread:0x17ddf470 run>"}

The stdin plugin is now waiting for input:

[2018-04-09T07:33:52,202][INFO ][logstash.agent ] Pipelines running {:count=>1, :pipelines=>["main"]}

hello logstash

{

"@timestamp" => 2018-04-09T07:35:03.141Z,

"host" => "5ef8026aa3bf",

"@version" => "1",

"message" => "hello logstash"

}

可以使用curl命令发送请求来查看ES是否接收到了数据：

$ curl 'http://localhost:9200/_search?pretty'

输出：

{

"took" : 154,

"timed_out" : false,

"_shards" : {

"total" : 5,

"successful" : 5,

"skipped" : 0,

"failed" : 0

"hits" : {

"total" : 1,

"max_score" : 1.0,

"hits" : [

{

"_index" : "logstash-2018.04.09",

"_type" : "doc",

"_id" : "WulUqWIB-5JaC2wTa7rt",

"_score" : 1.0,

"_source" : {

"@timestamp" : "2018-04-09T07:35:03.141Z",

"host" : "5ef8026aa3bf",

"@version" : "1",

"message" : "hello logstash"

}

]

}

至此，已经成功利用Elasticsearch和Logstash来收集日志数据了

问题：

可以访问127.0.0.1:9200，但不能访问公网IP:9200 ，需要在在elasticsearch.yml文件中增加以下配置：

network.bind_host: 0.0.0.0

安装elasticsearch插件

$ cd /usr/local/elasticsearch-6.2.3/

安装Head插件

$ ./bin/plugin install mobz/elasticsearch-head

安装安装elasticsearch-kopf插件

$ ./plugin -install lmenezes/elasticsearch-kopf

安装完成后在plugins目录下可以看到

$ ls plugin

可以在浏览器访问http://ip:9200/_plugin/kopf浏览保存在Elasticsearch中的数据

安装Kibana

$ tar -zxf kibana-6.2.3-linux-x86_64.tar.gz -C /usr/local/

启动kibana

$ /usr/local/kibana-6.2.3-linux-x86_64/bin/kibana

1>使用http://ip:5601访问Kibana，登录后，首先，配置一个索引，默认，Kibana的数据被指向Elasticsearch，使用默认的logstash-*的索引名称，并且是基于时间的，点击“Create”即可

2>点击“Discover”，可以搜索和浏览Elasticsearch中的数据，默认搜索的是最近15分钟的数据。可以自定义选择时间

问题：

可以访问127.0.0.1:5601，但不能访问公网IP:9200 ，需要在在kibaba.yml文件中增加以下配置：

server.host: "localhost"

更改为

server.host: "0.0.0.0"

配置logstash作为Indexer

将logstash配置为索引器，并将logstash的日志数据存储到Elasticsearch

input {

file {

type =>"syslog"

path => ["/var/log/messages", "/var/log/syslog" ]

}

syslog {

type => "syslog"

port => "5544"

}

output {

stdout { codec=> rubydebug }

elasticsearch {

hosts => "localhost:9200"

user => "tzhennan"

password => "xxx"

}

启动

$ /usr/local/logstash-6.2.3/bin/logstash -flogstash-indexer.conf

使用echo命令模拟写入日志

$ echo "`date` 本地系统测试" >>/var/log/messages

...

参考文章：

http://blog.51cto.com/baidu/1676798

https://my.oschina.net/itblog/blog/547250

猜你喜欢