EFK（Elasticsearch+fluentd+Kibana）日志分析系统搭建

我的EFK机构如下，一共39台VM，fluented从各个服务（图中标记为APP)中以tail的形式读取日志，提取日志可以使用http://fluentular.herokuapp.com/网站对正则进行测试，然后发送到ElasticSearch，然后使用Kibana来展示，EFK均运行在单独的docker容器中

搭建步骤如下

一 Fluented的搭建

1 创建docker-compose.yml,挂载两个挂载点，一个是fluentd的配置文件，一个是日志所在的文件夹

version: '2'
services:
  fluentd:
    build: .
    expose:
      - 24224
    ports:
      - "24224:24224"
    volumes:
      - /data/conf/fluent.conf:/fluentd/etc/fluent.conf
      - /data/logs/nginx:/data/logs/nginx
    restart: always

2 创建Dockerfile，增加elasticsearch插件和forward插件

FROM fluent/fluentd:v1.12.0-debian-1.0
USER root
RUN ["gem", "install","fluent-plugin-elasticsearch","--no-document", "--version", "4.3.3"]
RUN ["gem", "install","fluent-plugin-forest","--no-document"]
USER root

3 配置fluent.conf

<source>
  @type tail
  path /data/logs/nginx/https-access.log
  pos_file /data/logs/nginx/https-access.log.pos
 <parse>
    @type nginx	 
 </parse>
 tag *
</source>
<match *.**>
  @type forest
  subtype copy
    <template>
	  <store>
		@type elasticsearch
		host XX.XX.XX.XX
		port 9200
	  </store>
   </template>
</match>

2 ElasticSearch和Kibana的配置

配置docker-compose.xml

version: '2'
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.10.2
    environment:
      - "discovery.type=single-node"
    expose:
      - 9200
    ports:
      - "9200:9200"
  kibana:
    image: kibana:7.10.1
    links:
      - "elasticsearch"
    ports:
      - "80:5601"

3 启动EFK

分别在flentd和EK下运行 docker-compose up --d

 docker ps
CONTAINER ID        IMAGE                                                  COMMAND                  CREATED             STATUS              PORTS                                NAMES
4941a0198a4f        fluentd_fluentd                                        "tini -- /bin/entr..."   4 days ago          Up 4 days           5140/tcp, 0.0.0.0:24224->24224/tcp   fluentd_fluentd_1
a33dee4a2bdb        kibana:7.10.1                                          "/usr/local/bin/du..."   7 days ago          Up 6 days           0.0.0.0:80->5601/tcp                 efk_kibana_1
dd05f58e2cbe        docker.elastic.co/elasticsearch/elasticsearch:7.10.2   "/tini -- /usr/loc..."   7 days ago          Up 6 days           0.0.0.0:9200->9200/tcp, 9300/tcp     efk_elasticsearch_1

4 访问Kibana所在服务器设置index

Discover中进行kibana的日志查询即可

EFK（Elasticsearch+fluentd+Kibana）日志分析系统搭建

猜你喜欢