使用Docker搭建ELK日志搜集系统(二)

上篇介绍了ELF/EFK中Elasticsearch、Kibana的配置安装，这篇开始介绍Fluentd做为日志搜集工具的配置安装

Fluentd是什么？

• Fluented 是一个开源免费的日志搜集工具，经常代替Logstash(EFK)，支持的插件非常多，对docker支持较好
• Fluentd尽量将数据结构化为JSON:这允许Fluentd统一处理日志数据的所有方面:收集、过滤、缓冲和跨多个源和目的地输出日志(统一日志层)
• 使用JSON进行下游数据处理要容易得多，因为它有足够的结构，可以在保持灵活模式的同时进行访问

Fluentd可以搜集那些日志

使用Fluentd搜集Docker日志配置

系统安装运行需求和建议:

• 至少一台独立的linux host centos 7.4,docker 18.06.1-ce+以上版本，建议两台做高可用
• 生产环境推荐2核4G内存

安装步骤

1. 安装docker环境

   https://www.cnblogs.com/yjb007/p/9551878.html

2. 安装fluentd

mkdir -p /srv/volume/fluentd/
cd /srv/volume/fluentd/
mkdir -p plugins/
cat > Dockerfile << 'EOF'
FROM fluent/fluentd:v1.2.5-debian-onbuild

ENV TZ=Asia/Shanghai
RUN apt-get update \
        && apt-get -y install tzdata \
        && apt-get -y install curl \
        && ln -snf /usr/share/zoneinfo/$TZ /etc/localtime \
        && echo $TZ > /etc/timezone

RUN buildDeps="sudo make gcc g++ libc-dev ruby-dev" \
 && apt-get update \
 && apt-get install -y --no-install-recommends $buildDeps \
 && sudo gem install \
        fluent-plugin-elasticsearch \
 && SUDO_FORCE_REMOVE=yes \
    apt-get purge -y --auto-remove \
                  -o APT::AutoRemove::RecommendsImportant=false \
                  $buildDeps \
 && rm -rf /var/lib/apt/lists/* \
           /home/fluent/.gem/ruby/2.3.0/cache/*.gem

COPY fluent.conf /fluentd/etc/fluent.conf
EOF

cat > fluent.conf << 'EOF'
<source>
  @type tcp
  tag windows
  format none
  port 8888
  bind 0.0.0.0
</source>

<source>
  @type forward
  port 24224
  bind 0.0.0.0
</source>

<filter windows.**>
  @type parser
  format json
# time_format %Y-%m-%dT%H:%M:%S.%L%Z
  key_name message
  reserve_data false
  suppress_parse_error_log true
  ignore_key_not_exist true
  replace_invalid_sequence true
</filter>

<filter docker.**>
  @type parser
  format json
# time_format %Y-%m-%dT%H:%M:%S.%L%Z
  key_name log
  reserve_data false
  suppress_parse_error_log true
  ignore_key_not_exist true
  replace_invalid_sequence true
</filter>

<filter docker.nginx.**>
  type record_transformer
  enable_ruby true
  <record>
    "@timestamp" ${Time.now.strftime("%Y-%m-%dT%H:%M:%S.%L%z")}
  </record>
</filter>

<match windows.**>
  @type elasticsearch
  hosts 10.1.1.225:9200
  user elastic
  password 'qwe123!@#'
  
  target_index_key @target_index
  index_name default_index
  type_name fluentd

  include_tag_key true
  tag_key DockerName

  flush_interval 1s
  request_timeout 15s
  reload_connections false
  reconnect_on_error true
  resurrect_after 5s

  buffer_type memory
  num_threads 2
</match>


<match docker.**>
  @type elasticsearch
  hosts 10.1.1.225:9200
  user elastic
  password 'qwe123!@#'

  target_index_key @target_index
  index_name default_index
  type_name fluentd

  include_tag_key true
  tag_key DockerName

  flush_interval 1s
  request_timeout 15s
  reload_connections false
  reconnect_on_error true
  resurrect_after 5s

  buffer_type memory
  num_threads 2
</match>


<match **>
  @type elasticsearch
  hosts 10.1.1.225:9200
  user elastic
  password 'qwe123!@#'
  
  logstash_format true
  logstash_prefix unknown
  logstash_dateformat %Y.%m.%d
  
  time_key_format %Y-%m-%dT%H:%M:%S.%N%z
  
  target_index_key @target_index
  index_name default_index
  type_name fluentd

  include_tag_key true
  tag_key DockerName

  flush_interval 1s
  request_timeout 15s
  reload_connections false
  reconnect_on_error true
  resurrect_after 5s

  buffer_type memory
  num_threads 2
</match>
EOF

docker build --no-cache --pull -t fluentd/yujianbo:20180829 ./

docker rm -f pilipa-inside-fluentd
docker run -d \
        --name pilipa-inside-fluentd \
        --env FLUENTD_CONF=fluent.conf \
        -v /srv/fluentd/fluent.conf:/fluentd/etc/fluent.conf \
        -p 8888:8888 \
        -p 24224:24224 \
fluentd/yujianbo:20180829