Make a fluentd image with kafka plugin and es plugin

Preface

Fluentd is an open source data collector for the unified logging layer. It is the sixth CNCF graduated project after Kubernetes, Prometheus, Envoy, CoreDNS and containerd. It is often used to compare elastic logstash, which is relatively lightweight. Flexible, now the development is very fast, the community is very active.At the time of writing this blog, the star of github is 8.8k, and the fork is 1k.

premise

1. docker

Dockerfile file writing

Dockerfile

FROM fluent/fluentd:v1.3.2 
ADD fluent.conf /etc/fluent/
RUN echo "source 'https://mirrors.tuna.tsinghua.edu.cn/rubygems/'" > Gemfile && gem install bundler
RUN gem install fluent-plugin-kafka -v 0.12.3 --no-document
RUN gem install fluent-plugin-elasticsearch -v 4.0.3 --no-document
CMD ["fluentd"]

fluent.conf

<source>
  @type kafka

  brokers kafka:9092
  format json
  <topic>
    topic     kafeidou
  </topic>
</source>


<match *>
  @type elasticsearch
  host elasticsearch
  port 9200
  index_name fluentd
  type_name fluentd
</match>

The fluentd image with version v1.3.2 is used as the basic image. Since the processing layer extension of fluentd is extended by plug-ins, the corresponding kafka plug-in and
elasticsearch plug- in need to be installed when making this image. Here the fluentd plug-in version of kafka is 0.12.3, the fluentd plugin version of elasticsearch is 4.0.3.

Dockerfile and fluent.conf are ready, execute the mirroring command

docker build -t fluentd-es-kafka:v1.3.2 .

In this way, the fluentd image containing the es plugin and the kafka plugin is completed.

Running such a fluentd only needs a docker command to run.

docker run -it -d fluentd-es-kafka:v1.3.2

After the container is started, it will start listening for kafka messages with the host as kafka and transmit data to the elasticsearch node with the host as elasticsearch.

If the node addresses of es and kafka are different, you need to hang on the volume to override the default configuration file in the container.

docker run -it -v {存放fluent.conf的目录}:/etc/fluent -d fluentd-es-kafka:v1.3.2

Send a finished image:lypgcs/fluentd-es-kafka:v1.3.2

 

> Originating in four coffee beans  release! 
> Follow the official account->[Four Coffee Beans] Get the latest content