实现需求：整合 Flume 和 Kafka 完成实时数据收集，即使用 Flume 中的 Kafka Sink 将 Flume 实时收集到的日志信息输出到 Kafka
笔者使用的Flume版本为1.6，Kafka版本为0.11.0.0，Flume 中的Kafka Sink的使用参见Flume官方文档：
http://flume.apache.org/releases/content/1.6.0/FlumeUserGuide.html
注：上图中黑色加粗字体的项为必须要配置的，红色框中的官方给出了一个配置模板，下面我们参照这个模板来进行配置
在博客使用Log4j将日志实时写入Flume的基础上完成 Flume 整合 Kafka，实现Kafka不断消费Flume收集到的Log4j日志。
启动Kafka（先启动ZK）
kafka-server-start.sh $KAFKA_HOME/config/server.propertiesKafka 中创建 topic
[hadoop@Master ~]$ kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic flume2kafka
Created topic "flume2kafka".
[hadoop@Master ~]$ kafka-topics.sh --describe --zookeeper localhost:2181 --topic flume2kafka
Topic:flume2kafka PartitionCount:1 ReplicationFactor:1 Configs:
Topic: flume2kafka Partition: 0 Leader: 0 Replicas: 0 Isr: 0Flume 整合 Kafka Agent 配置文件编写
# Name the components on this agent
flume2kafka-agent.sources = arvo-source
flume2kafka-agent.sinks = kafka-sink
flume2kafka-agent.channels = memory-channel
# Describe/configure the source
flume2kafka-agent.sources.arvo-source.type = avro
flume2kafka-agent.sources.arvo-source.bind = Master
flume2kafka-agent.sources.arvo-source.port = 44444
# Flume 整合 Kafka
flume2kafka-agent.sinks.kafka-sink.type = org.apache.flume.sink.kafka.KafkaSink
flume2kafka-agent.sinks.kafka-sink.topic = flume2kafka
flume2kafka-agent.sinks.kafka-sink.brokerList = Master:9092
# 10 条记录写一次
flume2kafka-agent.sinks.kafka-sink.batchSize = 10
# Use a channel which buffers events in memory
flume2kafka-agent.channels.memory-channel.type = memory
# Bind the source and sink to the channel
flume2kafka-agent.sources.arvo-source.channels = memory-channel
flume2kafka-agent.sinks.kafka-sink.channel = memory-channel启动Agent
flume-ng agent \
--conf $FLUME_HOME/conf \
--conf-file $FLUME_HOME/config/flume2kafka-agent.conf \
--name flume2kafka-agent \
-Dflume.root.logger=INFO,consoleKafka 中启动消费者
kafka-console-consumer.sh --zookeeper localhost:2181 --topic flume2kafka运行项目，观察Kafka 消费者控制台
