1 flume
选型：access.log==控制台输出
exec
memory
logger
====================================
步骤：1书写配置文件
streaming_project.conf:
exec-memory-logger.source = exec-source
exec-memory-logger.sink=logger-sink
exec-memory-logger.channels=memory-channel
exec-memory-logger.cources.exec-source.type=exec
exec-memory-logger.sources.exec-source.command=tail -F /home/hadoop/data/project/logs/access.log
exec-memory-logger.sources.exec-source.shell=/bin/sh -c
exec-memory-logger.channels.memory-channel.type=memory
exec-memory-logger.sinks.logger-sink =logger
exec-memory-logger.sources.exec-source.channels=memory-channel
exec-memory-logger.sinks.logger-sink.channel=memory-channel
步骤二: 启动
flume-ng agent
--name exec-memory-logger
--conf $FLUME_HOME/conf
--conf-file /home/hadoop/data/project/streaming_project.conf: //配置文件路径
-Dflume.root.logger=INFO,console
==========
日志==》Flume==》Kafka
启动zk: ./zkServer.sh start
启动kafka: ./kafka-server-start.sh -daemon /home/hadoop/kafka/conf/server.properties
修改flume配置文件使得flume sink 数据到kafka
streaming_project2.conf
exec-memory-kafka.source = exec-source
exec-memory-kafka.sink=kafka-sink
exec-memory-kafka.channels=memory-channel
exec-memory-kafka.cources.exec-source.type=exec
exec-memory-kafka.sources.exec-source.command=tail -F /home/hadoop/data/project/logs/access.log
exec-memory-kafka.sources.exec-source.shell=/bin/sh -c
exec-memory-kafka.channels.memory-channel.type=memory
exec-memory-kafka.sinks.kafka-sink.type =org.apache.flume.sink.kafka.KafkaSink
exec-memory-kafka.sinks.kafka-sink.brokerList =hadoop000:9092
exec-memory-kafka.sinks.kafka-sink.topic=streamingtopic
exec-memory-kafka.sinks.kafka-sink.bathSize=5
exec-memory-kafka.sinks.kafka-sink.requiredAcks=1
exec-memory-kafka.sources.exec-source.channels=memory-channel
exec-memory-kafka.sinks.kafka-sink.channel=memory-channel
flume-ng agent
--name exec-memory-kafka
--conf $FLUME_HOME/conf
--conf-file /home/hadoop/data/project/streaming_project.conf: //配置文件路径
-Dflume.root.logger=INFO,console
===============================