集群配置了3台机器，分别为master,slave,slave2
ip地址分别为192.168.88.128,192.168.88.129,192.168.88.130
其中master的配置为:
agent.sources = source1 agent.channels = channel1 agent.sinks = sink1 sink2 agent.sinkgroups = group1 agent.sinkgroups.group1.sinks = sink1 sink2 agent.sinkgroups.group1.processor.type = load_balance agent.sinkgroups.group1.processor.selector = round_robin agent.sinkgroups.group1.processor.backoff = true agent.channels.channel1.type = file agent.channels.channel1.checkpointDir = /tmp/flume/loadcheckpoint agent.channels.channel1.dataDirs = /tmp/flume/loaddata # For each one of the sources, the type is defined agent.sources.source1.type = exec agent.sources.source1.command = tail -F /home/bigdata/datasource/flume_user_action.txt # The channel can be defined as follows. agent.sources.source1.channels = channel1 agent.sinks.sink1.channel = channel1 agent.sinks.sink1.type = avro agent.sinks.sink1.hostname = slave agent.sinks.sink1.port = 41414 #Specify the channel the sink should use agent.sinks.sink2.channel = channel1 agent.sinks.sink2.type = avro agent.sinks.sink2.hostname = slave2 agent.sinks.sink2.port = 41414
slave的配置为:
agent.sources = source1 agent.channels = channel1 agent.sinks = sink1 agent.sources.source1.type = avro agent.sources.source1.bind = 192.168.88.129 agent.sources.source1.port = 41414 agent.sources.source1.channels= channel1 agent.channels.channel1.type = file agent.channels.channel1.checkpointDir = /home/bigdata/tmp/flume/checkpoint agent.channels.channel1.dataDirs = /home/bigdata/tmp/flume/channeldata agent.sinks.sink1.channel = channel1 agent.sinks.sink1.type = file_roll agent.sinks.sink1.sink.directory = /home/bigdata/tmp/flume/loaddata agent.sinks.sink1.sink.rollInterval = 0
slave2的配置为:
agent.sources = source1 agent.channels = channel1 agent.sinks = sink1 agent.sources.source1.type = avro agent.sources.source1.bind = 192.168.88.130 agent.sources.source1.port = 41414 agent.sources.source1.channels = channel1 agent.channels.channel1.type = file agent.channels.channel1.checkpointDir = /home/bigdata/tmp/flume/checkpoint agent.channels.channel1.dataDirs = /home/bigdata/tmp/flume/channeldata agent.sinks.sink1.channel = channel1 agent.sinks.sink1.type = file_roll agent.sinks.sink1.sink.directory = /home/bigdata/tmp/flume/loaddata agent.sinks.sink1.sink.rollInterval = 0
有几个地方要注意一下，第一个是agent.sources.source1.bind的地址必须填ip地址，不然master的sink写不过来
报错如下：
Exception in thread "main" org.apache.flume.FlumeException:NettyAvroRpcClient{
host:xxxx,port:xxxx}:RPC connection error
第二个是agent.sinks.sink1.sink.directory这个参数的目录必须存在，不然写不进去
报错如下：
