About Flume-ng those things (c)

Why 80% of the code can not do farming architect? >>>

agentA （10.1.124.197）agentB（10.1.124.196）agent（10.1.124.198）

Test 1

agentA => agentB push data from the client to agentB, change the channel to jdbc events ensure data reliability.

## weblog agent config agent A 配置

#List sources, sinks and channels in the agent weblog-agent.sources = avro-sour weblog-agent.sinks = avro-forward-sink weblog-agent.channels = jdbc-channel #define the flow #webblog-agent sources config weblog-agent.sources.avro-sour.channels = jdbc-channel weblog-agent.sources.avro-sour.type = avro weblog-agent.sources.avro-sour.bind = 0.0.0.0 weblog-agent.sources.avro-sour.port = 41414 #avro sink properties weblog-agent.sinks.avro-forward-sink.channel = jdbc-channel weblog-agent.sinks.avro-forward-sink.type = avro weblog-agent.sinks.avro-forward-sink.hostname = 10.1.124.196 weblog-agent.sinks.avro-forward-sink.port = 41414 #channels config #weblog-agent.channels.jdbc-channel.type = memory weblog-agent.channels.jdbc-channel.type = jdbc

agnetB configuration:

# Define a memory channel called ch1 on agent1 agent1.channels.ch1.type = memory

# Define an Avro source called avro-source1 on agent1 and tell it # to bind to 0.0.0.0:41414. Connect it to channel ch1. agent1.sources.avro-source1.channels = ch1 agent1.sources.avro-source1.type = avro agent1.sources.avro-source1.bind = 0.0.0.0 agent1.sources.avro-source1.port = 41414

# Define a logger sink that simply logs all events it receives # and connect it to the other end of the same channel. agent1.sinks.log-sink1.channel = ch1 agent1.sinks.log-sink1.type = logger

# Finally, now that we've defined all of our components, tell # agent1 which ones we want to activate. agent1.channels = ch1 agent1.sources = avro-source1 agent1.sinks = log-sink1

agent A 上执行：flume-ng agent -n agent1 -c /opt/apps/flume-ng/conf/ -f flume.conf &

Test Two

According to a test events continue to test reliability, stop off flume agent service on agentB. The test write data on agent A. start agentB the flume agent services. events on agentA synchronized to the agentB. Test three agentA performed using exec tail as sources, and use agentB agentC as the collection of data points. | --------------- agentB agentA | --------------- agentC test content respectively agentA of souces use exec execute tail command reads nginx logs were forwarded to the event agentB, agentC, test load balancing and reliability agentB two sinks, agentC profile with a test of AgentB. agentA configuration file as follows:

#List sources, sinks and channels in the agent weblog-agent.sources = tail weblog-agent.sinks = avro-forward-sink01 avro-forward-sink02 weblog-agent.channels = jdbc-channel #define the flow #webblog-agent sources config weblog-agent.sources.tail.channels = jdbc-channel weblog-agent.sources.tail.type = exec weblog-agent.sources.tail.command = tail -f /opt/apps/nginx/logs/access.log #avro sink properties weblog-agent.sinks.avro-forward-sink01.channel = jdbc-channel weblog-agent.sinks.avro-forward-sink01.type = avro weblog-agent.sinks.avro-forward-sink01.hostname = 10.1.124.196 weblog-agent.sinks.avro-forward-sink01.port = 41414 #avro sink 02 weblog-agent.sinks.avro-forward-sink02.channel = jdbc-channel weblog-agent.sinks.avro-forward-sink02.type = avro weblog-agent.sinks.avro-forward-sink02.hostname = 10.1.124.198 weblog-agent.sinks.avro-forward-sink02.port = 41414 #channels config #weblog-agent.channels.jdbc-channel.type = memory weblog-agent.channels.jdbc-channel.type = jdbc weblog-agent.channels.jdbc-channel.sysprop.user.home = /tmp/flumedb

performed on agnetA: performing a flume-ng agent -n weblog-agent -c / opt / apps / flume-ng / conf / -f flume.conf & agnetB, C: flume-ng agent -n agent1 -c / opt / apps / flume-ng / conf / -f flume.conf & test results: agentA written respectively in the event agentB and agentC, similar lvs rr mode, the flume agent on the stop agentC event will be written on Agenta agentB after recovery agentC, agentA re-polling on the event and wrote on agentB C. Test Four agentB HA agentC do agentA sink and method of using the configuration Flume Sink Processors. It can be observed under the agnetA profile with three different test

#List sources, sinks and channels in the agent weblog-agent.sources = tail weblog-agent.sinks = avro-forward-sink01 avro-forward-sink02 weblog-agent.channels = jdbc-channel #define the flow #webblog-agent sources config weblog-agent.sources.tail.channels = jdbc-channel weblog-agent.sources.tail.type = exec weblog-agent.sources.tail.command = tail -f /opt/apps/nginx/logs/access.log #avro sink properties weblog-agent.sinks.avro-forward-sink01.channel = jdbc-channel weblog-agent.sinks.avro-forward-sink01.type = avro weblog-agent.sinks.avro-forward-sink01.hostname = 10.1.124.196 weblog-agent.sinks.avro-forward-sink01.port = 41414 #avro sink 02 weblog-agent.sinks.avro-forward-sink02.channel = jdbc-channel weblog-agent.sinks.avro-forward-sink02.type = avro weblog-agent.sinks.avro-forward-sink02.hostname = 10.1.124.198 weblog-agent.sinks.avro-forward-sink02.port = 41414 #sink FailoverSink weblog-agent.sinkgroups=group1 weblog-agent.sinkgroups.group1.sinks = avro-forward-sink01 avro-forward-sink02 weblog-agent.sinkgroups.group1.processor.type = failover weblog-agent.sinkgroups.group1.processor.priority.avro-forward-sink01 = 5 weblog-agent.sinkgroups.group1.processor.priority.avro-forward-sink02 = 10 #channels config #weblog-agent.channels.jdbc-channel.type = memory weblog-agent.channels.jdbc-channel.type = jdbc weblog-agent.channels.jdbc-channel.sysprop.user.home = /tmp/flumedb

Define a sinkgroups group. The members of this group only one of sinks work. The larger the priority value that represents the higher priority. priority values are the same, take the members of the last occurrence. Test results: avro-forward-sink02 provide sink work. sotp agentC off the flume agent, event on agentA flume agent used to the recovery on the agentC agentB avro-forward-sink01, sent to the respective new event agentC. Test five agentC ---- | | ====> agentB Agenta ----- | test content, aggregation, configuration files with agentA & agentC

## weblog agent config agentA & agentC

#List sources, sinks and channels in the agent weblog-agent.sources = avro-sour weblog-agent.sinks = avro-forward-sink weblog-agent.channels = jdbc-channel #define the flow #webblog-agent sources config weblog-agent.sources.avro-sour.channels = jdbc-channel weblog-agent.sources.avro-sour.type = avro weblog-agent.sources.avro-sour.bind = 0.0.0.0 weblog-agent.sources.avro-sour.port = 41414 #weblog-agent.sources.avro-sour.restart = true #avro sink properties weblog-agent.sinks.avro-forward-sink.channel = jdbc-channel weblog-agent.sinks.avro-forward-sink.type = avro weblog-agent.sinks.avro-forward-sink.hostname = 10.1.124.196 weblog-agent.sinks.avro-forward-sink.port = 41414 #channels config #weblog-agent.channels.jdbc-channel.type = memory weblog-agent.channels.jdbc-channel.type = jdbc

agentB

# Define a memory channel called ch1 on agent1 agent1.channels.ch1.type = memory

# Define a logger sink that simply logs all events it receives # and connect it to the other end of the same channel. agent1.sinks.log-sink1.channel = ch1 agent1.sinks.log-sink1.type = logger

# Finally, now that we've defined all of our components, tell # agent1 which ones we want to activate. agent1.channels = ch1 agent1.sources = avro-source1 agent1.sinks = log-sink1

The test results, brought together properly.

Test six

Test content, sources select type replicating, I called a log raid1. The event will agentA wrote agentB and C agentA configuration file as follows:

#List sources, sinks and channels in the agent

weblog-agent.sources = tail

weblog-agent.sinks = avro-forward-sink01 avro-forward-sink02

weblog-agent.channels = jdbc-channel01 jdbc-channel02

#define the flow

#webblog-agent sources config

weblog-agent.sources.tail.channels = jdbc-channel01 jdbc-channel02

weblog-agent.sources.tail.type = exec

weblog-agent.sources.tail.command = tail -f /opt/apps/nginx/logs/access.log

weblog-agent.sources.tail.selector.type = replicating

#avro sink properties

weblog-agent.sinks.avro-forward-sink01.channel = jdbc-channel01

weblog-agent.sinks.avro-forward-sink01.type = avro

weblog-agent.sinks.avro-forward-sink01.hostname = 10.1.124.196

weblog-agent.sinks.avro-forward-sink01.port = 41414

#avro sink 02

weblog-agent.sinks.avro-forward-sink02.channel = jdbc-channel02

weblog-agent.sinks.avro-forward-sink02.type = avro

weblog-agent.sinks.avro-forward-sink02.hostname = 10.1.124.198

weblog-agent.sinks.avro-forward-sink02.port = 41414

#channels config

#weblog-agent.channels.jdbc-channel.type = memory

weblog-agent.channels.jdbc-channel01.type = jdbc

weblog-agent.channels.jdbc-channel01.sysprop.user.home = /tmp/flumedb01

weblog-agent.channels.jdbc-channel02.type = jdbc

weblog-agent.channels.jdbc-channel02.sysprop.user.home = /tmp/flumedb02

The test results, see agentB and event entirely about C obtained from the agentA, note the time, agentC collection of event time than B would be late. Please indicate the source: http://www.ttlsa.com/html/2770.html

Reproduced in: https: //my.oschina.net/766/blog/211201