Flume聚合的原理与实现

原理

在每个服务器部署flume采集日志，由这些flume传输数据到统一收集日志的flume节点，最后由此flume将数据写入指定位置。

案例

hadoop01上的Flume-1监控文件/opt/module/group.log，
hadoop02上的Flume-2监控某一个端口的数据流，
Flume-1与Flume-2将数据发送给hadoop03上的Flume-3，Flume-3将最终数据打印到控制台。

实现

# agent1
# 定义sources、channels、sinks
a1.sources = r1
a1.channels = c1
a1.sinks = k1

# 配置source
a1.sources.r1.type = TAILDIR
# 日志索引记录
a1.sources.r1.positionFile = /opt/module/flume/inode/taildir_position.json
# 文件组
a1.sources.r1.filegroups = f1
a1.sources.r1.filegroups.f1 = /opt/module/group.log

# 配置channel
a1.channels.c1.type = memory

# 配置sink
a1.sinks.k1.type = avro
a1.sinks.k1.hostname = hadoop03
a1.sinks.k1.port = 6666

# 配置source、channel、sink关系
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1

# agent2
# 定义sources、channels、sinks
a1.sources = r1
a1.channels = c1
a1.sinks = k1

# 配置source
a1.sources.r1.type = netcat
a1.sources.r1.bind = hadoop02
a1.sources.r1.port = 44444

# channel
a1.channels.c1.type = memory

# sink
a1.sinks.k1.type = avro
a1.sinks.k1.hostname = hadoop03
a1.sinks.k1.port = 6666


# 配置关系
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1

# agent3
# 定义sources、channels、sinks
a1.sources = r1
a1.channels = c1
a1.sinks = k1

# 配置source
a1.sources.r1.type = avro
a1.sources.r1.bind = hadoop03
a1.sources.r1.port = 6666

# channel
a1.channels.c1.type = memory

# sink
a1.sinks.k1.type = logger


# 配置关系
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1

先启动数据接收端，再启动数据发送端。