flume实战

flume 三大组件

source 收集

channel 聚集

sink 输出

使用Flume关键就是写配置文件

A 配置source

B 配置channel

C 配置sink

D 把以上3个组件串起来

扫描二维码关注公众号，回复： 7385875 查看本文章

1.通过IP端口 接收数据

  

a1 agent名称
r1 数据源名称
k1 sinks名称
c1 channel名称

# example.conf: A single-node Flume configuration

# Name the components on this agent
a1.sources = r1
a1.sinks = k1 a1.channels = c1 # Describe/configure the source a1.sources.r1.type = netcat a1.sources.r1.bind =hadoop000
a1.sources.r1.port = 44444 # Describe the sink a1.sinks.k1.type = logger # Use a channel which buffers events in memory a1.channels.c1.type = memory # Bind the source and sink to the channel a1.sources.r1.channels = c1 a1.sinks.k1.channel = c1

启动agent

flume-ng agent  \

--name a1 \

--conf  $FLUME_HOME/conf \

--conf-file  $FLUME_HOME/conf/example.conf \

-Dflume.roog.logger=INFO,console

agent选型 ： exec source+ memory channel + logger skin

# Name the components on this agent

a1.sources = r1
a1.sinks = k1
a1.channels = c1

# Describe/configure the source
a1.sources.r1.type = exec
a1.sources.r1.command=tail -f /home/hadoop/data/data.log
a1.sources.r1.shell=/bin/sh -c

# Describe the sink
a1.sinks.k1.type = logger

# Use a channel which buffers events in memory
a1.channels.c1.type = memory
# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1