flume自定义sink source

flume

#定义source、channel、sink
a1.sources = s1
a1.channels = c1

a1.sinks = k1

#分别配置sources，channels，sinks
a1.sources.s1.type = spooldir
a1.sources.s1.spoolDir = /opt/performance/timeData/1476962511

a1.sources.s1.logPattern = *.log

a1.sinks.k1.type = file_roll

a1.sinks.k1.sink.directory = /opt/performance/timeData/flume

a1.channels.c1.type = memory
a1.channels.c1.capacity = 100000
a1.channels.c1.transactionCapacity = 10000

#配置source和sink 的channel

a1.sources.s1.channels = c1

a1.sinks.k1.channel = c1 #没有s 应为一个sink只能对一个channel

jar包依赖：

自定义jar包处理：

1、自定义java文件打包后，直接放在flume的lib包下

2、在flume-conf.properties中配置自定义type(包全称，否则找不到type包)

自定义sink：

extends AbstractSink implentments Configurable

实现public Status process() throws EventDeliveryException()  //实现event处理的核心程序

        public void configure(Contect context)  //从配置文件中读取配置参数

小demo:

@Override
public Status process() throws EventDeliveryException {
Status status = null;

Channel ch = getChannel();
Transaction txn = ch.getTransaction();  //flume的source+sink是在同一个事物中进行的，为了稳定性考虑
txn.begin();
try {
Event event = ch.take();
System.out.println(pro.get("ip")+" "+pro.getProperty("port")+" "+event);

txn.commit();
status = Status.READY;
} catch (Throwable t) {
txn.rollback();
status = status.BACKOFF;
if(t instanceof Error){
throw t;
}
}finally{
txn.close();
}
return status;
}


@Override
public void configure(Context context) {
//读取配置信息
String ip = context.getString("ip", "192.168.184.46");
String port = context.getString("port", "8086");

pro.setProperty("ip", ip);
pro.setProperty("port", port);

}

自定义source:

extends AbstractSource implements Configurable EventDrivenSource(后者其他的处理event接口)

关键是在核心进程中处理event ：getChannelProcessor().processEvent(EventBuilder.withBody(readDataBytes,headers))