今天接到客户是咨询,要对现有的数据进行分流,虽然感觉flume实现没问题,但是心里总不踏实,要是出什么幺蛾子就打脸了,随后做了一个小测试,测试的结果也验证了我的想法,废话不多说,下面直接上配置:
#配置文件名为:flume-test.conf
# list the sources, sinks and channels in the agent
a1.sources= source_1
a1.sinks= fs_sink_local_file_1 fs_sink_local_file_2
a1.channels= fs_channel_memory_1 fs_channel_memory_2
# source from spooldir
a1.sources.source_1.type = spooldir
a1.sources.source_1.spoolDir = /home/users/flume/data
a1.sources.source_1.fileHeader = true
#channel for source configure
a1.sources.source_1.channels= fs_channel_memory_1 fs_channel_memory_2
##sinks
#sink1
a1.sinks.fs_sink_local_file_1.channel = fs_channel_memory_1
a1.sinks.fs_sink_local_file_1.type = file_roll
a1.sinks.fs_sink_local_file_1.sink.directory = /home/users/flume/dataA
#sink2
a1.sinks.fs_sink_local_file_2.channel = fs_channel_memory_2
a1.sinks.fs_sink_local_file_2.type = file_roll
a1.sinks.fs_sink_local_file_2.sink.directory = /home/users/flume/dataB
###channels
#channels1
a1.channels.fs_channel_memory_1.type = memory
a1.channels.fs_channel_memory_1.capacity = 5000000
a1.channels.fs_channel_memory_1.transactionCapacity = 5000000
a1.channels.fs_channel_memory_1.byteCapacityBufferPercentage = 40
a1.channels.fs_channel_memory_1.byteCapacity = 8000000000
a1.channels.fs_channel_memory_1.keep-alive = 30
#channels2
a1.channels.fs_channel_memory_2.type = memory
a1.channels.fs_channel_memory_2.capacity = 5000000
a1.channels.fs_channel_memory_2.transactionCapacity = 5000000
a1.channels.fs_channel_memory_2.byteCapacityBufferPercentage = 40
a1.channels.fs_channel_memory_2.byteCapacity = 8000000000
a1.channels.fs_channel_memory_2.keep-alive = 30
可以直接拿去贴就可以跑的配置,启动命令如下:
flume-ng agent -n a1 -f /path/to/conf/flume-test.conf -Xmx100M
以上flume分流复制实现的功能为:把 /home/users/flume/data目录下的文件分别复制到/home/users/flume/dataA和/home/users/flume/dataB,既然这个功能能这样实现,那么扩展到其他source,sink和channel都是没问题的,
ok,今天就到这,
小小菜鸟,不喜勿喷