从卡夫卡到带有笛子的hdfs仅一个文件

Question

I'm trying to put data in hdfs from kafka via flume. The kafka_producer sends a message every 10 seconds. I'd to collect all messages in one file on hdfs. This is the configuration of flume that i used, but it stores many files on hdfs (one for message):

agent1.sources.kafka-source.type = org.apache.flume.source.kafka.KafkaSource
agent1.sources.kafka-source.zookeeperConnect = localhost:2181
agent1.sources.kafka-source.topic = prova
agent1.sources.kafka-source.groupId = flume
agent1.sources.kafka-source.channels = memory-channel
agent1.sources.kafka-source.interceptors = i1
agent1.sources.kafka-source.interceptors.i1.type = timestamp
agent1.sources.kafka-source.kafka.consumer.timeout.ms = 100
agent1.channels.memory-channel.type = memory
agent1.channels.memory-channel.capacity = 10000
agent1.channels.memory-channel.transactionCapacity = 1000
agent1.sinks.hdfs-sink.type = hdfs
agent1.sinks.hdfs-sink.hdfs.path = hdfs://localhost:9000/input
agent1.sinks.hdfs-sink.hdfs.rollInterval = 5
agent1.sinks.hdfs-sink.hdfs.rollSize = 0
agent1.sinks.hdfs-sink.hdfs.rollCount = 0
agent1.sinks.hdfs-sink.hdfs.fileType = DataStream
agent1.sinks.hdfs-sink.channel = memory-channel
agent1.sources = kafka-source
agent1.channels = memory-channel
agent1.sinks = hdfs-sink

PS I start from a file.csv. The kafka producer takes the file and select some fields of interest, then sends the entries one at time, every 10 seconds. Flume stores the entries on hadoop hdfs, but in many files (1 entry=1 file). I would like that all the entries are in one only file. how have to change the configuration of flume?

Answer 1

It appears that flume is indeed currently set up to create one file on HDFS for each input file.

As suggested here you could deal with this by writing a periodic pig (or mapreduce) job that takes all the input files and combines them.

An additional option to reduce the number of files, may be to reduce the frequency of inbound files.

Answer 2

Establish rollInterval to 0, as you don't want to make different files based on time. If you want to make it based on number entry-s or events, change rollCount value. For example if you want to save 10 events or entry-s in one single file:

agent1.sinks.hdfs-sink.hdfs.rollInterval = 0
agent1.sinks.hdfs-sink.hdfs.rollSize = 0
agent1.sinks.hdfs-sink.hdfs.rollCount = 10