Flume Configuration file_roll

Question

I have a Problem with Flume and/or log4j. I have JavaEE application with log4j and flume appender = everything is fine and works.

When I run flume with the following configuration everything is still fine.

Here the flume conf file

#define components of agent
a1.sources = avroSource
a1.channels = ch1
a1.sinks = hdfsSink
#define source
a1.sources.avroSource.type = avro
a1.sources.avroSource.bind = localhost
a1.sources.avroSource.port = 41415
a1.sources.avroSource.channels = ch1
#define interceptors
a1.sources.avroSource.interceptors = i1 i2 i3 i4 i5
a1.sources.avroSource.interceptors.i1.type = timestamp
a1.sources.avroSource.interceptors.i2.type = host
a1.sources.avroSource.interceptors.i3.type = regex_filter
a1.sources.avroSource.interceptors.i4.type = regex_filter
a1.sources.avroSource.interceptors.i5.type = static
a1.sources.avroSource.interceptors.i2.useIP = false
a1.sources.avroSource.interceptors.i2.hostHeader = host
a1.sources.avroSource.interceptors.i3.regex = .*AppLogging.*
a1.sources.avroSource.interceptors.i3.excludeEvents = false
a1.sources.avroSource.interceptors.i4.regex = .*;ERORR;.*
a1.sources.avroSource.interceptors.i4.excludeEvents = true
a1.sources.avroSource.interceptors.i5.key = instance
a1.sources.avroSource.interceptors.i5.value = JBT_TEST
#define channel
a1.channels.ch1.type = memory
a1.channels.ch1.capacity = 1000
a1.channels.ch1.transactionCapacity = 100
#define sink
a1.sinks.hdfsSink.type = hdfs
a1.sinks.hdfsSink.channel = ch1
a1.sinks.hdfsSink.hdfs.writeFormat = Text
a1.sinks.hdfsSink.hdfs.filePrefix =  %{host}
a1.sinks.hdfsSink.hdfs.fileType = DataStream
a1.sinks.hdfsSink.hdfs.path = hdfs://192.168.79.128:9000/hdfs/mr/input/flume/%{instance}/
a1.sinks.hdfsSink.hdfs.useLocalTimeStamp = true
a1.sinks.hdfsSink.hdfs.rollInterval = 0
a1.sinks.hdfsSink.hdfs.rollCount = 0
a1.sinks.hdfsSink.hdfs.rollSize = 0
a1.sinks.hdfsSink.hdfs.batchSize = 1

If I change the sink from hdfs to file_roll the flume-agent creates log-files but without any content. Here the new conf-file from flume.

#define components of agent
a1.sources = avroSource
a1.channels = ch1
a1.sinks = file

#define source
a1.sources.avroSource.type = avro
a1.sources.avroSource.bind = localhost
a1.sources.avroSource.port = 41415
a1.sources.avroSource.channels = ch1

#define channel
a1.channels.ch1.type = memory
a1.channels.ch1.capacity = 1000
a1.channels.ch1.transactionCapacity = 100

#define sink
a1.sinks.file.type = file_roll
a1.sinks.file.channel = ch1
a1.sinks.file.sink.directory = c:/temp/
a1.sinks.file.sink.rollInterval = 600

What is my mistake?

Answer 1

Add the below two lines at the end of your configuration to connect source and sink with channel. Let me know if this works.

a1.sources.avroSource.channels = ch1
a1.sinks.file.channel = ch1