Flume source has been removed due to an error in configuration

Question

I had a usecase where I have to cat file and load into local dir using FLUME.I know Flume is not designed for such use cases but I don't have any streaming data.So I decide to go with this and see how the flume works for this I have created my configuration file with sources.type as exec and channels.type as file and sink.types as file_roll and I also configured appropriate configuration for channels like checkpointDirs and dataDir. But, when I run the agent it failed at source saying (flume.log)

FYI conf file

#### Define a source, a channel, and a sink
agent.sources = src1
agent.channels = chan1
agent.sinks = sink1


#### Set the source type to Spooling Directory and set the directory
#### location to /home/flume/ingestion/


agent.sources.src1.type = exec
agent.sources.src1.commands= cat /xxxxx/xxxxxx/Xxxxx/xxxxx/test.dat



#### Configure the channel as simple in-memory queue
agent.channels.chan1.type = file
agent.channels.chan1.checkpointDir = /home/xxxxx/xxxxxxx/flume/checkpoint
agent.channels.chan1.dataDirs= /home/xxxxxxx/xxxxx/flume/data


#### Define the HDFS sink and set its path to your target HDFS directory
agent.sinks.sink1.type = file_roll
agent.sinks.sink1.sink.directory = /home/xxxxx/xxxxxx/xxxxx/flume/files
agent.sinks.sink1.sink.rollInterval = 0
agent.sinks.sink1.sink.rollSize = 0
agent.sinks.sink1.idleTimeout = 0


#### Connect source and sink
agent.sources.src1.channels = chan1
agent.sinks.sink1.channel = chan1

FYI flume log

27 Apr 2017 08:16:33,831 INFO  [lifecycleSupervisor-1-0] (org.apache.flume.node.PollingPropertiesFileConfigurationProvider.start:61)  - Configuration provider starting
27 Apr 2017 08:16:33,849 INFO  [conf-file-poller-0] (org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run:133)  - Reloading configuration file:/usr/lib/gphd/apache-flume-1.4.0-bin/conf/flume-agent1.con
f
27 Apr 2017 08:16:33,859 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)  - Processing:sink1
27 Apr 2017 08:16:33,860 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)  - Processing:sink1
27 Apr 2017 08:16:33,860 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)  - Processing:sink1
27 Apr 2017 08:16:33,861 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:930)  - Added sinks: sink1 Agent: agent
27 Apr 2017 08:16:33,861 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)  - Processing:sink1
27 Apr 2017 08:16:33,861 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)  - Processing:sink1
27 Apr 2017 08:16:33,861 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)  - Processing:sink1
27 Apr 2017 08:16:33,918 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration.validateConfiguration:140)  - Post-validation flume configuration contains configuration for agents: [agent]
27 Apr 2017 08:16:33,919 INFO  [conf-file-poller-0] (org.apache.flume.node.AbstractConfigurationProvider.loadChannels:150)  - Creating channels
27 Apr 2017 08:16:33,934 INFO  [conf-file-poller-0] (org.apache.flume.channel.DefaultChannelFactory.create:40)  - Creating instance of channel chan1 type file
27 Apr 2017 08:16:33,938 INFO  [conf-file-poller-0] (org.apache.flume.node.AbstractConfigurationProvider.loadChannels:205)  - Created channel chan1
27 Apr 2017 08:16:33,939 INFO  [conf-file-poller-0] (org.apache.flume.source.DefaultSourceFactory.create:39)  - Creating instance of source src1, type exec
27 Apr 2017 08:16:33,941 ERROR [conf-file-poller-0] (org.apache.flume.node.AbstractConfigurationProvider.loadSources:366)  - Source src1 has been removed due to an error during configuration
java.lang.IllegalStateException: The parameter command must be specified
        at com.google.common.base.Preconditions.checkState(Preconditions.java:145)
        at org.apache.flume.source.ExecSource.configure(ExecSource.java:221)
        at org.apache.flume.conf.Configurables.configure(Configurables.java:41)
        at org.apache.flume.node.AbstractConfigurationProvider.loadSources(AbstractConfigurationProvider.java:331)
        at org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:102)
        at org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:140)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
27 Apr 2017 08:16:33,944 INFO  [conf-file-poller-0] (org.apache.flume.sink.DefaultSinkFactory.create:40)  - Creating instance of sink: sink1, type: file_roll
27 Apr 2017 08:16:33,952 INFO  [conf-file-poller-0] (org.apache.flume.node.AbstractConfigurationProvider.getConfiguration:119)  - Channel chan1 connected to [sink1]
27 Apr 2017 08:16:33,962 INFO  [conf-file-poller-0] (org.apache.flume.node.Application.startAllComponents:138)  - Starting new configuration:{ sourceRunners:{} sinkRunners:{sink1=SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProc
essor@54560854 counterGroup:{ name:null counters:{} } }} channels:{chan1=FileChannel chan1 { dataDirs: [/home/phdftpuser/dev_phase/flume/data] }} }
27 Apr 2017 08:16:33,976 INFO  [conf-file-poller-0] (org.apache.flume.node.Application.startAllComponents:145)  - Starting Channel chan1
27 Apr 2017 08:16:33,976 INFO  [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.FileChannel.start:282)  - Starting FileChannel chan1 { dataDirs: [/home/phdftpuser/dev_phase/flume/data] }...
27 Apr 2017 08:16:33,996 INFO  [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.Log.<init>:322)  - Encryption is not enabled
27 Apr 2017 08:16:33,998 INFO  [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.Log.replay:369)  - Replay started
27 Apr 2017 08:16:34,000 INFO  [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.Log.replay:381)  - Found NextFileID 0, from []
27 Apr 2017 08:16:34,008 INFO  [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.EventQueueBackingStoreFileV3.<init>:53)  - Starting up with /home/phdftpuser/dev_phase/flume/checkpoint/checkpoint and /home/phdftpuser/dev_phase/flu
me/checkpoint/checkpoint.meta
27 Apr 2017 08:16:34,009 INFO  [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.EventQueueBackingStoreFileV3.<init>:57)  - Reading checkpoint metadata from /home/phdftpuser/dev_phase/flume/checkpoint/checkpoint.meta
27 Apr 2017 08:16:34,031 WARN  [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.Log.replay:442)  - Checkpoint may not have completed successfully. Forcing full replay, this may take a while.
org.apache.flume.channel.file.BadCheckpointException: The checkpoint metadata file does not exist or has zero length
        at org.apache.flume.channel.file.EventQueueBackingStoreFileV3.<init>(EventQueueBackingStoreFileV3.java:61)
        at org.apache.flume.channel.file.EventQueueBackingStoreFactory.get(EventQueueBackingStoreFactory.java:75)
        at org.apache.flume.channel.file.Log.replay(Log.java:412)
        at org.apache.flume.channel.file.FileChannel.start(FileChannel.java:302)
        at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
27 Apr 2017 08:16:34,032 INFO  [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.Serialization.deleteAllFiles:105)  - Skipping in_use.lock because it is in excludes set
27 Apr 2017 08:16:34,039 INFO  [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.Serialization.deleteAllFiles:118)  - Deleted the following files: , checkpoint, checkpoint.meta.
27 Apr 2017 08:16:34,064 INFO  [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.EventQueueBackingStoreFile.<init>:91)  - Preallocated /home/phdftpuser/dev_phase/flume/checkpoint/checkpoint to 8008232 for capacity 1000000
27 Apr 2017 08:16:34,064 INFO  [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.EventQueueBackingStoreFileV3.<init>:53)  - Starting up with /home/phdftpuser/dev_phase/flume/checkpoint/checkpoint and /home/phdftpuser/dev_phase/flu
me/checkpoint/checkpoint.meta
27 Apr 2017 08:16:34,068 ERROR [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.Log.replay:476)  - Failed to initialize Log on [channel=chan1]
java.lang.UnsupportedOperationException: This is supposed to be overridden by subclasses.
        at com.google.protobuf.GeneratedMessage.getUnknownFields(GeneratedMessage.java:180)

I tried several times changing the configuration file but it doesn't turn out any good result.Can someone please help me on this.

Answer 1

java.lang.IllegalStateException: The parameter command must be specified

For exec source, command is a required property. Your configuration file has commands instead.

Modify the configuration as,

agent.sources.src1.type = exec
agent.sources.src1.command = cat /xxxxx/xxxxxx/Xxxxx/xxxxx/test.dat

And these properties do not belong to file_roll sink, remove them.

agent.sinks.sink1.sink.rollSize = 0
agent.sinks.sink1.idleTimeout = 0