Exception in thread "main" java.lang.OutOfMemoryError: Java heap space in flume

Question

Issue : Exception in thread "main" java.lang.OutOfMemoryError: Java heap space in flume.

Executing COMMAND :

flume-ng agent --conf conf 
--conf-file /usr/local/flume/conf/twitter.conf --name TwitterAgent
-Dflume.root.logger=DEBUG.console

Output:

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space

Conf File config file

Flume env.sh flume-env.sh

Answer 1

I compared the config that you are using with the example here , and I can't see any mistakes. The only differences I could see are:

• you used a smaller value for TwitterAgent.channels.MemChannel.capacity
• you used a smaller for TwitterAgent.sinks.HDFS.hdfs.batchsize and TwitterAgent.sinks.HDFS.hdfs.rollcount
• the TwitterAgent.sinks.HDFS.hdfs.path is different.

I wouldn't have thought that any of those would increase memory utilisation.

The other thing to check would be whether data is actually flowing into HDFS. The same source says that you should use the Hadoop Administration Web UI to check that data is flowing into Hadoop. I imagine if that if data is not flowing, then it could be accumulating in the heap of the flume-ng application. If too much data accumulates, you will eventually overflow the heap. No matter what you set the heap size too.

Possible reasons why the data might not be flowing include:

• Hadoop is not running.
• Hadoop is running on a different host (or IP) and/or port to the one that you have specified in your flume config file.