hadoop windows（工作正常）linux java堆空间

Question

Here is my problem: First of all I'm working with hadoop and a single node configuration I'm developing an application and I made just one map function, in this map function I call like 10 functions, the application reads from a csv file and process a certain column, I already made the jar file and everything so when I run this app with a csv with 4000 rows on windows (windows 7) (using cygwin) on a 4 GB RAM machine, the application works fine, but when I run it on linux- ubuntu on a 2 GB RAM machine, it process some rows but then it throws a "Java heap space" error, or sometimes the thread is killed.

For the linux: I already tried to change the hadoop export HEAP_SIZE and also the Xmx and Xms parameters on the app and it made some difference but not too much, the error stills happening...

Do you know why it s happening? its because the 4GB and 2GB of RAM difference between machines?

Answer 1

One thing I ran into with a mapper is if you call/use functions/objects that start their own threads from within the map function, this can easily create enough threads to use all the heap space for that JVM. Each mapper will have the setup and cleanup function called once. In my situation I was able to process and put into an ArrayList all my data, then do the additional processing I needed in the cleanup function.