hadoop无法分配内存java.io.IOException：错误= 12

Question

i am getting the following error on hadoop greenplum

java.lang.Throwable: Child Error
    at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:271)
Caused by: java.io.IOException: Cannot run program "ln": java.io.IOException: error=12, Cannot allocate memory
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:488)
    at java.lang.Runtime.exec(Runtime.java:610)
    at java.lang.Runtime.exec(Runtime.java:448)
    at java.lang.Runtime.exec(Runtime.java:386)
    at org.apache.hadoop.fs.FileUtil.symLink(FileUtil.java:567)
    at org.apache.hadoop.mapred.TaskLog.createTaskAttemptLogDir(TaskLog.java:109)
    at org.apache.hadoop.mapred.DefaultTaskController.createLogDir(DefaultTaskController.java:71)
    at org.apache.hadoop.mapred.TaskRunner.prepareLogFiles(TaskRunner.java:316)
    at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:228)
Caused by: java.io.IOException: java.io.IOException: error=12, Cannot allocate memory
    at java.lang.UNIXProcess.<init>(UNIXProcess.java:164)
    at java.lang.ProcessImpl.start(ProcessImpl.java:81)
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:470)
    ... 8 more

the server has 7G ram and 1G swap.

heap size is 1024m and mapred.child.opts is set to 512m.

any ideas?

Answer 1

将tasktracker的内存减少到256M，并将每个节点的tasktracker的数量限制为1，任何更高的值都会导致子错误，并需要更多时间来运行mapreduce作业。

Answer 2

Whatever memory arrangement you come up with, Hadoop is likely to throw this anyway. The problem is that for simple file system tasks like creating symbolic links or checking for available disk space, Hadoop forks a process from the TaskTracker. That process will have as much memory allocated to it as its parent has.

Typical ways to prevent this problem are to leave as much physical memory unallocated as allocated to the TT, adding some swap to the host for these kind of tasks, or allowing "over commit".