9小时后Spark Streaming作业失败并出现此错误

Question

I am getting this error in yarn logs for my Spark Streaming job. Any advice on how to tune it would be helpful.

2018-10-04T23:13:33.496+0000: [GC (Allocation Failure) 2018-10-04T23:13:33.497+0000: [ParNew: 605346K->51088K(629120K), 0.0146063 secs] 4367136K->3824642K(6191580K), 0.0161894 secs] [Times: user=0.16 sys=0.00, real=0.02 secs] 
2018-10-04T23:13:34.501+0000: [GC (Allocation Failure) 2018-10-04T23:13:34.502+0000: [ParNew: 610320K->67154K(629120K), 0.0171408 secs] 4383874K->3854168K(6191580K), 0.0187251 secs] [Times: user=0.21 sys=0.00, real=0.02 secs] 
Heap
 par new generation   total 629120K, used 117091K [0x00007f2c50000000, 0x00007f2c7aaa0000, 0x00007f2cc7990000)
  eden space 559232K,   8% used [0x00007f2c50000000, 0x00007f2c530c4658, 0x00007f2c72220000)
  from space 69888K,  96% used [0x00007f2c72220000, 0x00007f2c763b4868, 0x00007f2c76660000)
  to   space 69888K,   0% used [0x00007f2c76660000, 0x00007f2c76660000, 0x00007f2c7aaa0000)
 concurrent mark-sweep generation total 5562460K, used 3787014K [0x00007f2cc7990000, 0x00007f2e1b1a7000, 0x00007f3b50000000)
 Metaspace       used 106113K, capacity 112552K, committed 129688K, reserved 131072K
End of LogType:stdout

In the driver output this is the error I get.

java.lang.Exception: Could not compute split, block input-1-1538678764402

Answer 1

Do you use Java.Maps/Sets in the job? To me this looks like as you are running out of memory.

If this is the case, you have at least to options:

1) Adjust the memory settings you are using for the Job. The memory per executor can be controlled via the --executor-memory flag. 2) Use a different Map/Set implementation - in case of Java you could switch to Trove

Hope, this helps.