hive 上的 tez 錯誤：java.lang.OutOfMemoryError

Question

在具有更多 70 列的 hive 表上按日期執行分區時遇到此錯誤：

ERROR: Status: Failed ERROR: Vertex failed, vertexName=Map 1, vertexId=vertex_1612203694878_0265_4_00, diagnostics=[Task failed, taskId=task_1612203694878_0265_4_00_000058, diagnostics=[TaskAttempt 0 failed, info=[Container container_e16_1612203694878_0265_01_000167 finished with diagnostics set to [Container failed,退出代碼=-104。 [2021-02-02 11:00:58.498]容器 [pid=1577,containerID=container_e16_1612203694878_0265_01_000167] 正在運行 3022848B 超出“物理”memory 限制。 當前使用情況：1GB物理memory使用1.0GB； 使用了 2.7 GB 的 2.1 GB 虛擬 memory。 殺死容器。 container_e16_1612203694878_0265_01_000167的consect-tree的轉儲：| - | - PID PPID PPID PGRPID SESSID CMD_NAME user_mode_time（millis）System_time（MILLIS）SYSTEM_TIME（MILLIS）VMEM_USAGE（BYTES）vmem_USAGE（BYTES）RSSMEM_EMEM_EMEM_EMEM_EMEM_EMEM_EMEM_ESMAGE（BYSSMEM_） -c /usr/jdk64/jdk1.8.0_112/bin/java -Xmx819m -server -Djava.net.preferIPv4Stack=true -Dhdp.version=3.1.4.0-315 -XX:+PrintGCDetails -verbose:gc -XX:+ PrintGCTimeStamps -XX:+UseNUMA -XX:+UseG1GC -XX:+ResizeTLAB -server -Djava.net.preferIPv4Stack=true -XX:NewRatio=8 -XX:+UseNUMA -XX:+UseG1GC -XX:+ResizeTLAB -XX: +PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=/usr /hadoop/yarn/log/application_1612203694878_0265/container_e16_1612203694878_0265_01_000167 -Dtez.root.logger=INFO,CLA -Djava.io.ya//localuserca che/hive/appcache/application_1612203694878_0265/container_e16_1612203694878_0265_01_000167/tmp org.apache.tez.runtime.task.TezChild slave-06-n.fawryhq.corp 43250 container_e16_1612203694878_0265_01_000167 application_1612203694878_0265 1 1>/usr/hadoop/yarn/log/application_1612203694878_0265/container_e16_1612203694878_0265_01_000167/ stdout 2>/usr/hadoop/yarn/log/application_1612203694878_0265/container_e16_1612203694878_0265_01_000167/stderr |- 1658 1577 1577 1577 (java) 1414 128 2788896768 262581 /usr/jdk64/jdk1.8.0_112/bin/java -Xmx819m -server -Djava. net.preferIPv4Stack=true -Dhdp.version=3.1.4.0-315 -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseG1GC -XX:+ResizeTLAB -server -Djava.net .preferIPv4Stack=true -XX:NewRatio=8 -XX:+UseNUMA -XX:+UseG1GC -XX:+ResizeTLAB -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -Dlog4j.configuratorClass=org.apache.tez。 common.TezLog4jConfigurator -Dlog4j.config uration=tez-container-log4j.properties -Dyarn.app.container.log.dir=/usr/hadoop/yarn/log/application_1612203694878_0265/container_e16_1612203694878_0265_01_000167 -Dtez.root.logger=INFO,CLA -Djava.io.tmpdir=/ usr/hadoop/yarn/local/usercache/hive/appcache/application_1612203694878_0265/container_e16_1612203694878_0265_01_000167/tmp org.apache.tez.runtime.task.TezChild slave-06-n.fawryhq.corp 43250 container_e16_1612203694878_0265_01_000167 application_1612203694878_0265 1 [2021-02-02 11 :00:58.512]容器應要求被殺死。 退出代碼為 143 [2021-02-02 11:00:58.521]容器以非零退出代碼 143 退出。]]，TaskAttempt 1 失敗，info=[錯誤：運行任務時出錯（失敗）：java.lang .OutOfMemoryError: Java heap space at java.nio.HeapByteBuffer.(HeapByteBuffer.java:57) at java.nio.ByteBuffer.allocate(ByteBuffer.java:335) at org.apache.tez.runtime.library.common.sort. impl.PipelinedSorter.allocateSpace(PipelinedSorter.java:256) at org.apache.tez.runtime.library.common.sort.impl.PipelinedSorter.(PipelinedSorter.java:205) at org.apache.tez.runtime.library.output .OrderedPartitionedKVOutput.start(OrderedPartitionedKVOutput.Z93F725A07423FE1C889F448B33D21 F46Z:146) at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:193) at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor. java:266) at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250) at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374) at org .apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73) at org.ZB6EFD606D118D0F6206D118D0F62066E31419Task.0TaskRunner2Callable.java:73) allable$1.run(TaskRunner2Callable.java:61) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation .doAs(UserGroupInformation.java:1730) at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61) at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37 ) at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36) at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(Tr ustedListenableFutureTask.java:125) at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69) at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78) at java .util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745), errorMessage=Cannot recover from this error:java.lang.OutOfMemoryError: Java heap space at java.nio.HeapByteBuffer.(HeapByteBuffer.Z93F725A074 23FE1C889F448B33D21F46Z:57) at java.nio.ByteBuffer.allocate(ByteBuffer.java:335) at org.apache.tez.runtime.library.common.sort.impl.PipelinedSorter.allocateSpace(PipelinedSorter.java:256) at org.apache .tez.runtime.library.common.sort.impl.PipelinedSorter.(PipelinedSorter.java:205) at org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput.start(OrderedPartitionedKVOutput.java:146) at org.apache. hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:193) at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.i nitializeAndRunProcessor(TezProcessor.java:266) at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250) at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java: 374) at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73) at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61) at java.security .AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.Z0238775C7BD96E2EAB9Z98security.Group. .doAs(UserGroupInformation.java:1730) at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61) at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37 ) at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36) at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125) at com.google.common. util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69) 在 com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.Z93F72507233 E1C889F448B33D21F46Z:78) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread .java:745) ]], Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:17, Vertex vertex_1612203694878_0265_4_00 [Map 1] killed/failed due to:OWN_TASK_FAILURE] ERROR: Vertex killed, vertexName=Reducer 2, vertexId=vertex_1612203694878_0265_4_01 , diagnostics=[Vertex 在 RUNNING state 中收到 Kill。，由於 OTHER_VERTEX_FAILURE，Vertex 未成功，failedTasks：0killedTasks：2，Vertex vertex_1612203694878_0265_4_01 [Reducer 2] 由於：DILAGVERTEX_ERROR 而殺死/失敗] VERTEX_FAILURE。 失敗頂點：1 殺死頂點：1

Answer 1

嘗試（按此順序）

1. 增加映射器並行度。 目標是獲得更多更小的映射器。 檢查它啟動了多少個映射器並調整了數字。 如果您有太大的文件（如 gzip）的不可拆分格式，這將不起作用，然后繼續接下來的兩個步驟。

    --This is example, check your current setings and adjust to get x2 or more mappers set hive.input.format=org.apache.hadoop.hive.ql.io.HiveInputFormat; set tez.grouping.max-size=32000000; --bigger files will be splitted set tez.grouping.min-size=32000; --smaller files will be combined on single mapper

2. 禁用地圖端聚合（地圖端聚合往往會導致OOM）

    set hive.map.aggr=false;

3. 如果上述兩個步驟沒有幫助，則增加映射器 memory（嘗試找到最小工作容器大小）

    set hive.tez.container.size=9216; --Adjust figures and chose minimum working size set hive.tez.java.opts=-Xmx6144m;