EMR Hadoop并未利用所有群集节点

Question

We are experimenting with Hadoop and processing of the Common Crawl.

Our problem is that if we create a cluster with 1 Master Node and 1 Core and 2 Task nodes, only one of the nodes per group will get high CPU/Network usage. We tried also with 2 Core and no Task nodes, but in this case also only one Core node was used.

Following some screenshots of the Node/Cluster monitoring. The job was running all the time (in the first two parallel map phases), and should have used most of the available CPU power, as you can see in the screenshot of the working Task node. But why is the idle Task node not utilized?

Our hadoop job, running as an Jar step, has no limits for the map jobs. It consists of multiple map/reduce steps chained. The last reduce job is limited to one Reducer.

Screenshots: https://drive.google.com/drive/folders/1xwABYJMJAC_B0OuVpTQ9LNSj12TtbxI1?usp=sharing

ClusterId: j-3KAPYQ6UG9LU6

StepId: s-2LY748QDLFLM9

We found the following in the System Logs of the idle Node during an other run, maybe it is an EMR problem?

ERROR main: Failed to fetch extraInstanceData from https://aws157-instance-data-1-prod-us-east-1.s3.amazonaws.com/j-2S62KOVL68GVK/ig-3QUKQSH7YJIAU.json?X-Amz-Algorithm=AWS4-HMAC-SHA256&X

Greetings Lukas

Answer 1

Late to the party, but have you tried setting these properties as part of the spark submit command.

--conf 'spark.dynamicAllocation.enabled=true' 
--conf 'spark.dynamicAllocation.minExecutors=<MIN_NO_OF_CORE_OR_TASK_NODES_YOU_WANT>'