许多死亡执行者 EMR

Question

我正在尝试通过创建一个步骤 Application Spark 在 AWS EMR 集群上执行我的 spark scala 应用程序。

我的集群包含 4 m3.xlarge

我使用以下命令启动我的应用程序：

spark-submit --deploy-mode cluster --class Main s3://mybucket/myjar_2.11-0.1.jar s3n://oc-mybucket/folder arg1 arg2

我的应用程序有 3 个参数，第一个是文件夹。

不幸的是，在启动应用程序后，我看到只有一个 Executor（+master）处于活动状态，而我有 3 个 Executor 死了，所以所有任务都只在第一个执行。 看图片

我尝试了很多方法来激活那些执行器，但没有任何结果（“spark.default.parallelism”、““spark.executor.instances”和“spark.executor.cores”）。 我应该怎么做才能让所有执行者都处于活动状态并处理数据？

另外，当查看 Ganglia 时，我的 cpu 总是低于 35%，有没有办法让 cpu 工作超过 75%？

谢谢你

UPDTAE

这是死执行者的标准错误内容

SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/mnt/yarn/usercache/hadoop/filecache/14/__spark_libs__3671437061469038073.zip/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/lib/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
20/08/15 23:28:56 INFO CoarseGrainedExecutorBackend: Started daemon with process name: 14765@ip-172-31-39-255
20/08/15 23:28:56 INFO SignalUtils: Registered signal handler for TERM
20/08/15 23:28:56 INFO SignalUtils: Registered signal handler for HUP
20/08/15 23:28:56 INFO SignalUtils: Registered signal handler for INT
20/08/15 23:28:57 INFO SecurityManager: Changing view acls to: yarn,hadoop
20/08/15 23:28:57 INFO SecurityManager: Changing modify acls to: yarn,hadoop
20/08/15 23:28:57 INFO SecurityManager: Changing view acls groups to: 
20/08/15 23:28:57 INFO SecurityManager: Changing modify acls groups to: 
20/08/15 23:28:57 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(yarn, hadoop); groups with view permissions: Set(); users  with modify permissions: Set(yarn, hadoop); groups with modify permissions: Set()
20/08/15 23:28:58 INFO TransportClientFactory: Successfully created connection to ip-172-31-36-83.eu-west-1.compute.internal/172.31.36.83:37115 after 186 ms (0 ms spent in bootstraps)
20/08/15 23:28:58 INFO SecurityManager: Changing view acls to: yarn,hadoop
20/08/15 23:28:58 INFO SecurityManager: Changing modify acls to: yarn,hadoop
20/08/15 23:28:58 INFO SecurityManager: Changing view acls groups to: 
20/08/15 23:28:58 INFO SecurityManager: Changing modify acls groups to: 
20/08/15 23:28:58 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(yarn, hadoop); groups with view permissions: Set(); users  with modify permissions: Set(yarn, hadoop); groups with modify permissions: Set()
20/08/15 23:28:58 INFO TransportClientFactory: Successfully created connection to ip-172-31-36-83.eu-west-1.compute.internal/172.31.36.83:37115 after 2 ms (0 ms spent in bootstraps)
20/08/15 23:28:58 INFO DiskBlockManager: Created local directory at /mnt1/yarn/usercache/hadoop/appcache/application_1597532473783_0002/blockmgr-d0d258ba-4345-45d1-9279-f6a97b63f81c
20/08/15 23:28:58 INFO DiskBlockManager: Created local directory at /mnt/yarn/usercache/hadoop/appcache/application_1597532473783_0002/blockmgr-e7ae1e29-85fa-4df9-acf1-f9923f0664bc
20/08/15 23:28:58 INFO MemoryStore: MemoryStore started with capacity 2.6 GB
20/08/15 23:28:59 INFO CoarseGrainedExecutorBackend: Connecting to driver: spark://CoarseGrainedScheduler@ip-172-31-36-83.eu-west-1.compute.internal:37115
20/08/15 23:28:59 INFO CoarseGrainedExecutorBackend: Successfully registered with driver
20/08/15 23:28:59 INFO Executor: Starting executor ID 3 on host ip-172-31-39-255.eu-west-1.compute.internal
20/08/15 23:28:59 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 40501.
20/08/15 23:28:59 INFO NettyBlockTransferService: Server created on ip-172-31-39-255.eu-west-1.compute.internal:40501
20/08/15 23:28:59 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
20/08/15 23:29:00 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(3, ip-172-31-39-255.eu-west-1.compute.internal, 40501, None)
20/08/15 23:29:00 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(3, ip-172-31-39-255.eu-west-1.compute.internal, 40501, None)
20/08/15 23:29:00 INFO BlockManager: external shuffle service port = 7337
20/08/15 23:29:00 INFO BlockManager: Registering executor with local external shuffle service.
20/08/15 23:29:00 INFO TransportClientFactory: Successfully created connection to ip-172-31-39-255.eu-west-1.compute.internal/172.31.39.255:7337 after 20 ms (0 ms spent in bootstraps)
20/08/15 23:29:00 INFO BlockManager: Initialized BlockManager: BlockManagerId(3, ip-172-31-39-255.eu-west-1.compute.internal, 40501, None)
20/08/15 23:29:03 INFO CoarseGrainedExecutorBackend: eagerFSInit: Eagerly initialized FileSystem at s3://does/not/exist in 3363 ms
20/08/15 23:30:02 ERROR CoarseGrainedExecutorBackend: RECEIVED SIGNAL TERM
20/08/15 23:30:02 INFO DiskBlockManager: Shutdown hook called
20/08/15 23:30:02 INFO ShutdownHookManager: Shutdown hook called

这个问题必须与memory有关吗？

Answer 1

spark-submit默认不使用所有的执行器，你可以指定执行器的数量--num-executors ， executor-core和executor-memory 。

例如，增加执行者（默认为2个）

spark-submit --num-executors N   #where N is desired number of executors like 5,10,50

在此处查看文档中的示例

如果它对 spark-submit 没有帮助或覆盖，您可以覆盖conf/spark-defaults.conf文件或类似文件中的spark.executor.instances ，这样您就不必在命令行上明确指定它

对于 CPU 利用率，您应该查看executor-core和executor-core并在 spark-submit 或 conf 中更改它们。 增加 CPU 核心数有望增加使用率。

更新：

正如@Lamanus 所指出的，我仔细检查过，大于 4.4 的 emr 将spark.dynamicAllocation.enabled设置为true ，我建议您仔细检查数据的分区，因为启用动态分配后，执行程序实例的数量取决于数量分区数，根据 DAG 执行的阶段而有所不同。 此外，通过动态分配，您可以尝试spark.dynamicAllocation.initialExecutors 、 spark.dynamicAllocation.maxExecutors 、 spark.dynamicAllocation.maxExecutors来控制执行器。

Answer 2

这可能有点晚了，但我发现这个 AWS 大数据博客很有见地，可以确保我的大部分集群都得到利用，并且我能够实现尽可能多的并行性。

https://aws.amazon.com/blogs/big-data/best-practices-for-successfully-managing-memory-for-apache-spark-applications-on-amazon-emr/

进一步来说：

每个实例的执行器数量 =（每个实例的虚拟核心总数 - 1）/ spark.executors.cores

总执行器 memory = 每个实例的总 RAM / 每个实例的执行器数量

然后，您可以使用spark.default.parallelism或repartitioning控制阶段期间并行任务的数量。