Spark UI Showing Wrong Memory Allocation

Question

We are currently running into an issue where Spark is showing that each of our nodes only have 4GB of memory. However, we have allocated 10GB of memory by setting spark-worker.jvmOptions = -Xmx10g . We can not figure out what is causing this unusual limitation/incorrect memory allocation.

When we go to run spark jobs it will run as if there is only 4GB of memory per worker.

Any help would be great! Thanks!

Screenshot of SOLR UI

Answer 1

You should set worker memory using: --executor-memory in your spark-submit

Answer 2

Try setting the following parameters inside the conf/spark-defaults.conf file:

spark.executor.memory           10g