Spark Executor：初始堆大小无效：-Xms0M

Question

I have configured Spark to query on hive table.

Run the Thrift JDBC/ODBC server using below command :

cd $SPARK_HOME
./sbin/start-thriftserver.sh --master spark://myhost:7077 --hiveconf hive.server2.thrift.bind.host=myhost --hiveconf hive.server2.thrift.port=9999

Then checked at Spark worker UI , executor startup failing with below error , JVM initialization failing because of wrong -Xms :

Invalid initial heap size: -Xms0M
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.

Following are changed configurations in conf/spark-env.sh

export SPARK_JAVA_OPTS="-Dspark.executor.memory=512M"
export SPARK_EXECUTOR_MEMORY=1G
export SPARK_DRIVER_MEMORY=512M
export SPARK_WORKER_MEMORY=2G
export SPARK_WORKER_INSTANCES=1

I really don't have clue from where this value -Xms0M is coming or how it has been derived ? Please help me understand issue and change this value.

Answer 1

It working now ...

Thrift server is not picking executor memory from spark-env.sh​ , then I added in thrift server startup script explicitly.

./sbin/start-thriftserver.sh

exec "$FWDIR"/sbin/spark-daemon.sh spark-submit $CLASS 1 --executor-memory 512M "$@"

With this , Executor start getting valid memory and JDBC queries are getting results.

conf/spark-env.sh​ (executor memory configurations not picked by thrift-server)

export SPARK_JAVA_OPTS="-Dspark.executor.memory=512M"
export SPARK_EXECUTOR_MEMORY=512M