通过--config时，火花提交失败

Question

I am trying to pass config information to Spark in Amazon EMR as below

spark-submit --jars "/home/hadoop/transfer_cluster/run_spark/spark_jars/jars/trove-3.0.2.jar" --class SparkPTE bin/pte_sc.jar arabic_undirected -–conf spark.yarn.nodemanager.vmem-check-enabled=false

But i am getting the below error as spark cannot parse my config information.

18/04/06 07:48:22 INFO YarnClientSchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.8
Exception in thread "main" java.lang.NumberFormatException: For input string: "-–conf"
    at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
    at java.lang.Integer.parseInt(Integer.java:580)
    at java.lang.Integer.parseInt(Integer.java:615)
    at SparkPTE.sparkContext(SparkPTE.java:91)
    at SparkPTE.main(SparkPTE.java:79)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:775)
    at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
    at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:119)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

If i give --config before --jar i get the below error.

spark-submit -–conf spark.yarn.nodemanager.vmem-check-enabled=false --jars "/home/hadoop/transfer_cluster/run_spark/spark_jars/jars/trove-3.0.2.jar" --class SparkPTE bin/pte_sc.jar arabic_undirected


Error: Unrecognized option: -–conf

The below worked for me

spark-submit --conf spark.yarn.nodemanager.vmem-check-enabled=false --jars "/home/hadoop/transfer_cluster/run_spark/spark_jars/jars/trove-3.0.2.jar" --class SparkPTE bin/pte_sc.jar arabic_undirected

Answer 1

You need to provide the --conf option before your jar name which you're trying to run. This is because whatever you write after your jar name would go as arguments to that jar.