AKS 中的火花。 错误：无法找到或加载主类 org.apache.spark.launcher.Main

Question

Update 1: After adding missing pieces and env variables from Spark installation - Error: Could not find or load main class org.apache.spark.launcher.Main , the command no longer throws an error, but prints itself and doesn't do anything else. This is the new result of running the command:

"C:\Program Files\Java\jdk1.8.0_271\bin\java" -cp "C:\Users\xxx\repos\spark/conf\;C:\Users\xxx\repos\spark\assembly\target\scala-2.12\jars\*" org.apache.spark.deploy.SparkSubmit --master k8s://http://127.0.0.1:8001 --deploy-mode cluster --conf "spark.kubernetes.container.image=xxx.azurecr.io/spark:spark2.4.5_scala2.12.12" --conf "spark.kubernetes.authenticate.driver.serviceAccountName=spark" --conf "spark.executor.instances=3" --class com.xxx.bigdata.xxx.XMain --name xxx_app https://storage.blob.core.windows.net/jars/xxx.jar

I have been following this guide for setting up Spark in AKS: https://docs.microsoft.com/en-us/azure/aks/spark-job . I am using Spark tag 2.4.5 with scala 2.12.12. I have done all the following steps:

• created AKS with ACR and Azure storage, serviceaccount and role
• built spark source
• built docker image and push to ACR
• built sample SparkPi jar and push to storage
• proxied api-server (kubectl proxy) and executed spark-submit:

./bin/spark-submit \
    --master k8s://http://127.0.0.1:8001 \
    --deploy-mode cluster \
    --name xxx_app\
    --class com.xxx.bigdata.xxx.XMain\
    --conf spark.executor.instances=3 \
    --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \
    --conf spark.kubernetes.container.image=xxx.azurecr.io/spark:spark2.4.5_scala2.12.12 \
    "https://storage.blob.core.windows.net/jars/xxx.jar"

All I am getting is Error: Could not find or load main class org.apache.spark.launcher.Main

Now, the funny thing is that it doesn't matter at all what I change in the command. I can mess up ACR address, spark image name, jar location, api-server address, anything, and I still get the same error.

I guess I must be making some silly mistake as it seems nothing can break the command more than it already is, but I can't really nail it down. Does someone have some ideas what might be wrong?

Answer 1

Looks like it might be a problem on the machine you are executing spark-submit . You might be missing some jars on the classpath on the machine you are executing spark-submit . Worth checking out Spark installation - Error: Could not find or load main class org.apache.spark.launcher.Main

Answer 2

Alright, so I managed to submit jobs with spark-submit.cmd, instead. It works, without any additional setup.

I didn't manage to get the bash script to work in the end and I do not have the time to investigate it further at this moment. So, sorry for providing a half-assed answer only partially resolving original problem, but it is a solution nonetheless.

The below command works fine

bin\spark-submit.cmd --master k8s://http://127.0.0.1:8001 --deploy-mode cluster --name spark-pi --class org.apache.spark.examples.SparkPi --conf spark.executor.instances=3 --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark --conf spark.kubernetes.namespace=dev --conf spark.kubernetes.container.image=xxx.azurecr.io/spark:spark-2.4.5_scala-2.12_hadoop-2.7.7  https://xxx.blob.core.windows.net/jars/SparkPi-assembly-0.1.0-SNAPSHOT.jar