如何在 Kubernetes 上向 Spark 发送作业。 无法实例化外部调度程序

Question

I have started a Minikube cluster but when I make spark-submit, job get an error.

My command:

bin/spark-submit --master k8s://https://192.168.99.101:8443 --deploy-mode cluster --name spark-test --class org.apache.spark.examples.SparkPi --conf spark.executor.instances=5 --conf spark.kubernetes.container.image=spark local:///opt/spark/examples/jars/spark-examples_2.11-2.4.0.jar

And I get the following error:

2019-03-13 18:26:57 ERROR SparkContext:91 - Error initializing SparkContext. org.apache.spark.SparkException: External scheduler cannot be instantiated at org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$createTaskScheduler(SparkContext.scala:2794) ...30 more Caused by: io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://kubernetes.default.svc/api/v1/namespaces/default/pods/spark-test-1552501611130-driver . Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. pods "spark-test-1552501611130-driver" is forbidden: User "system:serviceaccount:default:default" cannot get resource "pods" in API group "" in the namespace "default". ... 30 more

Answer 1

You need to create ServiceAccount in Kubernetes. It can be done if you download the YAML config from this link

And then run

kubectl apply -f spark-rbac.yaml

The YAML file will create the namespace and ServiceAccount, which are needed for Spark on Kubernetes. However, you also need to modify your spark-submit command to be like:

bin/spark-submit --master k8s://https://192.168.99.101:8443 --deploy-mode cluster --name spark-test --class org.apache.spark.examples.SparkPi \
--conf spark.kubernetes.namespace=spark \
--conf spark.kubernetes.authenticate.driver.serviceAccountName=spark-sa \
--conf spark.executor.instances=5 \
--conf spark.kubernetes.container.image=spark local:///opt/spark/examples/jars/spark-examples_2.11-2.4.0.jar