为 AWS EKS 配置 Apache Spark

Question

I'd like to start working with apache spark on k8s but I don't have experience with it. I installed Spark via Helm chart with ServiceType "LoadBalancer".

spark-submit --master 'spark://LOADBALANCER.elb.eu-central-1.amazonaws.com:7077' \ 
--deploy-mode client \
--conf spark.kubernetes.container.image='MY_IMAGE' test.py

This is my test code test.py

from pyspark.sql import SparkSession

spark_session = SparkSession.builder \
    .getOrCreate()
l = [('Alice', 1)]
spark_session.createDataFrame(l).show()

Running locally on microk8s cluster works but the same way on AWS EKS cluster fails with following endless log warning

22/02/16 17:36:01 INFO TaskSchedulerImpl: Adding task set 0.0 with 1 tasks resource profile 0
22/02/16 17:36:16 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources

Is there a way to develop the user code and run it against the kube.netes cluster or should I create a new docker image everytime? Maybe there are some best practices for Apache Spark on EKS?

Answer 1

Try to change to --deploy-mode cluster