将 requirements.txt 传递给 Google Cloud Pyspark 批处理作业

Question

I am trying to run a pyspark script as through a Google Dataproc Batch Job.

My script should connect to firestore to collect some data from there, so I need to access the library firebase-admin . When I run the script on Google Cloud through the following command:

gcloud dataproc batches submit \
        --project {PROJECT} \
        --region europe-west1 \
        --subnet {SUBNET} \
        pyspark spark_image_matching/main.py \
        --jars=gs://spark-lib/bigquery/spark-bigquery-latest_2.12.jar \
        --deps-bucket={DEPS_BUCKET} 

I receive the following error:

Traceback (most recent call last):
  File "/tmp/srvls-batch-0127aaf6-a438-4439-af56-beb1a66f45ed/main.py", line 4, in <module>
    import firebase_admin
ModuleNotFoundError: No module named 'firebase_admin'

I already tried creating a setup.py file to generate an.egg file that specifies the dependency along with the --py-files flag. This idea was highly inspired by this repo:

http://www.restez-en-bonne-sante-leh.com/?_=%2FGoogleCloudPlatform%2Fdataproc-templates%2Fblob%2Fmain%2Fpython%2Fsetup.py%23BQyskaWdLgo6VQOkV2YyLaeS

Answer 1

To customize Dataproc Serverless for Spark execution environment it is recommended to use custom container images: https://cloud.google.com/dataproc-serverless/docs/guides/custom-containers

As an alternative you can take a look at Spark-supported ways of managing Python dependencies: https://spark.apache.org/docs/latest/api/python/user_guide/python_packaging.html