dataflow error with sdk_container_image apache/beam_python3.9_sdk:2.40.0

Question

I spent a while trying to deploy a custom container to Google Cloud Dataflow (in python). Eventually I scoped my bug down to this minimal example:

python -m apache_beam.examples.wordcount \
--output gs://<your-ouput-dir> \
--runner=DataflowRunner \
--project=<your-project-id> \
--region us-central1 \
--temp_location=gs://<your-tmp-location> \
--worker_harness_container_image=apache/beam_python3.9_sdk:2.40.0 \
--experiment=use_runner_v2

Many variants of this job failed, always with the error log

Error message from worker: Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py", line 285, in loads
    return dill.loads(s)
  File "/usr/local/lib/python3.9/site-packages/dill/_dill.py", line 275, in loads
    return load(file, ignore, **kwds)
  File "/usr/local/lib/python3.9/site-packages/dill/_dill.py", line 270, in load
    return Unpickler(file, ignore=ignore, **kwds).load()
  File "/usr/local/lib/python3.9/site-packages/dill/_dill.py", line 472, in load
    obj = StockUnpickler.load(self)
  File "/usr/local/lib/python3.9/site-packages/dill/_dill.py", line 462, in find_class
    return StockUnpickler.find_class(self, module, name)
AttributeError: Can't get attribute '_create_code' on <module 'dill._dill' from '/usr/local/lib/python3.9/site-packages/dill/_dill.py'>

Ultimately I found a solution: downgrading the sdk_container_image. Specifically this job completed successfully

python -m apache_beam.examples.wordcount \
--output gs://<your-ouput-dir> \
--runner=DataflowRunner \
--project=<your-project-id> \
--region us-central1 \
--temp_location=gs://<your-tmp-location> \
--worker_harness_container_image=apache/beam_python3.9_sdk:2.38.0 \
--experiment=use_runner_v2

From here I was able to build a custom worker harness (by pip installing apache-beam[gcp]==2.38.0 , and copying the scripts in /opt/apache/beam from the apache/beam_python3.9_sdk:2.38.0 docker image). Simply downgrading was a bit unsatisfying. Has anyone encountered this StockUnpickler bug, and found a way around it on apache-beam 2.40.0?

Answer 1

There is a common issue in this case, are you sure the runner of your Dataflow job (virtual env) has the same versions of Apache Beam Python, used by the Docker image (2.40.0)?

The runner instantiates the Dataflow job and the Docker image is used by workers.

To work well the versions used by the runner should be the same as the versions used in the Docker image.

It's a little detail, but for the recent versions of Beam, you can use the sdk_container_image param instead of worker_harness_container_image .