使用 setup.py 在数据流中运行 apache 光束作业时出现 ModuleNotFoundError

Question

I'm having an apache beam pipeline that I used to submit to google dataflow and it runs successfully. By time, my code keep growing and I want to structure it into multiple file dependencies. That's why I referred to apache beam section Multiple File Dependencies

When I structured my code as follows:

​root_dir/
  ​setup.py
  ​main.py
  __init__.py
  ​extract/
    __init__.py
    extract.py

When I execute it in local, it runs good, when I submit it to dataflow, I receive the following error:

ModuleNotFoundError: No module named 'extract'

My setup.py looks like this:

from setuptools import setup, find_packages

setup(
    name="g_dataflow",
    version="0.1.0",
    install_requires=[
        'google-cloud-storage==1.42.0'
    ],
    packages=find_packages()

)

I tried to follow the Juliaset example by apache beam, but with no success. Has anyone faced the same issue before?

Answer 1

我不得不将--save_main_session添加到我的命令行，这解决了这个问题