Dataflow Flex模板官方教程用不了

Question

我正在尝试重现本教程以在 Dataflow 上运行 Flex 模板。

提交作业几分钟后，它失败并出现以下错误：

Output from execution of subprocess: b'Collecting apache-beam[gcp]==2.26.0
 Downloading apache-beam-2.26.0.zip (2.4 MB)
 Saved /tmp/dataflow-requirements-cache/apache-beam-2.26.0.zip
Collecting avro-python3!=1.9.2,<1.10.0,>=1.8.1
 Downloading avro-python3-1.9.2.1.tar.gz (37 kB)
 Saved /tmp/dataflow-requirements-cache/avro-python3-1.9.2.1.tar.gz
Collecting crcmod<2.0,>=1.7
 Downloading crcmod-1.7.tar.gz (89 kB)
 Saved /tmp/dataflow-requirements-cache/crcmod-1.7.tar.gz
Collecting dill<0.3.2,>=0.3.1.1
 Downloading dill-0.3.1.1.tar.gz (151 kB)
 Saved /tmp/dataflow-requirements-cache/dill-0.3.1.1.tar.gz
Collecting fastavro<2,>=0.21.4
 Downloading fastavro-1.2.2.tar.gz (663 kB)
 Saved /tmp/dataflow-requirements-cache/fastavro-1.2.2.tar.gz
Collecting future<1.0.0,>=0.18.2
 Downloading future-0.18.2.tar.gz (829 kB)
 Saved /tmp/dataflow-requirements-cache/future-0.18.2.tar.gz
Collecting grpcio<2,>=1.29.0
 Downloading grpcio-1.34.0.tar.gz (21.0 MB)
 Saved /tmp/dataflow-requirements-cache/grpcio-1.34.0.tar.gz
Collecting hdfs<3.0.0,>=2.1.0
 Downloading hdfs-2.5.8.tar.gz (41 kB)
 Saved /tmp/dataflow-requirements-cache/hdfs-2.5.8.tar.gz
Collecting httplib2<0.18.0,>=0.8
 Downloading httplib2-0.17.4.tar.gz (256 kB)
 Saved /tmp/dataflow-requirements-cache/httplib2-0.17.4.tar.gz
Collecting mock<3.0.0,>=1.0.1
 Downloading mock-2.0.0.tar.gz (73 kB)
 Saved /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting numpy<2,>=1.14.3
 Downloading numpy-1.19.4.zip (7.3 MB)
 Installing build dependencies: started
 Installing build dependencies: still running...
 Installing build dependencies: finished with status 'done'
 Getting requirements to build wheel: started
 Getting requirements to build wheel: finished with status 'done'
 Preparing wheel metadata: started
 Preparing wheel metadata: finished with status 'done'
 Saved /tmp/dataflow-requirements-cache/numpy-1.19.4.zip
Collecting pymongo<4.0.0,>=3.8.0
 Downloading pymongo-3.11.2.tar.gz (770 kB)
 Saved /tmp/dataflow-requirements-cache/pymongo-3.11.2.tar.gz
Collecting oauth2client<5,>=2.0.1
 Downloading oauth2client-4.1.3.tar.gz (155 kB)
 Saved /tmp/dataflow-requirements-cache/oauth2client-4.1.3.tar.gz
Collecting protobuf<4,>=3.12.2
 Downloading protobuf-3.14.0.tar.gz (227 kB)
 Saved /tmp/dataflow-requirements-cache/protobuf-3.14.0.tar.gz
Collecting pyarrow<0.18.0,>=0.15.1
 Downloading pyarrow-0.17.1.tar.gz (2.6 MB)
 Installing build dependencies: started
 Installing build dependencies: still running...
 Installing build dependencies: still running...
 Installing build dependencies: still running...
 Installing build dependencies: finished with status 'done'
 Getting requirements to build wheel: started
 Getting requirements to build wheel: finished with status 'done'
 Preparing wheel metadata: started
 Preparing wheel metadata: finished with status 'error'
 ERROR: Command errored out with exit status 1:
 command: /usr/local/bin/python /usr/local/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /tmp/tmp_p9la_sd
 cwd: /tmp/pip-download-uramyun1/pyarrow
 Complete output (42 lines):
 Traceback (most recent call last):
 File "/usr/local/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py", line 280, in <module>
 main()
 File "/usr/local/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py", line 263, in main
 json_out['return_val'] = hook(**hook_input['kwargs'])
 File "/usr/local/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py", line 133, in prepare_metadata_for_build_wheel
 return hook(metadata_directory, config_settings)
 File "/tmp/pip-build-env-5__v4cuc/overlay/lib/python3.7/site-packages/setuptools/build_meta.py", line 161, in prepare_metadata_for_build_wheel
 self.run_setup()
 File "/tmp/pip-build-env-5__v4cuc/overlay/lib/python3.7/site-packages/setuptools/build_meta.py", line 254, in run_setup
 self).run_setup(setup_script=setup_script)
 File "/tmp/pip-build-env-5__v4cuc/overlay/lib/python3.7/site-packages/setuptools/build_meta.py", line 145, in run_setup
 exec(compile(code, __file__, 'exec'), locals())
 File "setup.py", line 605, in <module>
 url='https://arrow.apache.org/'
 File "/tmp/pip-build-env-5__v4cuc/overlay/lib/python3.7/site-packages/setuptools/__init__.py", line 153, in setup
 return distutils.core.setup(**attrs)
 File "/usr/local/lib/python3.7/distutils/core.py", line 108, in setup
 _setup_distribution = dist = klass(attrs)
 File "/tmp/pip-build-env-5__v4cuc/overlay/lib/python3.7/site-packages/setuptools/dist.py", line 424, in __init__
 k: v for k, v in attrs.items()
 File "/usr/local/lib/python3.7/distutils/dist.py", line 292, in __init__
 self.finalize_options()
 File "/tmp/pip-build-env-5__v4cuc/overlay/lib/python3.7/site-packages/setuptools/dist.py", line 695, in finalize_options
 ep(self)
 File "/tmp/pip-build-env-5__v4cuc/overlay/lib/python3.7/site-packages/setuptools/dist.py", line 702, in _finalize_setup_keywords
 ep.load()(self, ep.name, value)
 File "/tmp/pip-build-env-5__v4cuc/overlay/lib/python3.7/site-packages/setuptools_scm/integration.py", line 26, in version_keyword
 dist.metadata.version = _get_version(config)
 File "/tmp/pip-build-env-5__v4cuc/overlay/lib/python3.7/site-packages/setuptools_scm/__init__.py", line 173, in _get_version
 parsed_version = _do_parse(config)
 File "/tmp/pip-build-env-5__v4cuc/overlay/lib/python3.7/site-packages/setuptools_scm/__init__.py", line 119, in _do_parse
 parse_result = _call_entrypoint_fn(config.absolute_root, config, config.parse)
 File "/tmp/pip-build-env-5__v4cuc/overlay/lib/python3.7/site-packages/setuptools_scm/__init__.py", line 54, in _call_entrypoint_fn
 return fn(root)
 File "setup.py", line 539, in parse_git
 return parse(root, **kwargs)
 File "/tmp/pip-build-env-5__v4cuc/overlay/lib/python3.7/site-packages/setuptools_scm/git.py", line 95, in parse
 require_command("git")
 File "/tmp/pip-build-env-5__v4cuc/overlay/lib/python3.7/site-packages/setuptools_scm/utils.py", line 151, in require_command
 raise EnvironmentError("%r was not found" % name)
 OSError: 'git' was not found
 ----------------------------------------
ERROR: Command errored out with exit status 1: /usr/local/bin/python /usr/local/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /tmp/tmp_p9la_sd Check the logs for full command output.
WARNING: You are using pip version 20.2.3; however, version 20.3.3 is available.
You should consider upgrading via the '/usr/local/bin/python -m pip install --upgrade pip' command.

还有其他人有同样的问题吗？

如果有帮助，这里是图像的 Dockerfile：

# Copyright 2020 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#    http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

FROM gcr.io/dataflow-templates-base/python3-template-launcher-base

ARG WORKDIR=/dataflow/template
RUN mkdir -p ${WORKDIR}
WORKDIR ${WORKDIR}

# Due to a change in the Beam base image in version 2.24, we need to install
# libffi-dev manually as a dependency. For more information:
#   https://github.com/GoogleCloudPlatform/python-docs-samples/issues/4891
RUN apt-get update && apt-get install -y libffi-dev && rm -rf /var/lib/apt/lists/*

COPY requirements.txt .
COPY streaming_beam.py .

ENV FLEX_TEMPLATE_PYTHON_REQUIREMENTS_FILE="${WORKDIR}/requirements.txt"
ENV FLEX_TEMPLATE_PYTHON_PY_FILE="${WORKDIR}/streaming_beam.py"

RUN pip install -U -r ./requirements.txt

Answer 1

看起来您需要在 dockerfile 中安装 git。添加此行：

RUN apt-get install git

Answer 2

这看起来像是一个错误，一些上游依赖项开始需要git才能构建，但基础映像目前不包含它。 我在这里提出了一个问题： https://issuetracker.google.com/issues/176570473