Google Cloud Dataflow Python SDK 更新

Question

On using the Google Cloud Dataflow Python SDK happens that at start reading a lot of data from the Cloud Storage it takes a while and causes the error AssertionError: Job did not reach to a terminal state after waiting indefinitely .

Doing a search we found the open issue BEAM-5529 which refers to the patch #6535 released in version 2.8.0 but not present in the release note.

In the other hand the current published version is google-cloud-dataflow 2.5.0

There is any update policy or is it the individual responsibility to perform the compilation and generation of a new version with the latest releases?

Please any help or comment will be welcome.

Answer 1

As per official Google Cloud Platform docs here :

The Cloud Dataflow SDK 2.5.0 is the last Cloud Dataflow SDK release that is separate from the Apache Beam SDK releases. The Cloud Dataflow service fully supports official Apache Beam SDK releases.

So yes, google-cloud-dataflow 2.5.0 is the last release, and from that version on you should use the official apache-beam releases. Bear in mind that you will need to install the library using the extra [gcp]:

pip install apache-beam[gcp]

Finally, the fix in 6535 should be applied already, since I installed the library "pip install apache-beam[gcp]===2.8.0" and I went to the file "apache_beam/runners/dataflow/dataflow_runner.py" and it has the fix applied there.