Apache Beam Python throwing 503 errors on Cloud Dataflow - Why?

Question

I am seeing intermittent 503 errors from Cloud Dataflow when running my Apache Beam Python graph. Why is this happening?

The error was does not provide enough context to decipher source of problem.

File "/usr/local/lib/python2.7/dist-packages/apache_beam/io/gcp/gcsio.py", line 553, in finish
    raise self._upload_thread.last_error  # pylint: disable=raising-bad-type
HttpError: HttpError accessing <https://www.googleapis.com/resumable/upload/storage/v1/b/******/o?uploadType=resumable&alt=json&upload_id=AEnB2UotBsLaGqKqOtjpRZcvZLVmwiikA99j64_WEnOVh42zUt8F1m5hXxA-PwpUVJdtwhbfqAf5C-WJzJtsVfXg6vmGh9ogjA&name=temp%2Fbeamapp-cloude-0908010623-413022.1536368783.413249%2F5154533467808153215%2Fdax-tmp-2018-09-07_18_07_37-7050531770796171347-S01-0-406ba749ef65e1f5%2F-shard--try-15bbe0de7ff515c5-endshard.json>: response: <{'status': '410', 'content-length': '177', 'vary': 'Origin, X-Origin', 'server': 'UploadServer', 'x-guploader-uploadid': 'AEnB2UotBsLaGqKqOtjpRZcvZLVmwiikA99j64_WEnOVh42zUt8F1m5hXxA-PwpUVJdtwhbfqAf5C-WJzJtsVfXg6vmGh9ogjA', 'date': 'Sat, 08 Sep 2018 01:12:32 GMT', 'content-type': 'application/json; charset=UTF-8'}>, content <{
 "error": {
  "errors": [
   {
    "domain": "global",
    "reason": "backendError",
    "message": "Backend Error"
   }
  ],
  "code": 503,
  "message": "Backend Error"
 }
}
>

Answer 1

The messages 'HttpError accessing

Looking at Cloud Storage error codes , the following is the explanation for the error codes that appeared in your message:

410—Gone (). You have attempted to use a resumable upload session that is no longer available. If the reported status code was not successful and you still wish to upload the file, you must start a new session.

503—Service Unavailable (backendError). We encountered an internal error. Please try again using truncated exponential backoff.

According to this, the issue it is supposed to disappear after trying later, but if not, you can try truncated exponential backoff mechanism.

Is your issue already solved?