PATCH 中的 google-cloud-storage 随机 503 错误

Question

I have a code that reads files from a bucket then edit the content_encoding from all files metadata. It usually works fine, but since 3 days ago i'm heaving problems with the google-cloud-storage API, it seems to be returning 503 errors randomly. I already tried to use a retry strategy with an 600 seconds deadline but the code still returning errors.

Im running this code with python3 in a databricks notebook with an 9.1 LTS (includes Apache Spark 3.1.2, Scala 2.12) cluster.

google-cloud-srtorage API version -> 2.5.0

I read some old threads about this problem, and it seems to be a known issue but im still not able to resolve it. Here is the code:

def blob_list(bucket_name):
  try:
    client = storage.Client()
    blobs = client.list_blobs(bucket_name)
    print('Bucket read')
    return blobs
  except Exception as e:
    print('Counld not read the bucket', e)
    
b = blob_list(bucket_name)
count = 0
modified_retry = DEFAULT_RETRY.with_deadline(600)
modified_retry = modified_retry.with_delay(initial=1.5, multiplier=1.2, maximum=45.0)

for item in b:
  CS = storage.Client()
  blob = CS.bucket(bucket_name).blob(item.name)
  blob.patch(retry=modified_retry)
#  print(blob.content_encoding)
  if blob.content_encoding == 'gzip' or blob.content_encoding == 'txt': 
    blob.content_encoding = 'csv'
    blob.patch(retry=modified_retry)
    count +=1
print('Changed',count,'metadata files')

The code is taking too long to run and still throwing This error

Deadline of 600.0s exceeded while calling target function, last exception: 503 PATCH https://storage.googleapis.com/storage/v1/b/bucket_name/o/yof0soyd7668_2022-08-15T060000_06db87dcb5e5ee06ec13ab5fbefe4df0_be822a.csv.gz?projection=full&prettyPrint=false : We encountered an internal error. Please try again.

The error seems to occurs in the patch() method.

Answer 1

This is unfortunately an issue on Google's end. I am getting the same 503 error as you on processes that I have been running daily for over 2 years. I created a support case through GCP and am waiting to hear back.