client.get_bucket() returns error: api_request() got an unexpected keyword argument 'extra_api_info'

Question

I'm trying to store a newline-delimited JSON string in a GCS bucket using a cloud function, but seeing an error. I start by converting a dataframe to ndjson, then attempt to upload this to my GCS bucket as below. There is more code above this, but not relevant to my problem:

import pandas as pd
from google.cloud import storage
from google.cloud.storage import blob

df = df.to_json(orient="records", lines=True)

storage_client = storage.Client(project='my-project')
bucket = storage_client.get_bucket('my-bucket')
blob = bucket.blob('my-blob')
blob.upload_from_string(df)

When running this, I find the error below in the logs:

Exception on / [POST] Traceback (most recent call last): 
File "/layers/google.python.pip/pip/lib/python3.7/site-packages/flask/app.py", line 2073, in wsgi_app response = self.full_dispatch_request() 
File "/layers/google.python.pip/pip/lib/python3.7/site-packages/flask/app.py", line 1518, in full_dispatch_request rv = self.handle_user_exception(e) 
File "/layers/google.python.pip/pip/lib/python3.7/site-packages/flask/app.py", line 1516, in full_dispatch_request rv = self.dispatch_request() 
File "/layers/google.python.pip/pip/lib/python3.7/site-packages/flask/app.py", line 1502, in dispatch_request return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) 
File "/layers/google.python.pip/pip/lib/python3.7/site-packages/functions_framework/__init__.py", line 99, in view_func return function(request._get_current_object()) 
File "/workspace/main.py", line 66, in my_request bucket = storage_client.get_bucket('my-bucket') 
File "/layers/google.python.pip/pip/lib/python3.7/site-packages/google/cloud/storage/client.py", line 787, in get_bucket retry=retry, 
File "/layers/google.python.pip/pip/lib/python3.7/site-packages/google/cloud/storage/bucket.py", line 1037, in reload retry=retry, 
File "/layers/google.python.pip/pip/lib/python3.7/site-packages/google/cloud/storage/_helpers.py", line 244, in reload _target_object=self, 
File "/layers/google.python.pip/pip/lib/python3.7/site-packages/google/cloud/storage/client.py", line 373, in _get_resource _target_object=_target_object, 
File "/layers/google.python.pip/pip/lib/python3.7/site-packages/google/cloud/storage/_http.py", line 73, in api_request return call() 
File "/layers/google.python.pip/pip/lib/python3.7/site-packages/google/api_core/retry.py", line 288, in retry_wrapped_func on_error=on_error, 
File "/layers/google.python.pip/pip/lib/python3.7/site-packages/google/api_core/retry.py", line 190, in retry_target return target() TypeError: api_request() got an unexpected keyword argument 'extra_api_info'

This 'extra_api_info' argument appears to be the culprit, but I have no idea what this means, and never used to get this error when following exactly the same approach, so I wonder whether this is down to some change between different versions of the 'google.cloud' Python module.

Answer 1

I had the same problem; google.cloud.storage==1.44.0 fixed it.

Answer 2

I worked out the answer to my own question. It was indeed a module version issue as I suspected.

Specifying google.cloud.storage==1.44.0 in my requirements.txt file solved the problem, as my code is seemingly not compatible with the latest version of that module (for reasons that escape me).

Answer 3

Thanks for reporting this and apologies that we missed it until now.

We are tracking this bug on Github here https://github.com/googleapis/python-storage/issues/831

If you're still having this issue, it would be great to know a few things, you can respond either here or on Github.

• Is this issue intermittent, only happening occasionally, or is it always happening?
• Can you share the results of running pip freeze in your source directory so we can see if there's any versioning issues?
• Can you try passing retry=None to the get_bucket method and see if you still get the error?