How to list buckets from Google Storage in Python?

Question

I have the following code in Python script:

from google.oauth2 import service_account
SCOPES = ['https://www.googleapis.com/auth/sqlservice.admin']
SERVICE_ACCOUNT_FILE = 'JSON_AUTH_FILE'
credentials = service_account.Credentials.from_service_account_file(
            SERVICE_ACCOUNT_FILE, scopes=SCOPES)
import os
os.environ['GOOGLE_APPLICATION_CREDENTIALS'] = SERVICE_ACCOUNT_FILE

This does the auth with google.

Now I want to list all buckets :

from google.cloud import storage
storage_client = storage.Client()
buckets = list(storage_client.list_buckets())
print(buckets)

But this doesn't work. I get:

google.api_core.exceptions.Forbidden: 403

xxx@yyy.iam.gserviceaccount.com does not have storage.buckets.list access to project

It also has a link when I click it i see (which is weird because it says 403 but here it shows 401:

{
 "error": {
  "errors": [
   {
    "domain": "global",
    "reason": "required",
    "message": "Anonymous caller does not have storage.buckets.list access to project NUMBER.",
    "locationType": "header",
    "location": "Authorization"
   }
  ],
  "code": 401,
  "message": "Anonymous caller does not have storage.buckets.list access to project NUMBER."
 }
}

What am I doing wrong?

Answer 1

A couple of things to suggest in reference to this link: https://cloud.google.com/storage/docs/reference/libraries#client-libraries-install-python

There are a couple of ways you can set roles for service accounts to access Google Storage: https://cloud.google.com/iam/docs/understanding-roles#cloud_storage_roles

Using a default role when creating a service account

When you create your service account, select the Project Role: Storage -> Storage Admin . Setting this role will allow your service account to access and manipulate objects from Cloud Storage. Using the Storage Admin role ensures that you give the least amount of privilege to the service account so that it can't access other services.

If you're having problems with authentication perhaps look at setting the role to Project -> Editor which should give the service account edit access to most of the GCP services. Just be aware that if the service account is compromised the user will have access to most of your services in the GCP project.

Using a custom role to inherit multiple roles

By setting a custom role you can inherit the permissions given to a number of the default roles. A good way to do this is by using the "Create Role From Selection" in the IAM & Admin -> Roles section of the GCP Console.

For example you could combine the BigQuery Admin and Storage Object Admin into a single custom role by selecting the check-boxes for each role and creating your own custom role which you can then allocate to your service account in the IAM section of GCP .

---

Once you have a service account with the correct permissions you should be able set the GOOGLE_APPLICATION_CREDENTIALS environment variable and use the google library to access your storage buckets.

Try this modification to your code once you have a service account with the correct role to test to see if it can list all the buckets the account has access to

import os
from google.cloud import storage

os.environ['GOOGLE_APPLICATION_CREDENTIALS'] = "/home/user/Downloads/[FILE_NAME].json"

# Instantiates a client
storage_client = storage.Client()

# List all the buckets available
for bucket in storage_client.list_buckets():
    print(bucket)

Answer 2

You need to use Storage Admin not Storage Object Admin to list the buckets. The Storage Object Admin role didn't have the storage.buckets.list permission.all credit goes to ScottMcC for the excellent answer.

Answer 3

You can probably do something like this:

    import os
    from google.oauth2 import service_account
    from google.cloud import storage

    credentials = service_account.Credentials.from_service_account_file('/test-path-to-your-file/file.json')
    client_storage = storage.Client(credentials=credentials)

    # List all the buckets available (@ScottMcC code)
    for bucket in storage_client.list_buckets():
       print(bucket)
    
    #or you can list it
    list(client_storage.list_buckets())