如何从Google Cloud Storage Python远程读取公共文件？

Question

I need to reed some CSV files that were shared which are in google Cloud Storage. My Script will run from another server outside from Google Cloud.

I am using this code:

from google.cloud import storage

client = storage.Client()
bucket = client.get_bucket('/stats/installs')
blob = storage.Blob('installs_overview.csv', bucket)  
content = blob.download_as_string()

print(content)

Getting this error: Apparently I haven't specified the project but I don't have one

OSError: Project was not passed and could not be determined from the environment.

Answer 1

There are some wrong assumptions in the previous answers in this topic.

If it is a public bucket you do not have to worry about what project it is connected to. It is well documented how you, for example, can use a bucket to host a public website that browsers can access. Obviously the browser does not have to worry about what project it belongs to.

The code samples are a bit lacking on using public buckets & files,-- in all the examples you supply a project and credentials, which will

1) Bill bucket egress on the project you supply instead of the project the bucket is connected to

2) Assumes that you need to authenticate and authorise.

For a public file or bucket, however, all you have to worry about is the bucket name and file location.

You can

from google.cloud import storage
source="path/to/file/in/bucket.txt"
target="/your/local/file.txt"
client = storage.Client.create_anonymous_client()
# you need to set user_project to None for anonymous access
# If not it will attempt to put egress bill on the project you specify,
# and then you need to be authenticated to that project.
bucket = client.bucket(bucket_name="your-bucket", user_project=None)
blob = storage.Blob(source, bucket)
blob.download_to_filename(filename=target, client=client)

It is important that your file in the bucket has read access to "AllUsers"

Answer 2

First of all, I think there might be some confusion regarding Cloud Storage and how to access it. Cloud Storage is a Google Cloud Platform product, and therefore, to use it, a GCP Project must exist. You can find the project number and project ID for your project in the Home page of the Console, as explained in this documentation page .

That being said, let me refer you to the documentation page about the Python Cloud Storage Client Library . When you create the client to use the service, you can optionally specify the project ID and/or the credentials files to use:

client = storage.Client(project="PROJECT_ID",credentials="OAUTH2_CREDS")

If you do not specify the Project ID, it will be inferred from the environment.

Also, take into account that you must set up authentication in order to use the service. If you were running the application inside another GCP service (Compute Engine, App Engine, etc.), the recommended approach would be using the Application Default Credentials . However, given that that is not your case, you should instead follow this guide to set up authentication , downloading the key for the Service Account having permission to work with Cloud Storage and pointing to it in the environment variable GOOGLE_APPLICATION_CREDENTIALS .

Also, it looks like the configuration in your code is not correct, given that the bucket name you are using ( '/stats/installs' ) is not valid:

Bucket names must be between 3 and 63 characters. A bucket name can contain lowercase alphanumeric characters, hyphens, and underscores . It can contain dots (.) if it forms a valid domain name with a top-level domain (such as .com). Bucket names must start and end with an alphanumeric character .

Note that you can see that the given bucket does not exist by working with exceptions , specifically google.cloud.exceptions.NotFound . Also, given that the files you are trying to access are public, I would not recommend to share the bucket and file names, you can just obfuscate them with a code such as <BUCKET_NAME>, <FILE_NAME> .

So, as a summary, the course of action should be:

1. Identify the project to which the bucket you want to work with belongs.
2. Obtain the right credentials to work with GCS in that project.
3. Add the project and credentials to the code.
4. Fix the code you shared with the correct bucket and file name. Note that if the file is inside a folder (even though in GCS the concept of directories itself does not exist, as I explained in this other question ), the file name in storage.Blob() should include the complete path like path/to/file/file.csv .

Answer 3

I am not a google-cloud expert, but as some of the commentators have said, I think the problem will be that you haven't explicitly told the storage client which Project you are talking about. The error message implies that the storage client tries to figure out for itself which project you are referring to, and if it can't figure it out, it gives that error message. When I use the storage Client I normally just provide the project name as an argument and it seems to do the trick, eg:

client = storage.Client(project='my-uber-project')

Also, I just saw your comment that your bucket "doesn't have a project" - I don't understand how this is possible. If you log in to the google cloud console area and go to storage, surely your bucket is listed there and you can see your project name at the top of the page?

As @Mangu said, the bucket name in your code is presumably just to hide the real bucket name, as forward-slashes are not allowed in bucket names (but are allowed in blob names and can be used to represent 'folders').