Accessing Google Cloud Storage from local Jupyter and Notebooks in Google AI Platform / Vertex AI

Question

Problem statement: have Google Cloud Storage with some Buckets. Need to import data from such buckets into:

• a local Jupyter instance running on my local computer
• a Google Colab notebook
• a JupyterLab notebook in Vertex AI (and/or AI Platform)

Any reference code to be able these cases would be appreciated. Kind Regards

Answer 1

local Jupyter instance : First authenticate your local env using gcloud auth login then use gsutil to copy the content to local env.

# Authenticate with your account
!gcloud auth login --no-browser

# Copy from your bucket to local path (note -r is for recursive call)
!gsutil cp -r gs://BUCKET/DIR_PATH ./TARGET_DIR

Colab : First authenticate your Colab session to get access to the cloud APIs.Then you can use gsutil to copy the content to the local env.

# Authenticate with your account
from google.colab import auth as google_auth
google_auth.authenticate_user()

# Copy from your bucket to local path (note -r is for recursive call)
!gsutil cp -r gs://BUCKET/DIR_PATH ./TARGET_DIR

JupyterLab notebook in Vertex AI : Your env is already authenticated. Use gsutil to copy the content to local env.

# Copy from your bucket to local path (note -r is for recursive call)
!gsutil cp -r gs://BUCKET/DIR_PATH ./TARGET_DIR

You can also directly access the files in your Google Cloud Storage via Python using the Cloud Storage client libraries . You will need to authenticate your environment first as mentioned above.

# Imports the Google Cloud client library
from google.cloud import storage

# Instantiates a client
storage_client = storage.Client()

# The name for the new bucket
bucket_name = "my-new-bucket"

# Creates the new bucket
bucket = storage_client.create_bucket(bucket_name)

print(f"Bucket {bucket.name} created.")