从 GOOGLE CLOUD STORAGE BUCKET 下载多个文件

Question

Does anyone know how to pull all the data for a file that start with a 'Test' in a GCS bucket? I have tried to use the glob.glob method and wildcard glob.glob('*.csv') and that has not worked. Any help would be much appreciated.

from google.cloud import storage

def download_blob(bucket_name, source_blob_name, destination_file_name):
    
  bucket_name = 'gcs-bucket'
  source_blob_name = 'folder1/folder2/Test_data_01052022104530.csv'
  destination_file_name = r'C:\Users\path\path1\path2\path3\path4\Test.csv'
  storage_client = storage.Client()
  bucket = storage_client.bucket(bucket_name)
  blob = bucket.blob(source_blob_name)
  blob.download_to_filename(destination_file_name)

Answer 1

GCS doesn't have the concept of directories, so "folder1/folder2/Test_data_01052022104530.csv" is not a file called "Test_data_01052022104530.csv", but an object called "folder1/folder2/Test_data_01052022104530.csv". So you don't want to look for files starting "Test", but objects with the string "/Test" anywhere in their name (or that start with "Test", for the root").

GCS provides no facility to do this, you will have to list objects and filter to do it. For example:

  client = storage.Client()
  for blob in client.list_blobs(BUCKET):
    if "/Test" in blob.name or blob.name.startswith("Test"):
      blob.download_to_filename(somewhere)

For large buckets, this might be slow, especially if there are few files that match. Some ways to speed it up:

• If your data is structured so that you have a small number of directories, and a small fraction of the files will match: it may be faster to "walk the tree" using the prefix and delimiter options of list_blobs and then for each "directory" do a list_blobs with prefix directory+"/Test" to list matching objects only.

• Do listing or downloads in parallel