如何将所有 CSV 文件从谷歌云存储位置读取到单个 pandas dataframe？

Question

I have gcs bucket and can list out all the files in the bucket using google colab like this:-

!gsutil ls gs://custom_jobs/python_test/

This lists out all the files which are:-

test_1.csv
test_2.csv

I can read a single file at a time like this:-

d = pd.read_csv('gs://custom_jobs/python_test/test_1.csv')

What I intend to do is read test_1.csv and test_2.csv in a single dataframe like we can do locally:-

import glob
files = glob.glob("/home/shashi/python_test/*.csv")

all_dat = pd.DataFrame()

for file in files:
  dat = pd.read_csv(file)
  all_dat = all_dat.append(dat, ignore_index=True)

How is this possible in google colab when my files are on google storage bucket ?

Answer 1

Try using the ls command in gsutil

Ex:

import subprocess

result = subprocess.run(['gsutil', 'ls', 'gs://custom_jobs/python_test/*.csv'], stdout=subprocess.PIPE)

all_dat = pd.DataFrame()
for file in result.stdout.splitlines():
    dat = pd.read_csv(file.strip())
    all_dat = all_dat.append(dat, ignore_index=True)

Answer 2

One simple solution might be:

from google.cloud import storage

bucket_name = "your-bucket-name"
all_dat = pd.DataFrame()

storage_client = storage.Client()

# Note: Client.list_blobs requires at least package version 1.17.0.
blobs = storage_client.list_blobs(bucket_name)

for blob in blobs:
    dat = pd.read_csv("gs://{}/{}".format(bucket_name, blob.name)) 
    all_dat = all_dat.append(dat, ignore_index=True)

Answer 3

One simple solution that I found was:-

files = !gsutil ls -r gs://custom_jobs/python_test/