使用 python 从 GCP 存储桶中递归读取所有子文件夹中的 csv 个文件

Question

I was trying to load all csv files recursively from all sub folders available in a GCP bucket using python pandas.

Currently I am using dask to load data, but its very slow.

import dask
path = "gs://mybucket/parent_path + "*/*.csv"
getAllDaysData = dask.dataframe.read_csv(path).compute()

Can someone help me with better way.

Answer 1

I would suggest reading into parquet files instead. And use pd.read_parquet(file, engine = 'pyarrow') to convert it into a pandas dataframe.

Answer 2

Alternatively you might want to consider loading data into BigQuery first. You can do something like this, as long as all csv-files have the some structure.

uri = f"gs://mybucket/parent_path/*.csv"
job_config = bigquery.LoadJobConfig(
    source_format=bigquery.SourceFormat.CSV
)

load_job = client.load_table_from_uri(
    uri,
    'destination_table',
    job_config=job_config,
    location=GCP_LOCATION
)
load_job_result = load_job.result()