[英]Write a Pandas DataFrame to Google Cloud Storage or BigQuery
[英]Efficiently write a Pandas dataframe to Google BigQuery
我正在尝试使用此处记录的pandas.DataFrame.to_gbq()
函数将pandas.DataFrame
上传到 Google Big Query。 问题是to_gbq()
需要 2.3 分钟,而直接上传到 Google Cloud Storage 需要不到一分钟。 我打算上传一堆数据帧(~32),每个数据帧的大小都差不多,所以我想知道什么是更快的选择。
这是我正在使用的脚本:
dataframe.to_gbq('my_dataset.my_table',
'my_project_id',
chunksize=None, # I have tried with several chunk sizes, it runs faster when it's one big chunk (at least for me)
if_exists='append',
verbose=False
)
dataframe.to_csv(str(month) + '_file.csv') # the file size its 37.3 MB, this takes almost 2 seconds
# manually upload the file into GCS GUI
print(dataframe.shape)
(363364, 21)
我的问题是,什么更快?
Dataframe
pandas.DataFrame.to_gbq()
函数上传数据框Dataframe
保存为 CSV,然后使用Python API将其作为文件上传到 BigQueryDataframe
保存为 CSV,然后使用此过程将文件上传到 Google Cloud Storage,然后从 BigQuery 中读取它更新:
备选方案 1 似乎比备选方案 2 更快,(使用pd.DataFrame.to_csv()
和load_data_from_file()
17.9 secs more in average with 3 loops
):
def load_data_from_file(dataset_id, table_id, source_file_name):
bigquery_client = bigquery.Client()
dataset_ref = bigquery_client.dataset(dataset_id)
table_ref = dataset_ref.table(table_id)
with open(source_file_name, 'rb') as source_file:
# This example uses CSV, but you can use other formats.
# See https://cloud.google.com/bigquery/loading-data
job_config = bigquery.LoadJobConfig()
job_config.source_format = 'text/csv'
job_config.autodetect=True
job = bigquery_client.load_table_from_file(
source_file, table_ref, job_config=job_config)
job.result() # Waits for job to complete
print('Loaded {} rows into {}:{}.'.format(
job.output_rows, dataset_id, table_id))
我使用以下代码在Datalab
中对备选方案 1 和 3 进行了比较:
from datalab.context import Context
import datalab.storage as storage
import datalab.bigquery as bq
import pandas as pd
from pandas import DataFrame
import time
# Dataframe to write
my_data = [{1,2,3}]
for i in range(0,100000):
my_data.append({1,2,3})
not_so_simple_dataframe = pd.DataFrame(data=my_data,columns=['a','b','c'])
#Alternative 1
start = time.time()
not_so_simple_dataframe.to_gbq('TestDataSet.TestTable',
Context.default().project_id,
chunksize=10000,
if_exists='append',
verbose=False
)
end = time.time()
print("time alternative 1 " + str(end - start))
#Alternative 3
start = time.time()
sample_bucket_name = Context.default().project_id + '-datalab-example'
sample_bucket_path = 'gs://' + sample_bucket_name
sample_bucket_object = sample_bucket_path + '/Hello.txt'
bigquery_dataset_name = 'TestDataSet'
bigquery_table_name = 'TestTable'
# Define storage bucket
sample_bucket = storage.Bucket(sample_bucket_name)
# Create or overwrite the existing table if it exists
table_schema = bq.Schema.from_dataframe(not_so_simple_dataframe)
# Write the DataFrame to GCS (Google Cloud Storage)
%storage write --variable not_so_simple_dataframe --object $sample_bucket_object
# Write the DataFrame to a BigQuery table
table.insert_data(not_so_simple_dataframe)
end = time.time()
print("time alternative 3 " + str(end - start))
以下是 n = {10000,100000,1000000} 的结果:
n alternative_1 alternative_3
10000 30.72s 8.14s
100000 162.43s 70.64s
1000000 1473.57s 688.59s
从结果来看,方案 3 比方案 1 快。
to_gbq() 也遇到了性能问题,我刚刚尝试了原生谷歌客户端,它的速度更快(大约 4 倍),如果你省略等待结果的步骤,它大约快 20 倍。
值得注意的是,最佳实践是等待结果并检查它,但在我的情况下,稍后还有额外的步骤来验证结果。
我正在使用 pandas_gbq 0.15 版(撰写本文时的最新版本)。 尝试这个:
from google.cloud import bigquery
import pandas
df = pandas.DataFrame(
{
'my_string': ['a', 'b', 'c'],
'my_int64': [1, 2, 3],
'my_float64': [4.0, 5.0, 6.0],
'my_timestamp': [
pandas.Timestamp("1998-09-04T16:03:14"),
pandas.Timestamp("2010-09-13T12:03:45"),
pandas.Timestamp("2015-10-02T16:00:00")
],
}
)
client = bigquery.Client()
table_id = 'my_dataset.new_table'
# Since string columns use the "object" dtype, pass in a (partial) schema
# to ensure the correct BigQuery data type.
job_config = bigquery.LoadJobConfig(schema=[
bigquery.SchemaField("my_string", "STRING"),
])
job = client.load_table_from_dataframe(
df, table_id, job_config=job_config
)
# Wait for the load job to complete. (I omit this step)
# job.result()
您可以使用pandas.DataFrame.to_gbq()
这是文档
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.