[英]GCP Cloud Functions to BigQuery - parquet support error
I am trying to run a simple test cloud function
where I create a BigQuery
table and insert a value.我正在尝试运行一个简单的测试
cloud function
,在其中创建一个BigQuery
表并插入一个值。 The error I receive sounds like I need to import pyarrow
, so I've tried doing that but I keep receiving the same error.我收到的错误听起来像是我需要导入
pyarrow
,所以我尝试这样做,但我一直收到相同的错误。 When I run an equivalent script locally, there are no issues, the table is created, and I don't even need to import pyarrow
.当我在本地运行等效脚本时,没有问题,表已创建,我什至不需要导入
pyarrow
。 What am I missing here?我在这里缺少什么?
The error
: error
:
ImportError: Unable to find a usable engine; tried using: 'pyarrow', 'fastparquet'. pyarrow or fastparquet is required for parquet support
The main.py
:主要
main.py
:
import pandas as pd
from google.cloud import bigquery
import pyarrow
def main_func(data, context):
df = pd.DataFrame({'Test': ['Success']})
client = bigquery.Client()
dataset_id = #removed here but specified in the real code
dataset = bigquery.Dataset(dataset_id)
dataset.location = #removed here but specified in the real code
dataset = client.create_dataset(dataset, exists_ok=True)
print("Created dataset {}.{}".format(client.project, dataset.dataset_id))
table_id = #removed here but specified in the real code
job_config = bigquery.LoadJobConfig(
schema=[
bigquery.SchemaField("Test", bigquery.enums.SqlTypeNames.STRING),
],
write_disposition="WRITE_TRUNCATE",
)
job = client.load_table_from_dataframe(
df, table_id, job_config = job_config
)
job.result()
The requirements.txt
: requirements.txt
:
pandas
google-cloud-bigquery
pyarrow
You have an issue with pyarrow version.您的 pyarrow 版本有问题。 Pandas does not detect any
pyarrow<0.4
because of compatibility issues, therefore you should try adding pyarrow>=0.4
in your requirements.txt
.由于兼容性问题,Pandas 没有检测到任何
pyarrow<0.4
,因此您应该尝试在requirements.txt
添加pyarrow>=0.4
。
Pyarrow is not properly detected after importing ray导入光线后未正确检测到 Pyarrow
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.