从 Cloud Functions 到 BigQuery 的 pandas df - 仅 PARQUET 和 CSV？

Question

I'm querying an API with GCP Cloud Functions and would like to write the result to BigQuery. I'm getting this error:

Got unexpected source_format: 'NEWLINE_DELIMITED_JSON'. Currently, only PARQUET and CSV are supported

This is my code

from google.cloud import bigquery
import pandas as pd
import requests
import datetime

def hello_pubsub(event, context):
 
 response = requests.get("https://api.openweathermap.org/data/2.5/weather?q=berlin&appid=12345&units=metric&lang=de")
 responseJson = response.json()
 
 # Creates DataFrame
 df = pd.DataFrame({'datetime':pd.to_datetime(format(datetime.datetime.now())),
               'name':str(responseJson['name']),
               'temp':float(responseJson['main']['temp']),
               'windspeed':float(responseJson['wind']['speed']),
               'winddeg':int(responseJson['wind']['deg'])
               }, index=[0])  

 project_id = 'myproj'
 client = bigquery.Client(project=project_id)
 dataset_id = 'weather'

 dataset_ref = client.dataset(dataset_id)
 job_config = bigquery.LoadJobConfig()
 job_config.autodetect = True
 job_config.write_disposition = "WRITE_APPEND"
 job_config.source_format = bigquery.SourceFormat.NEWLINE_DELIMITED_JSON
 load_job = client.load_table_from_dataframe(df, dataset_ref.table("weather_de"), job_config=job_config)

What's the best way to do this?

Answer 1

The BigQuery client library reference states that this is intended behavior when loading into a table from a dataframe using load_table_from_dataframe() :

By default, this method uses the parquet source format. To override this, supply a value for source_format with the format name. Currently only CSV and PARQUET are supported.

Something you can try is replacing that method with load_table_from_json() , which is also available , and uses NEWLINE_DELIMITED_JSON as the source format. This method clearly will not accept a dataframe as input, so I could recommend using a JSON object to store the data you need from the API response. Otherwise, you can convert the existing dataframe you created to json using the to_json() method from the pandas doc .

You can read more into how the BigQuery client works from the reference , and you can also see the built source formats .