[英]Python API call to BigQuery using cloud functions
I'm trying to build my first cloud function. Its a function that should get data from API, transform to DF and push to bigquery.我正在尝试构建我的第一个云 function。它是一个 function,应该从 API 获取数据,转换为 DF 并推送到 bigquery。 I've set the cloud function up with a http trigger using validate_http as entry point.
我已将云 function 设置为使用 validate_http 作为入口点的 http 触发器。 The problem is that it states the function is working but it doesnt actually write anything.
问题是它声明 function 正在工作,但它实际上并没有写任何东西。 Its a similiar problem as the problem discussed here: Passing data from http api to bigquery using google cloud function python
这是一个与此处讨论的问题类似的问题: Passing data from http api to bigquery using google cloud function python
import pandas as pd
import json
import requests
from pandas.io import gbq
import pandas_gbq
import gcsfs
#function 1: Responding and validating any HTTP request
def validate_http(request):
request.json = request.get_json()
if request.args:
get_api_data()
return f'Data pull complete'
elif request_json:
get_api_data()
return f'Data pull complete'
else:
get_api_data()
return f'Data pull complete'
#function 2: Get data and transform
def get_api_data():
import pandas as pd
import requests
import json
#Setting up variables with tokens
base_url = "https://"
token= "&token="
token2= "&token="
fields = "&fields=date,id,shippingAddress,items"
date_filter = "&filter=date in '2022-01-22'"
data_limit = "&limit=99999999"
#Performing API call on request with variables
def main_requests(base_url,token,fields,date_filter,data_limit):
req = requests.get(base_url + token + fields +date_filter + data_limit)
return req.json()
#Making API Call and storing in data
data = main_requests(base_url,token,fields,date_filter,data_limit)
#transforming the data
df = pd.json_normalize(data['orders']).explode('items').reset_index(drop=True)
items = df['items'].agg(pd.Series)[['id','itemNumber','colorNumber', 'amount', 'size','quantity', 'quantityReturned']]
df = df.drop(columns=[ 'items', 'shippingAddress.id', 'shippingAddress.housenumber', 'shippingAddress.housenumberExtension', 'shippingAddress.address2','shippingAddress.name','shippingAddress.companyName','shippingAddress.street', 'shippingAddress.postalcode', 'shippingAddress.city', 'shippingAddress.county', 'shippingAddress.countryId', 'shippingAddress.email', 'shippingAddress.phone'])
df = df.rename(columns=
{'date' : 'Date',
'shippingAddress.countryIso' : 'Country',
'id' : 'order_id'})
df = pd.concat([df, items], axis=1, join='inner')
#Push data function
bq_load('Return_data_api', df)
#function 3: Convert to bigquery table
def bq_load(key, value):
project_name = '375215'
dataset_name = 'Returns'
table_name = key
value.to_gbq(destination_table='{}.{}'.format(dataset_name, table_name), project_id=project_name, if_exists='replace')
The problem is that the script doesnt write to bigquery and doesnt return any error.问题是脚本不会写入 bigquery,也不会返回任何错误。 I know that the get_api_data() function is working since I tested it locally and does seem to be able to write to BigQuery.
我知道 get_api_data() function 正在工作,因为我在本地对其进行了测试并且似乎能够写入 BigQuery。 Using cloud functions I cant seem to trigger this function and make it write data to bigquery.
使用云函数我似乎无法触发这个 function 并将数据写入 bigquery。
There are a couple of things wrong with the code that would set you right.代码中有几处错误会使您正确。
this would mean updating (and probably renaming) the JsonArrayStore
class and its methods to work with CSV.这意味着更新(并可能重命名)
JsonArrayStore
class 及其方法以使用 CSV。
Once you have completed the above and written well formed csv, you can proceed to this:完成上述内容并编写格式正确的 csv 后,您可以继续执行此操作:
csv
in the del_btn
method would then look like this:del_btn
方法中读取csv
将如下所示:import python
class ToDoGUI(tk.Tk):
...
# methods
...
def del_btn(self):
a = JsonArrayStore('test1.csv')
# read to list
with open('test1.csv') as csvfile:
reader = csv.reader(csvfile)
data = list(reader)
print(data)
Good work, you have a lot to do, if you get stuck further please post again.干得好,你有很多事情要做,如果你进一步卡住,请再次发帖。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.