Python API call to BigQuery using cloud functions

Question

I'm trying to build my first cloud function. Its a function that should get data from API, transform to DF and push to bigquery. I've set the cloud function up with a http trigger using validate_http as entry point. The problem is that it states the function is working but it doesnt actually write anything. Its a similiar problem as the problem discussed here: Passing data from http api to bigquery using google cloud function python

import pandas as pd
import json
import requests
from pandas.io import gbq
import pandas_gbq
import gcsfs



#function 1: Responding and validating any HTTP request

def validate_http(request):
  request.json = request.get_json()
  
  if request.args:
    get_api_data()
    return f'Data pull complete'
  
  elif request_json:
    get_api_data()
    return f'Data pull complete'
  
  else:
    get_api_data()
    return f'Data pull complete'

#function 2: Get data and transform

def get_api_data():

    import pandas as pd
    import requests
    import json

    #Setting up variables with tokens
    base_url = "https://"
    token= "&token="
    token2= "&token="
    fields = "&fields=date,id,shippingAddress,items"
    date_filter = "&filter=date in '2022-01-22'"
    data_limit = "&limit=99999999"

    #Performing API call on request with variables
    def main_requests(base_url,token,fields,date_filter,data_limit):
        req = requests.get(base_url + token + fields +date_filter + data_limit)
        return req.json()

        #Making API Call and storing in data
    data = main_requests(base_url,token,fields,date_filter,data_limit)

    #transforming the data
    df = pd.json_normalize(data['orders']).explode('items').reset_index(drop=True)
    items = df['items'].agg(pd.Series)[['id','itemNumber','colorNumber', 'amount', 'size','quantity', 'quantityReturned']]
    df = df.drop(columns=[ 'items', 'shippingAddress.id', 'shippingAddress.housenumber', 'shippingAddress.housenumberExtension', 'shippingAddress.address2','shippingAddress.name','shippingAddress.companyName','shippingAddress.street', 'shippingAddress.postalcode', 'shippingAddress.city', 'shippingAddress.county', 'shippingAddress.countryId', 'shippingAddress.email', 'shippingAddress.phone'])
    df = df.rename(columns=
         {'date' : 'Date',
          'shippingAddress.countryIso' : 'Country',
          'id' : 'order_id'})

    df = pd.concat([df, items], axis=1, join='inner')      
  
  #Push data function
    bq_load('Return_data_api', df)
 

#function 3: Convert to bigquery table
  
def bq_load(key, value):
  
  project_name = '375215'
  dataset_name = 'Returns'
  table_name = key
  
  value.to_gbq(destination_table='{}.{}'.format(dataset_name, table_name), project_id=project_name, if_exists='replace')

The problem is that the script doesnt write to bigquery and doesnt return any error. I know that the get_api_data() function is working since I tested it locally and does seem to be able to write to BigQuery. Using cloud functions I cant seem to trigger this function and make it write data to bigquery.

Answer 1

There are a couple of things wrong with the code that would set you right.

• you have list data, so store as a csv file (in preference to json).

this would mean updating (and probably renaming) the JsonArrayStore class and its methods to work with CSV.

Once you have completed the above and written well formed csv, you can proceed to this:

• reading the csv in the del_btn method would then look like this:

import python

class ToDoGUI(tk.Tk):
    ...
    # methods
    ...

    def del_btn(self):
        a = JsonArrayStore('test1.csv')
        # read to list
        with open('test1.csv') as csvfile:
            reader = csv.reader(csvfile)
            data = list(reader)
        
        print(data)

Good work, you have a lot to do, if you get stuck further please post again.