Table is not being created and data is not loaded to BigQuery Dataset Table through load_table_from_dataframe using cloud functions

Question

I am using python in cloud functions with pandas and have written 7 different functions with same approach. 4 out of 7 created and loaded the data in BigQuery Table successfully but the rest of 3 are not going to perform load-table-from-dataframe.

df_ABC = pd.DataFrame(creditnotes)
df_XYZ = df_ABC[
    "id",
    "subscription_id",
    "status",
    "amount_allocated",
    "amount_available",
    "amount_refunded",
    "customer_id",
    "date",
    "voided_at",
    "sub_total",
    "total",
    "taxes",
    "reference_invoice_id",
]
df_XYZ["amount_allocated"] = df_XYZ["amount_allocated"] / 100
df_XYZ["amount_available"] = df_XYZ["amount_available"] / 100
df_XYZ["amount_refunded"] = df_XYZ["amount_refunded"] / 100
df_XYZ["sub_total"] = df_XYZ["sub_total"] / 100
df_XYZ["total"] = df_XYZ["total"] / 100
df_XYZ["date"] = pd.to_datetime(df_XYZ["date"], unit="s")
df_XYZ["voided_at"] = pd.to_datetime(df_XYZ["voided_at"], unit="s")
df_XYZ["taxes"] = round(df_XYZ["total"] - df_XYZ["sub_total"], 2)

client = bigquery.Client()
dataset_ref = client.dataset("aus_intl")
table_ref = dataset_ref.table("creditNotes")
job_config = bigquery.LoadJobConfig()
job_config.write_disposition = "WRITE_TRUNCATE"
client.load_table_from_dataframe(df_XYZ, table_ref, job_config=job_config).result()

Here is the same logic/Approach and it is loading the table from dataframe successfully.

df = pd.DataFrame(customers)
df_customers = df[['id','first_name','last_name','company','email','created_at','cf_city','cf_customer_type','payment_method']]
df_customers['created_at'] = pd.to_datetime(df_customers['created_at'], unit='s')
      
    client = bigquery.Client()
    dataset_ref = client.dataset('au_intl')
    table_ref = dataset_ref.table('au_customers')
    job_config = bigquery.LoadJobConfig()
    job_config.write_disposition = 'WRITE_TRUNCATE'
    client.load_table_from_dataframe(df_customers, table_ref, job_config=job_config).result()

Where am i missing something? i tried to find the issue from logs but there are only 200 status code.

Answer 1

It was because of "not in index". as fields coming from chargebee API were having conflict with the fields i was using to create the columns using dataframes.