将时间戳从Dataframe加载到BigQuery数据集

Question

I have a timestamp field loaded_at in my BigQuery table result_data and it's epoch equivalent loaded_at_epoch . I'm using Python to regularly get new data from an external source, add these two fields to the dataframe and load this dataframe in my BigQuery table.

    loaded_at = datetime.utcnow()
    loaded_at_epoch = int((loaded_at - datetime(1970, 1, 1)).total_seconds()) 
    df['loaded_at'] = pd.Series(loaded_at, index=df.index)
    df['loaded_at_epoch'] = pd.Series(loaded_at_epoch, index=df.index)

    job_config = bigquery.LoadJobConfig()
    job_config.write_disposition = bigquery.WriteDisposition.WRITE_APPEND
    job_config.schema_update_options = [bigquery.SchemaUpdateOption.ALLOW_FIELD_ADDITION]

    bq_client.load_table_from_dataframe(df, result_data, location="US", job_config=job_config,)

It used to work but since a couple of weeks, the loaded_at has wrong values such as 1970-01-19 03:32:09.693 UTC, while the loaded_at_epoch has correct timestamp values. It looks like somehow the timestamps are in seconds but are interpreted as being in milliseconds when loaded from the dataframe.

I'm not sure how to make this work. I've been trying to have loaded_at as a string but then I get an error: google.api_core.exceptions.BadRequest: 400 Provided Schema does not match Table <project_id>:<dataset_id>.result_data. Field loaded_at has changed type from TIMESTAMP to STRING google.api_core.exceptions.BadRequest: 400 Provided Schema does not match Table <project_id>:<dataset_id>.result_data. Field loaded_at has changed type from TIMESTAMP to STRING

I also tried adding job_config.autodetect = False in the job configuration but doesn't solve the issue as well.

Any idea on how I can get the date to always work?

Thanks!

Answer 1

Can you try hardcoding the loaded_at for a sample run into a dummy table and see what happens? Your code looks fine, so I am sure it is something to do with pandas based loading.

Alternatively, if you want to avoid pandas loading data into BigQuery, you can use bq cli to do the job for you:

import subprocess
#--you compose your df in this block
# df = ...

loaded_at = datetime.utcnow()
loaded_at_epoch = int((loaded_at - datetime(1970, 1, 1)).total_seconds()) 
df['loaded_at'] = pd.Series(loaded_at, index=df.index)
df['loaded_at_epoch'] = pd.Series(loaded_at_epoch, index=df.index)

#--write the file locally
df.to_csv('temp-data.csv', sep=',', index=False, header=False)

#--load via bq cli
cmd = '''bq --location=US load yourdataset.yourtable temp-data.csv col:type,col:type...'''
subprocess.call(cmd, shell=True)

Answer 2

Thanks Khan, this actually helped me figuring out how to fix it. I first tried with a hardcoded timestamp in a string format but got the same issue. Then I tried with a hardcoded panda Timestamp and it worked. The following code now works.

df['loaded_at'] = pd.Series(pd.Timestamp(loaded_at_epoch, unit='s', tz='UTC'), index=df.index)