从 python3.7 在 bigquery 中插入时间戳值时出错

Question

i am calling a rest api from python3.7 and I am getting api response in json format. One of the fields is a timestamp string, but in the json response i am getting that in this format.

create_time {
  seconds: 1604562539
  nanos: 418758000
}

I need to insert this into a bigquery table in UTC format which is a timestamp field

so I convert this to Json string like this

createdtime=json.dumps(create_time, default=datetime_handler)

def datetime_handler(dt):
    if isinstance(dt, datetime):
        return dt.isoformat()
    raise TypeError("Unknown type")

this gives me a json string in this format 2020-11-05T07:48:59.418758+00:00

next I need to try to insert this create_time into big-query table in a timestamp field by passing this value to the bigquery request and i get the insert error

 'bigquery#tableDataInsertAllResponse', 'insertErrors': [{'index': 0, 'errors': [{'reason': 'invalid', 'location': 'createdtime', 'debugInfo': '', 'message': 'Could not parse \'"2020-11-05T07:48:59.418758+00:00"\' as a timestamp. Required format is YYYY-MM-DD HH:MM[:SS[.SSSSSS]]'}]}

so I remove the offset(00:00) and now the date becomes 2020-11-05T07:48:59.418758 and i pass this to bigquery again i get the same error.

'bigquery#tableDataInsertAllResponse', 'insertErrors': [{'index': 0, 'errors': [{'reason': 'invalid', 'location': 'createdtime', 'debugInfo': '', 'message': 'Could not parse \'"2020-11-05T07:48:59.418758"\' as a timestamp. Required format is YYYY-MM-DD HH:MM[:SS[.SSSSSS]]'}]},

so finally i replace the offset with Z and the createdtime becomes 2020-11-05T07:48:59.418758Z and i pass it to bigquery and get the same result

**Error: {'kind': 'bigquery#tableDataInsertAllResponse', 'insertErrors': [{'index': 0, 'errors': [{'reason': 'invalid', 'location': 'createdtime', 'debugInfo': '', 'message': 'Could not parse \'"2020-11-05T07:48:59.418758Z"\' as a timestamp. Required format is YYYY-MM-DD HH:MM[:SS[.SSSSSS]]'}]}**

And finally based on one of the suggestions i received below i have replaced T also so now the date becomes 2020-11-05 07:48:59.418758 and I still get the same error:

Error: {'kind': 'bigquery#tableDataInsertAllResponse', 'insertErrors': [{'index': 0, 'errors': [{'reason': 'invalid', 'location': 'createdtime', 'debugInfo': '', 'message': 'Could not parse \'"2020-11-05 07:48:59.418758"\' as a timestamp. Required format is YYYY-MM-DD HH:MM[:SS[.SSSSSS]]'}]},

I am very new to python3.7 and also not much familiar with timestamp values in python and bigquery, Can anyone please suggest a solution? Thanks

This is the code I have put all the irrelevant parts also for understanding:

    row={}
rows=[]
     def datetime_handler(dt):
            if isinstance(dt, datetime):
                return dt.isoformat()
            raise TypeError("Unknown type")
     createdtime=json.dumps(snapshotresponse.create_time,default=datetime_hander)
    createdtime=createdtime.replace("+00:00", "").replace('T'," ")
    row["createdTime"]= createdtime
    json_msg = {
                           "json": row
                        }
    rows.append(json_msg)
    body = {
            "kind": "bigquery#tableDataInsertAllRequest",
            "skipInvalidRows": "false",
            "rows": rows
                }
 

Answer 1

Required format is stated as: YYYY-MM-DD HH:MM[:SS[.SSSSSS]]

You have a 'T' you need to get rid of:

date_time_str = '2020-11-05T07:48:59.418758'.replace('T', ' ')

---

EDIT:

I was avoiding creating a datetime object just to make it a string and then send it to be parsed into a dt object again since you already had a json value. But, if you do a simple print(date_time_str) I'd imagine you could easily match it up to the required format and see what is wrong.

If you don't see the formatting issue, I'd say go with the suggestion of hatef Alipoor and just stringify an actual datetime.

Answer 2

Required format: YYYY-MM-DD HH:MM[:SS[.SSSSSS]]

from datetime import datetime

sec = 1604562539
nano = 418758000
total_sec = sec + nano / 1000000000
dt = datetime.utcfromtimestamp(total_sec)
dt.strftime('%Y-%m-%d %H:%M:%S.%f')
# '2020-11-05 07:48:59.418758'