将 panda df 移至 teradata 表：[HY000] [Teradata][ODBC Teradata Driver][Teradata Database] 时间戳无效

Question

I have a df that i want to move to a teradata table. I am using a framework that was discussed on this platform. However I am getting a few errors. The entire logic behind loading the df to teradata is:

1) If table doesnt exist then create table else skip creation.

2) Start loading the df to the table. (Note i will be passing multiple xlsx files to a df and eventually appending it to the teradata table)

I have written a bteq script to create a table which goes like this:

    FROM DBC.TABLES WHERE DATABASENAME = 'abc' AND TABLENAME = 'sample';

.IF ACTIVITYCOUNT <> 0 THEN .GOTO SKIP_CREATION
.IF ACTIVITYCOUNT = 0 THEN .GOTO TABLE_NOT_EXISTS

.LABEL TABLE_NOT_EXISTS 
CREATE TABLE abc.sample ( 
col1 VARCHAR(400) CHARACTER  SET LATIN NOT CASESPECIFIC, 
col2 VARCHAR(400) CHARACTER SET LATIN NOT CASESPECIFIC,
.
.
col23  TIMESTAMP(0) WITH TIME ZONE FORMAT 'YYYY-MM-DD HH:MI:SSZ', 
col24 TIMESTAMP(0) WITH TIME ZONE FORMAT 'YYYY-MM-DD HH:MI:SSZ'
);

.LABEL SKIP_CREATION
.LOGOFF

My python code to move the df to teradata is:

df=some data frame
host,username,password = 'host','username', "password"
num_of_chunks = 1000
insert_query= "INSERT INTO abc.sample VALUES(?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)"
udaExec = teradata.UdaExec (appName="IMC", version="1.0", logConsole=False)
with udaExec.connect(method="odbc",system=host, username = username,
                         password=password, driver="Teradata") as session:
    file_exist=session.execute(file=r"Path of the bteq file" ,fileType="bteq",ignoreErrors=[3803])
    schedule_chunks = np.array_split(df, num_of_chunks)

    for i,_ in enumerate(schedule_chunks):

        data = [tuple(x) for x in schedule_chunks[i].to_records(index=False)]

            session.executemany(insert_query, data,batch=True) 

When I run this i get the following error message:

DatabaseError: [HY000] [Teradata][ODBC Teradata Driver][Teradata Database] Invalid timestamp.

Can someone help me with whee am i going wrong? Also need some suggestion if I am writing the bteq script correctly. I want to avoid dropping tables and creating a new one each time.

Answer 1

I was able to push my dataframe into Teradata successfully. All i did was convert my Timestamp columns in my dataframe from datetime64 to object. Below is the only line of code i added before running the above code

df=df.astype(object).where(pd.notnull(df),'')