捕获 cx_Oracle executemany() 批量插入错误“期望数字”

Question

I have 56 columns and some million rows, which I am inserting in my Oracle Table with the following code:

import traceback
import sys
try:
    conn = cx_Oracle.connect()
    cursor = conn.cursor()
    df_local = df_scores.astype(object).where(pd.notnull(df_scores), None).copy()
    q_t = '''INSERT INTO Table VALUES ( to_date(  :1, 'yyyy-mm-dd hh24:mi:ss'), '''
    s=''
    for i in range(2, 56):
       s = s + ':'+str(i)+','
    s = s[0:-1] + ' )'
    q_t = q_t + s
    rows = [tuple(x) for x in df_local.values]
    cursor.executemany(q_t, rows)
    conn.commit()
except:    
   print(traceback.print_exc())
   print(sys.exc_info()[2])

at some row I am getting the TypeError: expecting number error. I would like to know if there is a way I can get the exact line of data which is generating the error coz otherwise, it would be very difficult to debug every time I get such error? The used stack trace does not give any further details.

Answer 1

If you are looking for which rows contain invalid data, you can set batcherrors=True during an executemany() . This will give you a list of all rows that failed during the insert. You can find an example with a more detailed explanation here: https://cx-oracle.readthedocs.io/en/latest/user_guide/batch_statement.html#handling-data-errors .

Answer 2

I had this issue. The src csv had mixed dtypes and no header.

pd.read_csv(datafile, error_bad_lines=True, sep='|', encoding='utf-8-sig', dtype=str, header=None,iterator=True, chunksize=200000):

--assigned col names via dataframe.columns

df.columns = ['ID','Col2',...]

--then cast astype(str) to all columns

for col in df.columns: df[col] = df[col].astype(str).str.encode('ascii', 'replace').str.decode('ascii')

worked for me.