python pandas dataframe to_sql converting an object to Mysql INT datatype yields incorrect results

Question

I am trying to read a csv file into a Pandas dataframe and insert the final dataframe into Mysql using pandas.to_sql function.

All the columns are inserting the correct data except for one column in dataframe which has a length of 25 characters. This column (transaction_id) is defined as a INT(25) in MYSQL and I have not been able to figure out why this column has wrong data.

And the weird thing is, the transaction_id column in MySQL has the same value for more than 360K rows per csv file.

Any help would be great.

Client transaction ID example:

format: transaction id_page id-banner id
2343213254646775357496618_12-586542237
2343213254646775357881218_14-586542237 
2343213254646775357886268_10-586542237
2343213254646775357886218_27-586542237
2343213254646775357886248_10-586542237

Here is my code:

xls = pd.ExcelFile(path_value)
df = xls.parse('report', skiprows=13, index_col=None, na_values=['NA'])

# remove last row
df = df[:-1]
df['transaction_datetime'] = pd.to_datetime(df['transaction_datetime'])

# add transaction date column to data frame:
df['transaction_date'] = df['transaction_datetime'].dt.date
df.loc[:, 'created_date'] = datetime.datetime.now()

# convert client transaction id into three parts
df['transaction_id'], df['placeholder'] = zip(
                        *df['Client Transaction ID'].apply(lambda x: x.split('_', 1)))
df['page_id'], df['banner_id'] = zip(*df['placeholder'].apply(lambda x: x.split('-', 1)))


df.drop('placeholder', axis=1, inplace=True)
df.drop('Client Transaction ID', axis=1, inplace=True)

print datetime.datetime.now()
# connect to mysql
engine = create_engine(
                        'connection string'
                        echo=False)
                    df.to_sql(name='table', con=engine, if_exists='append', index=False)
print datetime.datetime.now()

Answer 1

If i understood your code correctly you can parse transaction_id , page_id , banner_id using vectorized .str.extract() method and doing it "in one shot":

In [32]: df
Out[32]:
    Client Transaction ID
0  tx0001_page01-banner01
1  tx0002_page01-banner23
2  tx0003_page33-banner56
3  tx0004_page12-banner76
4  tx0005_page44-banner11

In [33]: df[['transaction_id','page_id','banner_id']] = \
    ...:     df.pop('Client Transaction ID').str.extract(r'^([^_]*)_([^-]*)-(.*)',
    ...:                                                 expand=True)
    ...:

In [34]: df
Out[34]:
  transaction_id page_id banner_id
0         tx0001  page01  banner01
1         tx0002  page01  banner23
2         tx0003  page33  banner56
3         tx0004  page12  banner76
4         tx0005  page44  banner11

PS you didn't provide any sample data so i had to reconstruct it by your code...

Answer 2

Apparently, the issue stems from MySQL. My transaction id, having a length of 25, was big for BIGINT. I have to convert it to VARCHAR(25) to get the right value in the table. Thanks @MaxU for improving my code.