pandas dataframe.to_sql throws error when writing to MariaDB

Question

When I try and execute the code below, python gives me a mariadb.ProgrammingError: Cursor is closed even though the data was successfully written to the db. I guess there is something wrong with returning the rowcount (see the full traceback below). Is this a possible bug or am I missing something?

code sample

import pandas as pd
from sqlalchemy import create_engine
conn_str = 'mariadb+mariadbconnector://user:password@localhost:3306/database_name'
engine = create_engine(conn_str)
df = pd.DataFrame(data={'col1': [1, 2, 3], 'col2': ['A', 'B', 'C']})
df.to_sql('pandas_test_table', index=False, con=engine, if_exists='replace')

traceback

Traceback (most recent call last):
  File "C:\Program Files\Python310\lib\code.py", line 90, in runcode
    exec(code, self.locals)
  File "<input>", line 1, in <module>
  File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pandas\core\generic.py", line 2951, in to_sql
    return sql.to_sql(
  File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pandas\io\sql.py", line 697, in to_sql
    return pandas_sql.to_sql(
  File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pandas\io\sql.py", line 1739, in to_sql
    total_inserted = sql_engine.insert_records(
  File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pandas\io\sql.py", line 1322, in insert_records
    return table.insert(chunksize=chunksize, method=method)
  File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pandas\io\sql.py", line 950, in insert
    num_inserted = exec_insert(conn, keys, chunk_iter)
  File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pandas\io\sql.py", line 858, in _execute_insert
    return result.rowcount
  File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\sqlalchemy\util\langhelpers.py", line 1113, in __get__
    obj.__dict__[self.__name__] = result = self.fget(obj)
  File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\sqlalchemy\engine\cursor.py", line 1691, in rowcount
    self.cursor_strategy.handle_exception(self, self.cursor, e)
  File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\sqlalchemy\engine\cursor.py", line 846, in handle_exception
    raise err
  File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\sqlalchemy\engine\cursor.py", line 1689, in rowcount
    return self.context.rowcount
  File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\sqlalchemy\engine\default.py", line 1439, in rowcount
    return self.cursor.rowcount
  File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\mariadb\cursors.py", line 541, in rowcount
    self.check_closed()
  File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\mariadb\cursors.py", line 55, in check_closed
    raise mariadb.ProgrammingError("Cursor is closed")
mariadb.ProgrammingError: Cursor is closed

package version

mariadb==1.1.4
pandas==1.4.3
SQLAlchemy==1.4.40

Answer 1

I had attempted Georg Richter's recommendation regarding installing MariaDB 1.1.5 from source or rolling back to 1.1.3 and was still receiving the error message.

Eventually, I came across this documentation and had changed the engine DBAPI from

uri = 'mariadb+mariadbconnector://Username:Password@localhost/test'

to

uri = 'mariadb+pymysql://Username:Password@localhost/test'

And it worked.

Answer 2

It's a bug in MariaDB Connector/Python.

It's already fixed in 1.1.5 (not released yet). If you don't want to downgrade to 1.1.3 you have to build 1.1.5 from source.

Answer 3

Hummm... I'm using mariadb version 1.1.5.post3, but I get the same error. Just the last lines of the data insertion code, which doesn't raise any error in case I use the pymysql connector:

new_employee = {'id': '105', 'first_name': 'Joao',
                'last_name': 'da Silva', 'description': 'Lorem Ipsum', 'active': '1'}
emp = {k: [v] for k, v in new_employee.items()}
df = pd.DataFrame(
    emp, columns=['id', 'first_name', 'last_name', 'description', 'active'])
df.to_sql('employees', con=cnx, index=False, if_exists='append')

The error:

    raise mariadb.ProgrammingError("Cursor is closed")
mariadb.ProgrammingError: Cursor is closed

Just in case, I've tried to downgrade to the version 1.1.5 mentioned above (using pip install , not from the source). Other packages versions:

SQLAlchemy        1.4.46
pandas            1.5.2

PS Although we have this error, the data is inserted into the Maria DB table.