pandas to_sql() gives a SADeprecationWarning

Question

The to_sql() function in pandas is now producing a SADeprecationWarning.

df.to_sql(name=tablename, con=c, if_exists='append', index=False )

[..]/lib/python3.8/site-packages/pandas/io/sql.py:1430: SADeprecationWarning:The Connection.run_callable() method is deprecated and will be removed in a future release.  Use a context manager instead. (deprecated since: 1.4)

I was getting this even with df.read_sql() command, when running sql select statements. Changing it to the original df.read_sql_query() that it wraps around, got rid of it. I'm suspecting there would be some linkage there.

So, question is, how to do I write a dataframe table to SQL without it getting deprecated in a future release? What does "use a context manager" mean, how can I implement that?

Versions:
pandas: 1.1.5 | SQLAlchemy: 1.4.0 | pyodbc: 4.0.30 | Python: 3.8.0
Working with a mssql database.
OS: Linux Mint Xfce, 18.04. Using a python virtual environment.

If it matters, connection created like so:

conn_str = r'mssql+pyodbc:///?odbc_connect={}'.format(dbString).strip()
sqlEngine = sqlalchemy.create_engine(conn_str,echo=False, pool_recycle=3600)
c = sqlEngine.connect()

And after the db operation,

c.close()

Doing so keeps the main connection sqlEngine "alive" between api calls and lets me use a pooled connection rather than having to connect anew.

Answer 1

Update: according to the pandas team, this will be fixed in Pandas 1.2.4 which as of the time of writing has not been released yet.

Adding this as an answer since Google led here but the accepted answer is not applicable.

Our surrounding code that uses Pandas does use a context manager:

with get_engine(dbname).connect() as conn:
    df = pd.read_sql(stmt, conn, **kwargs)
    return df

In my case, this error is being thrown from within pandas itself, not in the surrounding code that uses pandas:

  /Users/tommycarpenter/Development/python-indexapi/.tox/py37/lib/python3.7/site-packages/pandas/io/sql.py:1430: SADeprecationWarning: The Engine.run_callable() method is deprecated and will be removed in a future release.  Use the Engine.connect() context manager instead. (deprecated since: 1.4)

The snippet from pandas itself is:

def has_table(self, name, schema=None):
    return self.connectable.run_callable(
        self.connectable.dialect.has_table, name, schema or self.meta.schema
    )

I raised an issue: https://github.com/pandas-dev/pandas/issues/40825

Answer 2

You could try...

connection_string = r'mssql+pyodbc:///?odbc_connect={}'.format(dbString).strip()
engine = sqlalchemy.create_engine(connection_string, echo=False, pool_recycle=3600)
with engine.connect() as connection:
    df.to_sql(name=tablename, con=connection, if_exists='append', index=False)

This approach uses a ContextManager . The ContextManager of the engine returns a connection and automatically invokes connection.close() on it, see . Read more about ContextManager here . Another useful thing to know is, that a connection is a ContextManager as well and handles transactions for you. This means it begins and ends a transaction and in case of an error it automatically invokes a rollback.