pandas dataframe 到 aws postgresql 使用 sqlalchemy 不将数据注入表

Question

I am running this code:

import sqlalchemy

engine = sqlalchemy.create_engine('postgresql://user:pass@aws_db_endpoint:5432/db')
print(bool(engine)) # <- just to keep track of the process

with engine.connect() as conn:
    print(bool(conn)) # <- just to keep track of the process

    df.to_sql('mytable', schema='public', con=conn, if_exists='append')
    print("end") # <- just to keep track of the process

I get the true , true and end which means I get the connection done and the df.to_slq executed.

THe problem is that the table in aws postresql is still with no data at all.

What I am doing wrong here?

Thanks

Answer 1

Using engine.connect like this I think requires that you call commit() explicitly. You can see the explaination towards the end of the Basic Usage :

When the connection is returned to the pool for re-use, the pooling mechanism issues a rollback() call on the DBAPI connection so that any transactional state or locks are removed, and the connection is ready for its next use.

So in this case you need to call conn.commit() . You can always commit unless there is an exception by changing your usage to engine.begin() :

# Commit is called unless an exception occurs.
with engine.begin() as conn:
    print(bool(conn)) # <- just to keep track of the process

    df.to_sql('mytable', schema='public', con=conn, if_exists='append')
    print("end") # <- just to keep track of the process

Example

Here is an example based on the panda docs. This script runs against an empty database, specifically Base.metadata.create_all(engine) creates the tables in the db first so pandas is forced to append.

import sys
from sqlalchemy import (
    create_engine,
    Integer,
    String,
)
from sqlalchemy.schema import (
    Column,
)
from sqlalchemy.sql import select
from sqlalchemy.orm import declarative_base
import pandas as pd


Base = declarative_base()


username, password, db = sys.argv[1:4]


engine = create_engine(f"postgresql+psycopg2://{username}:{password}@/{db}", echo=False)


class User(Base):
    __tablename__ = "users"
    id = Column(Integer, primary_key=True)
    name = Column(String(8), index=True)


Base.metadata.create_all(engine)


with engine.begin() as conn:
    df = pd.DataFrame({
        'name': ['User 1', 'User 2', 'User 3']
    })

    df.to_sql('users', schema="public", con=conn, if_exists='append', index_label='id')


with engine.begin() as conn:
    for user in conn.execute(select(User)).all():
        print(user.name)