使用sqlalchemy将字典列表批量复制到Postgres

Question

I have a list of dictionary list_dict like the one below:

[{u'default_bhp': False, u'price_period': u'Monthly'},{u'default_bhp': False, u'price_period': u'Yearly'}]

At the moment I am inserting this to my db using:

conn = engine.connect()
conn.execute(ModelClass.__table__.insert(), list_dict)

I just wanted to check if there is a faster way to insert data to the database, because I have a huge amount of data.

Can we use a bulk copy or something here?

How to use use_batch_mode functionality?? Something like below:

 engine = create_engine('postgresql+psycopg2://postgres:postgres@localhost/test_db', use_batch_mode=True)
conn = engine.connect()
    conn.execute_batch(ModelClass.__table__.insert(), list_dict)

Answer 1

Consider using bulk_insert_mappings (if you don't know about this already), this is probably closest to what you want to achieve.

But if you're actually have a lot of data, eager unpacking into list of dicts may be not the good way to go, so you may need grouper/chunk management on top of bulk_insert