数据的多处理加载并提交到sqlalchemy

Question

I'm loading data from files. I have alot of files, so I have a few processes loading lists of files:

 with concurrent.futures.ProcessPoolExecutor(max_workers=4) as executor:
    for x, y, z in executor.map(load_my_file, path_list):

Load my file: loads data, stores "USERS" and "POSTS" in two dicts and returns them, merges into one dict each for user and posts, and then I bulk commit them.

Each user may have many posts, but in the file each record is just one post and one user together. So this is the reason behind the dict, so I dont have have primary key duplicates on insert with sqlalchemy.

However, this uses up a lot of memory. I have around 1.6 million records, 600k users, and my python program is using up a HUGE amount of memory (more than my 16gb of ram allows).

I looked into using session.merge but it seems to query the database every time I call it, making the process extremely slow. Is there any other way around this? I want to be able to make commits within each process rather than merge it all into one big dict at the end, but I dont want to break any relationships or have primary key errors.

Answer 1

It's pretty strange that parallel loading of 80 local files is much faster than loading it by one at a time. I may propose some reasons for it though.

But, Ok. You can import data to a temporary denormalized table as is. After that, copy data to target normalized tables using SQL queries. Then drop the temporary table (or truncate if you need it on regular basis). Also, look at you SQLAlchemy queries: not only merge degrades performance. "Massive" inserts via add_all are not turn into a single insert, really. You would use insert query with list of dicts: I'm inserting 400,000 rows with the ORM and it's really slow! .

I looked into using session.merge but it seems to query the database every time I call it

It is even worse. It should check record existence (first query) and then insert or update the record (second query). So, it looks questionable to use it for processing of large arrays of data.