内存，磁盘和数据库获取的数据

Question

Let's say that I am going to extract a big dataset from a relational db. However, I do not want to fill more than 100MB of memory (this is an arbitrary limit). Also, I want to perform certain operations on this dataset.

Normally, in a language like python, I would just put all the fetched data in memory. But I would like to avoid this. So, probably I have to introduce a middle step where I write the queried data on disk and then I process them chunk by chunk.

What would be the best way to handle this scenario?

Answer 1

Something like this happened to me recently. A database table without a unique index (has one now) was getting the same data inserted over and over again up to 30 times. The table was about 55 M rows.

I wrote a Python program to find one row, and delete all duplicates. mysqldb crashed on trying to create the query, even before the fetchone call.

However, I was able to extract the data into a spreadsheet, filter using Python's CSV library, and replace the data in the table. It was a mess.

It would be helpful to know the database brand/type in question and the platform you are using, but platform is a little less important.

Edit:

As a rule, I have found that sometimes creating data to be batch loaded can be a lot faster than updating a table one row at a time. I have proved this empirically today by cutting in some changes to calculate and print tax bills. Instead of updating a table in a transaction block (one row at a time) the program prints a delimited "report" (data to be loaded into MySQL) and batch loads it after the bills have been calculated and printed. The speed increase was quite noticeable.