从大型数据库读取时SQLite3内存错误（> 1GB）

Question

I have a small Python 2.7 script that uses LIKE statements to extract pieces of information from textual data stored in a SQLite database.

sql = "SELECT user_id, loc,\
        FROM entity\
        WHERE loc LIKE '%\"place\":%'\
        AND loc LIKE '%\"geo\":%'\
    AND loc LIKE '%\"coordinates\":%'" 
cin.execute(sql)
entities = cin.fetchall()

cin is a cursor to a SQLite database (table entity with >10^6 rows) (~1.5GB) which was established using

import sqlite3
try:
    dbin = sqlite3.connect(database=args['dbi'].name)
    dbin.row_factory = sqlite3.Row
    cin = dbin.cursor()
except sqlite3.Error, e:
    errorLogger.error('... %e' % e)
    sys.exit()

The script ran fine with database sizes of 10^2 MB, but now I get a

Traceback (most recent call last):
  File "C:\Users\...\migrate.py", line 247, in <module>
    entities = cin.fetchall()
MemoryError

after some seconds. I am running a W7 64bit machine with 8GB RAM . When the script is running, by looking at the resources monitor of W7, I can tell that successively all free memory is used and python.exe consumes as much as 1.9GB just before the program crashes. Still, there is about 3GB standby memory available (but don't ask me what's the difference between standby and free memory).

What can I do about this besides pre-filtering my query, eg by only looking at, let's say, 10'000 rows per query?

Answer 1

Calling fetchall requires allocating memory for all result records.

You should instead read the result records from cin one by one.