Peewee ORM: how to efficiently iterate over a large resultset

Question

I want my program to start processing rows, as soon as they are received from the MySQL server (many rows and slow connection).

The docs recommend for querying lots of rows : MyModel.select().iterator() .

However, it seems that first the DB server sends all the data, before the iterator yields its first result (verified with tcpdump in another terminal).

I tried accomplishing this with the raw DB drivers MySQLdb and pymysql but there the results seem to get buffered as well.

Is it at all possible? How do other Peewee devs handle iterating over large datasets?

Answer 1

Willem, for this problem, Postgresql provides named cursors (or server-side cursors), which are supported by peewee:

http://docs.peewee-orm.com/en/latest/peewee/playhouse.html#server-side-cursors

I am not super familiar with MySQL but perhaps it provides something similar?

If not you can always use a chunked iterator.