使用数据库时，迭代器如何工作？

Question

Making queries like SELECT * FROM people could create serious memory problems, ie, as many like to call it, "memory exceeded".

Many programmers invoke using iterators to reduce memory consumption. Iterators are a sort of "play-n-pause video recording", meaning that they procuce a value, they stop, they produce a value and so on, avoiding to generate all the values all at once. An example in Python could be the following:

def myIterator():
  for i in range(10):
    yield i * 2

Now, when it comes to use iterators to fetch data from databases, I get pretty confused. I was always told to limit the number of queries toward a database because it could become a bottleneck and, using iterators, it seems to me to increment the number of queries. Is it so? How does an iterator work with database queries? Is this the well-known problem of the short bed sheet, again? In addition, what happens if, during the fetch phase with an iterators, some data is added in the database?

PS My question isn't related to a particular programming language, but it is in general.

Answer 1

RDBMS comes to help when using iterators. DB client sends a request, to which DB replies with the first batch of rows; once the client acknowledges the receipt, the server stops sending, and waits.

DB client library lets the program that uses iterators "digest" the data, until the program iterates past what's already buffered. At that point it asks RDBMS to continue sending the data.

It is up to the program to decide what to do with the data. If it does not store the whole thing in memory, it uses up only as much data as is buffered by DB client.