在 Python 中循环：在第一次迭代之前做一些事情

Question

I want to optimize.

Simple solution

connection = get_db_connection()
for item in my_iterator:
    push_item_to_db(item, connection)

Drawback:

get_db_connection() is slow. If my_iterator is empty, then I want to avoid to call it.

"if None" solution

connection = None
for item in my_iterator:
    if connection is None:
        connection = get_db_connection()
    push_item_to_db(item, connection)

Drawback:

If there are 100k items in my_iterator , then if connection is None gets called 100k times (although it is needed only once). I want to avoid this.

Perfect solution ...

1. don't call get_db_connection() if iterator is empty
2. don't call if connection is None: uselessly for every iteration.

Any idea?

Answer 1

You can do something like:

connection = None
for item in my_iterator:
    if connection is None:
        connection = get_db_connection()
    push_item_to_db(item, connection)

Simple solution. Don't need to overthink it. Even with 100k operations, x is None is just a reference comparison taking one Python opcode. You really don't need to optimise this compared to a full tcp roundtrip + disk write that happens on every insert.

Answer 2

I am not an expert in Python but I would do something like this:

def put_items_to_database (iterator):
    try:
        item = next(iterator)

        # We connect to the database only after we 
        # know there at least one element in the collection            
        connection = get_db_connection()

        while True:
            push_item_to_db(item, connection)
            item = next(iterator)
    except StopIteration:
        pass

It is probably true that the performance is tied to the database here. However the question is about finding a way to avoid doing unnecessary work, and the above is a basic way of controlling precisely what happens during iteration.

Other solutions are "simpler", in some way, but on the other hand I think this one is more explicit and follows the principle of least astonishment.

Answer 3

for item in my_iterator:
    # First item (if any)
    connection = get_db_connection()
    push_item_to_db(item, connection)
    for item in my_iterator:
        # Next items
        push_item_to_db(item, connection)

Answer 4

Solution 1

This works without a while True loop.

try:
    next(my_iterator)
    connection = get_db_connection()
    push_item_to_db(item, connection)
except StopIteration:
    pass
for item in my_iterator:
    push_item_to_db(item, connection)

Solution 2

If you know that that iterator never returns None (or any other unique object), you could take advantage of the default of next() :

if next(my_iterator, None) is not None:
    connection = get_db_connection()
    push_item_to_db(item, connection)
for item in my_iterator:
    push_item_to_db(item, connection)

Solution 3

If you cannot guaranty a value that never is returned by the iterator, you could use a sentinel.

sentinel = object()
if next(my_iterator, sentinel) is not sentinel:
    connection = get_db_connection()
    push_item_to_db(item, connection)
for item in my_iterator:
    push_item_to_db(item, connection)

Solution 4

Using itertools.chain() :

from itertools import chain

for first_item in my_iterator:
    connection = get_db_connection()
    for item in chain([first_item], my_iterator):
        push_item_to_db(item, connection)

Answer 5

You Could check the iterator count before the entire section of code

if (len(my_iterator)>0): 
    connection = get_db_connection()
    for item in my_iterator:
        push_item_to_db(item, connection)