In Python, difference in expected performance of reduce and any/all?

Question

I was trying to find out if the below operations in Python 2.7.3 short circuit or not:

• reduce with a logical 'or' or an 'and'
• any/all

So I defined a function which sleeps for 500ms, prints the passed in value and returns it.

def f(x):
    time.sleep(0.5)
    print 'x', x
    return x

Then I ran the below code using any , which essentially calls f() on each list element and compares the return value to 0. This will happen only on the first element of the list, which is 0.

In [1]: any([f(i)==0 for i in range(3)])
x 0
x 1
x 2

This indicates any does not short-circuit as it calls f() on all the list elements. It just indicates that the entire list is computed first. Next, I used reduce to evaluate the same.

In [2]: reduce(lambda x,y: x or f(y)==0, range(3), False)
x 0

Clearly, reduce short-circuits. *Its only clear that the entire list is not computed once the call back function or short-circuits.* Just to make sure, I also timed the following code

In [3]: timeit reduce(operator.or_, [True] * 10000, True)
1000 loops, best of 3: 390 us per loop

for reduce and

In [4]: timeit any([True] * 10000)
10000 loops, best of 3: 43.8 us per loop

for any . From this one will conclude that any is faster than reduce, because it probably short-circuits. So what is it?

===============

Answer

Wrt performance, the best way forward seems might be to compute a generator from the list and then use any on it.

In [1]: any(f(i)==0 for i in range(3))
x 0

Answer 1

 any([f(i)==0 for i in range(3)]) 

This is the same as:

l = [f(i)==0 for i in range(3)]  # [True, False, False]
any(l)

You're generating the entire list and are running f for every element, before any is invoked. You would need to use a generator instead to avoid having the entire list processed:

any(f(i)==0 for i in range(3))

This now short-circuits as expected.

And as noted by @Jon: reduce never short-circuits, it's only your callback that does. The purpose of any is to return True at the first True that is encountered, the purpose of reduce is to iterate all elements and return a single value at the end; and that's exactly what they do. The problem is in your part of the code here.