Using % operator to check number of iterations

Question

In my program, I use the function for i in range... and since the number of iterations is very high while the speed is relatively slow (cca "10 i per second"), I start the cycle with

if i % 1000 == 0:
    print(i//1000)

showing the progress of program.

My question is, how complicated is this for Python? Does it actually try to divide i by 1000 on every iteration? I do not believe it has tricks for these situations, the interpreter just blindly follows what is written. A human would look at the last 3 digits to check for divisibility (as computers use binary maths, I could try replacing with 1024).

How is this operation therefore complicated and costly? Is there an easier (not by coding, but by execution) solution?

Answer 1

Yes. Python does the modulo operation each time. But, optimizing at this level is generally a waste of energy.

If you really need to optimize your code at that level, implement whatever simple approach you find appropriate, and then profile the function to understand where you need to optimize.

#!/usr/bin/env python3
import cProfile
import time

def dowork():
    # This takes SOME time for each iteration (100 usec here)
    time.sleep(0.0001)

def dolog(i):
    if 0 == i % 1000:
        print(i//1000)

def fn():
    for i in range(100000):
        dowork()
        dolog(i)

cProfile.run('fn()')

This results in:

<snip>
98
99
    300104 function calls in 13.412 seconds

Ordered by: standard name

ncalls  tottime  percall  cumtime  percall filename:lineno(function)
     1    0.000    0.000   13.412   13.412 <string>:1(<module>)
     1    0.120    0.120   13.412   13.412 foo.py:13(fn)
100000    0.183    0.000   13.228    0.000 foo.py:5(dowork)
100000    0.060    0.000    0.064    0.000 foo.py:9(dolog)
     1    0.000    0.000   13.412   13.412 {built-in method builtins.exec}
   100    0.004    0.000    0.004    0.000 {built-in method builtins.print}
100000   13.045    0.000   13.045    0.000 {built-in method time.sleep}
     1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}    

So for all that logging, this code incurring under 0.5% penalty. Obviously your code is different which is why you profile YOUR code, since you often discover other things that can be improved.

Answer 2

The operation if i % 1000 == 0: runs in O(1). Generally if-statements run in O(1) so long as they're simple. (ie not containing any complicated functions that recurse or iterate, etc.) This code is as fast as it's going to get. The only thing faster is O(0) operations, meaning your piece of code would be non-existing.

Python blindly checks each iteration to whether i % 1000 , and if you think about it, so does a human. One has to keep looking at each iteration to determine for divisibility.

If you're interested in speeding this up, look into your iteration. If you can creatively decrease the amount of loops you make or keep each i type int (as opposed to long, etc. You're already doing this with range ), you can decrease the cost of your code. This may not be possible, depending on what your code does.

TL;DR: if i % 1000 == 0 is as simple as it gets. Try decreasing the amount of iterations you have in range to speed your overall code. In some cases, this may not be possible.