Python won't unbuffer print statements

Question

I have the following program:

def main():
    print "Running"
    primes = sieve(100000)
    print "Sieve is done"

def sieve(n):
    print "starting sieve"
    primes = []
    times = 0

    numbers = range(2, n):
    print "sieve array filled"

    while len(numbers) > 0:
        current = numbers[0]
        primes.append(current)
        numbers.remove(current)

        times = times + 1
        if (times % 10 == 0):
            print str(times) + "th prime is " + str(current)

        # Remove every multiple
        for i in numbers:
            if (i % current == 0):
                numbers.remove(i)

When finding all the primes up to a large number (lets say ten thousand) I wanted to be able to see how far along the program is by looking at the output. So I decided I would print out every tenth prime. However, when printing it out, it waits until the very end of the program to print it. I added in sys.stdout.flush() right after the print statement but it didn't make any difference. I then tried running the script with python -u <file name> and still, no difference at all.

This is what I get as output:

Running
starting sieve
sieve array filled

Then after about a minute the rest of the output is shown at once.

Why can't I turn the buffer off? I'm trying to modify the code as little as possible.

Answer 1

Having tested a few things, I'm not sure that your problem is actually the output buffering, it's just the behaviour of your algorithm. Try printing current near the top of your while loop, and you'll see that the early numbers take a very long time to work through, and then as numbers gets shorter and shorter, each new value of current gets much faster to process and you start seeing the primes pop up.

Try:

while len(numbers) > 0:
    current = numbers[0]
    print current
    primes.append(current)
    numbers.remove(current)

Answer 2

The reason this is so slow is the loop where you remove elements from numbers:

    # Remove every multiple
    for i in numbers:
        if (i % current == 0):
            numbers.remove(i)

Every time you remove a number, Python has to go and shift all the elements after that number that you removed back one place. Every delete is O(n)*, and you do O(n) deletes, so each iteration of this step takes O(n^2) time.

If you replace this with a list comprehension, then Python builds a new list from the old list -- no moving involved -- which is an O(n) operation. Here's the way I would do that step:

    # Remove every multiple
    numbers = [i for i in numbers if (i % current) != 0]

With this change, your code runs way faster for me. It's done in under 5 seconds, and there's no output buffering problem.

*There's a nice table of time complexities for Python list operations here .

Answer 3

Try using sys.stdout.write instead of print . This should work better with sys.stdout.flush .