python generators for concurrency

Question

I am following slides from Python's guru David Beazley. It states "Generators are also used for concurrency. Here is an example:

from collections import deque

def countdown(n):
    while n > 0:
        print("T-minus", n)
        yield
        n -=1

def countup(n):
    x = 0
    while x > n:
        print("Up we go", x)
        yield
        x +=1

# instantiate some tasks in a queue
tasks = deque([countdown(10),
               countdown(5),
               countup(20)
               ])

# run a little scheduler
while tasks:
    t = tasks.pop()  # get a task
    try:
        next(t)   # run it until it yields
        tasks.appendleft(t) # reschedule
    except StopIteration:
        pass

Here is the output:

T-minus 5
T-minus 10
T-minus 4
T-minus 9
T-minus 3
T-minus 8
T-minus 2
T-minus 7
T-minus 1
T-minus 6
T-minus 5
T-minus 4
T-minus 3
T-minus 2
T-minus 1

The question is how is concurrency introduced by generators and how is it manifesting?

Answer 1

This bit of code implements the concept of "green threads", cooperative, userland (as opposed to Preemptive, kernel) threading.

The "threads" are the generators, each function with yeild or yield from in it. The scheduler lives, obviously, inside the if __name__ == '__main__': bit.

So, lets imagine we have not generators but regular lists, and each list has in it, a sequence of functions.

def doThis(): pass
def sayThis(): pass
def doThat(): pass
...

myThread = [doThis, doThat, doAnother]
yourThread = [sayThis, sayThat, sayAnother]

We could run all of the functions in order:

for thread in [myThread, yourThread]:
    for stmt in thread:
        stmt()

Or we could do them in some other order:

for myStmt, yourStmt in zip(myThread, yourThread):
    myStmt()
    yourStmt()

In the first "scheduler", we exhaust the first thread, and then proceed to the second thread. In the second scheduler, we interleave statements out of both threads, first mine, then yours, then back to mine.

It's because we are interleaving "statements" across multiple "threads" before exausting those threads that we can say that the second scheduler gives concurrency.

Note that concurrency doesn't neccesarily mean parallelism. It's not simultaneous execution, just overlapping.

Answer 2

Here is an example to clarify:

from collections import deque

def coro1():
    for i in range(1, 10):
        yield i

def coro2():
    for i in range(1, 10):
        yield i*10

print('Async behaviour'.center(60, '#'))
tasks = deque()
tasks.extend([coro1(), coro2()])

while tasks:
    task = tasks.popleft()  # select and remove a task (coro1/coro2).
    try:
        print(next(task))
        tasks.append(task)  # add the removed task (coro1/coro2) for permutation.
    except StopIteration:
        pass

Out:

######################Async behaviour#######################
1
10
2
20
3
30
4
40
5
50
6
60
7
70
8
80
9
90