python: iterate over either a list or an async generator

Question

Since iterators were introduced in python, it's always been possible to not care whether you are dealing with an iterator or a list:

from random import random

def gen_list():
    print('gen')
    for i in range(10):
        yield i

def return_list():
    print('return')
    return [i for i in range(10)]


if random() > 0.5:
    x = gen_list()
else:
    x = return_list()

for i in x:
    pass

PEP 492 introduced asynchronous iterators and the async for syntax. What I can't see is any justification for the new burden of adding syntax to the consumer of the async iterator.

In my code, I sometimes am dealing with a list (from a cache), and sometimes with an async generator:

import asyncio
from random import random

def is_small_and_in_cache():
    if random() > 0.5:
        print('in fake cache')
        return [i for i in range(10)]

async def get_progressively():
    print('gen')
    for i in range(10):
        # e.g. an await here
        await asyncio.sleep(0.1)
        yield i

async def main():
    x = is_small_and_in_cache()
    if x is None:
        x = get_progressively()

    async for i in x:
        pass

asyncio.run(main())

But the above fails (half the time) with TypeError: 'async for' requires an object with __aiter__ method, got list .

Main Question: How to write this so that we can deal with either? Should I try to convert the list to a dummy async generator, or wrap the async generator so that it produces a list?

Side Quest: Are there any proposals to get rid of the (clearly unpythonic, to me) async for construct, ie why can't a regular for loop handle an asynchronous generator? Has Python3x lost it's way in terms of usability??

Answer 1

The syntax exists to warn you that your “loop” might actually include suspending your entire call, allowing other code to run, so that you know to have appropriate data in a consistent state at the top of each iteration. It's not going anywhere.

Of course, a coroutine doesn't have to suspend, and you can use that to make wrapping any iterable trivial:

async def desync(it):
  for x in it: yield x

This is more generally useful than the opposite number which—still asynchronously, as it must—gathers into a list:

async def gather(ai):
  ret=[]
  async for x in ai: ret.append(x)
  return ret

since it allows for proper interleaving in the fully asynchronous case.