Ordered execution for a Python asyncio.Semaphore

Question

I'm running a number of parallel executions with asyncio.Semaphore over an ordered List .

    async def send_cus(cus_id):
        async with ClientSession(connector=TCPConnector(limit=0)) as session:
            num_parallel = asyncio.Semaphore(20)

            async def send_cust_2(cust_id):
                async with num_parallel:
                    # do something ...

            tasks = list(
                send_cust_2(cust_id)
                for cus_id in my_ordered_lst
            )
            await asyncio.gather(*tasks)

The point is that it executes in an unordered manner, regardless of the my_ordered_lst order. I understand it is because we cannot get guaranteed the execution order for threads or process-forks without syncing elements.

Is there a way to get an ordered execution throughout the List , while still having a parallel execution? Like first send 1st. acquire the lock, the send then send, acquire the lock, and so on.

Answer 1

Prior to Python 3.7, asyncio.gather() started the tasks in an arbitrary order. For 3.7 it was rewritten for efficiency with the side effect that it now creates the tasks in the order specified.

In Python 3.6 and earlier you can work around the issue by spawning the tasks manually in the desired order, and then calling gather to wait for them to finish:

tasks = [asyncio.create_task(send_cust_2(cust_id))
         for cust_id in my_ordered_lst]
await asyncio.gather(*tasks)

Note that this will just start the tasks in the specified order. How they proceed executing past the first await will depend on what the individual tasks do and the timings of their respective operations.