asyncio.gather(*coroutines) 没有使用正确的负载

Question

I am new to AsyncIO in Python and i stumbled about a strange behaviour of asyncio.gather(*coroutines) .

In my project a want to request an API and download one report per day. And i want to do parallel requests with asyncio. After that i want to combine the results. I also do some request polling - but anyways: I tried to break down my code to the most simplest version to show you the issue
(including test output; using Python 3.8)

import asyncio

class ReportingService:

    async def _poll_and_get_report(self, payload:dict):

        print(2, payload)

        return [] # just a dummy for illustration

    def _get_report_for_dates(self, payload):

        dates = [
            '20200705',
            '20200706',
            '20200707',
            '20200708',
            '20200709'
        ]

        async def _(payload:dict, dates:list):
            coroutines = []
            for date in dates:
                payload['reportDate'] = date
                print(1, payload)
                coroutines.append(self._poll_and_get_report(payload))

            results =  await asyncio.gather(*coroutines)
            return results

        response_json = []
        results = asyncio.run( _(payload, dates) )
        for json_data in results:
            response_json.extend(json_data)

        return response_json


service = ReportingService()
service._get_report_for_dates({'foo': 'bar'})

This is the output i expect:

1 {'foo': 'bar', 'reportDate': '20200705'}
1 {'foo': 'bar', 'reportDate': '20200706'}
1 {'foo': 'bar', 'reportDate': '20200707'}
1 {'foo': 'bar', 'reportDate': '20200708'}
1 {'foo': 'bar', 'reportDate': '20200709'}
2 {'foo': 'bar', 'reportDate': '20200705'}
2 {'foo': 'bar', 'reportDate': '20200706'}
2 {'foo': 'bar', 'reportDate': '20200707'}
2 {'foo': 'bar', 'reportDate': '20200708'}
2 {'foo': 'bar', 'reportDate': '20200709'}

But this is the output i actually get:

1 {'foo': 'bar', 'reportDate': '20200705'}
1 {'foo': 'bar', 'reportDate': '20200706'}
1 {'foo': 'bar', 'reportDate': '20200707'}
1 {'foo': 'bar', 'reportDate': '20200708'}
1 {'foo': 'bar', 'reportDate': '20200709'}
2 {'foo': 'bar', 'reportDate': '20200709'}  <-- WTF???
2 {'foo': 'bar', 'reportDate': '20200709'}
2 {'foo': 'bar', 'reportDate': '20200709'}
2 {'foo': 'bar', 'reportDate': '20200709'}
2 {'foo': 'bar', 'reportDate': '20200709'}

This is superconfusing and annoying to me. Currently completly blocking the progress of a project. I am sure i am missunderstanding something fundamentally on using coroutines. Maybe payload data is not intended to use at all with coroutines - but what would be a working alternative to it?

Answer 1

Your problem is that you're passing the same dictionary object into each call to _poll_and_get_report . This doesn't actually have anything to do with asyncio; it just seems like it does because the execution of _poll_and_get_report is deferred until you await gather .

You need to either split the date out into a separate argument or pass a new dictionary in each call. The simplest way to do this is

async def _(payload:dict, dates:list):
    coroutines = []
    for date in dates:
        payload['reportDate'] = date
        print(1, payload)
        coroutines.append(self._poll_and_get_report({**payload}))

    results =  await asyncio.gather(*coroutines)
    return results

If you need payload to be preserved for whatever is calling _get_report_for_dates , you'll only want to set reportDate in the new dictionary.

async def _(payload:dict, dates:list):
    coroutines = []
    for date in dates:
        unique_payload = {**payload, **{"reportDate": date}}
        print(1, new_payload)
        coroutines.append(self._poll_and_get_report(new_payload))

    results =  await asyncio.gather(*coroutines)
    return results