Nested function call in async way python

Question

I have an api that returns response of pagination only 10 records at a time. I want to process 10 record (index=0 and limit=10) then next 10(index=10 and limit=10) and so on till it returns empty array.

I want to do it in async way.

I am using the following deps:

yarl==1.6.0
Mako==1.1.3
asyncio==3.4.3
aiohttp==3.6.2

The code is:

loop = asyncio.get_event_loop()
loop.run_until_complete(getData(id, token,0, 10))
logger.info("processed all data")



async def getData(id, token, index, limit):
    try:
        async with aiohttp.ClientSession() as session:
            response = await fetch_data_from_api(session, id, token, index, limit)
            if response == []:
                logger.info('Fetched all data')
            else:
                # process data(response)
                getData(session, id, limit, limit+10)
    except Exception as ex:
        raise Exception(ex)


async def fetch_data_from_api(
        session, id, token, index, limit
):
    try:
        url = f"http://localhost:8080/{id}?index={index}&limit={limit}"
        async with session.post(
                url=url,
                headers={"Authorization": token}
        ) as response:
            response.raise_for_status()
            response = await response.json()
            return json.loads(json.dumps(response))
    except Exception as ex:
        raise Exception(
            f"Exception {ex} occurred"
        )

I issue is that it works fine for first time but when i am calling the method getData(session, id, limit, limit+10) again from async def getData(id, token, index, limit). It is not been called.

How can i resolve the issue?

Answer 1

There are a few issues I see in your code.

First, and this is what you talk about, is the getData method. It is unclear to me a bit by looking at the code, what is that "second" getData . In the function definition your arguments are getData(id, token, index, limit) , but when you call it from within the function you call it with getData(session, id, limit, limit+10) where the id is the second parameter. Is that intentional? This looks to me like there is another getData method, or it's a bug.

In case of the first option: (a) you probably need to show us that code as well, as it's important for us to be able to give you better responses, and (b), more importantly, it will not work. Python doesn't support overloading and the getData you are referencing from within the wrapping getData is the same wrapping method.

In case it's the second option: (a) you might have an issue with the function parameters, and (b) - you are missing an await before the getData (ie await getData ). This is actually probably also relevant in case it's the "first option".

---

Other than that, your exception handling is redundant. You basically just re raise the exception, so I don't see any point in having the try-catch blocks. Even more, for some reason in the first method, you create an Exception from the base exception class (not to be confused with BaseException ). Just don't have the try block.