使用 aiohttp 分離異步請求並保存

Question

我目前正在多次調用外部 API 並從每次調用中下載響應的內容。 我正在使用 aiohttp 和 asyncio 來加速這個過程，但是在弄清楚如何將獲取功能與保存功能分開時遇到了麻煩。

設置

import asyncio
import os

from aiohttp import ClientSession

目前，我正在使用以下 function：

async def fetch_and_save(link, path, client):
    async with await client.get(link) as response:
        contents = await response.read()

        if not os.path.exists(os.path.dirname(path)):
            os.makedirs(os.path.dirname(path))
        with open(path, "wb") as f:
            f.write(contents)

我的主要調用如下所示：

async def fetch_and_save_all(inputs):
    async with ClientSession() as client:
        tasks = [asyncio.ensure_future(fetch_and_save(link, path, client))
                 for link, path in inputs]
        for f in asyncio.as_completed(tasks):
            await f


def main(inputs):
    loop = asyncio.get_event_loop()
    loop.run_until_complete(fetch_and_save_all(inputs))

if __name__ == "__main__":
    inputs = [
        (f"https://httpbin.org/range/{i}", f"./tmp/{i}.txt") for i in range(1, 10)]
    main(inputs)

鑒於這個基本示例，是否可以在fetch_and_save中分離獲取和保存功能？

Answer 1

只需為fetch部分和save部分創建獨立的函數。

async def fetch(link, client):
    async with await client.get(link) as response:
        contents = await response.read()
    return contents

def save(contents, path):
    if not os.path.exists(os.path.dirname(path)):
        os.makedirs(os.path.dirname(path))
    with open(path, 'wb') as f:
        bytes_written = f.write(contents)
    return bytes_written

async def fetch_and_save(link, path, client):
    contents = await fetch(link, client)
    save(contents, path)