使用 python-asyncio，我如何读取 url 而不是在主 function 中列出 url？

Question

I have a list of urls saved inside a text file called file.txt. How can I have this script read from the txt file and save the response to a text file instead of listing the urls inside the function? There are too many urls to list so the function will get complicated to look at. Thank you

import asyncio
import aiohttp
from codetiming import Timer

async def task(name, work_queue):
    timer = Timer(text=f"Task {name} elapsed time: {{:.1f}}")
    async with aiohttp.ClientSession() as session:
        while not work_queue.empty():
            url = await work_queue.get()
            print(f"Task {name} getting URL: {url}")
            timer.start()
            async with session.get(url) as response:
                await response.text()
            timer.stop()

async def main():
    """
    This is the main entry point for the program
    """
    # Create the queue of work
    work_queue = asyncio.Queue()

    # Put some work in the queue
    for url in [
        "http://google.com",
        "http://yahoo.com",
        "http://linkedin.com",
        "http://apple.com",
        "http://microsoft.com",
        "http://facebook.com",
        "http://twitter.com",
    ]:
        await work_queue.put(url)

    # Run the tasks
    with Timer(text="\nTotal elapsed time: {:.1f}"):
        await asyncio.gather(
            asyncio.create_task(task("One", work_queue)),
            asyncio.create_task(task("Two", work_queue)),
        )

if __name__ == "__main__":
    asyncio.run(main())

Text.file contents

http://google.com
http://yahoo.com
http://linkedin.com
http://apple.com
http://microsoft.com
http://facebook.com

Answer 1

I'm guessing you don't need to load the urls asynchronously. You can load the urls before running the main function. Define the main function so it accepts the parameter urls and finally loop through the urls.

async def main(urls):
  queue = asyncio.Queue()
  for url in urls:
    await queue.put(url)
  
  # rest of your code


if __name__ == "__main__":
  with open("urls.txt") as infile:
    data = infile.read()
    urls = data.split("\n")[:-1]
  asyncio.run(main(urls))