从 csv 文件读取 url 并在 csv 文件中获取输出时出现问题

Question

The below code is supposed to return the status code and model number of few products from the website https://www.selexion.be/ . It was working fine when i put all the urls in urls array inside the code but when i am fetching url form csv file i am getting this error.

Also i want to store the output url, status code and model number in an array and want to flush( .flush() & os.fsync() ) that array to csv file when all the link's status code and model number are fetched. Because i am getting the output in terminal but i want the output as well in a csv file.

Error:

PS C:\Users\Zandrio> & C:/Users/Zandrio/AppData/Local/Programs/Python/Python38/python.exe "c:/Users/Zandrio/Documents/Advanced Project/Selexion.py"
Traceback (most recent call last):
  File "c:/Users/Zandrio/Documents/Advanced Project/Selexion.py", line 49, in <module>
    asyncio.run(main())
  File "C:\Users\Zandrio\AppData\Local\Programs\Python\Python38\lib\asyncio\runners.py", line 43, in run
    return loop.run_until_complete(main)
  File "C:\Users\Zandrio\AppData\Local\Programs\Python\Python38\lib\asyncio\base_events.py", line 612, in run_until_complete
    return future.result()
  File "c:/Users/Zandrio/Documents/Advanced Project/Selexion.py", line 41, in main
    await asyncio.gather(*(worker(f'w{index}', url, session)
  File "c:/Users/Zandrio/Documents/Advanced Project/Selexion.py", line 32, in worker
    response = await session.get(url, headers=header)
  File "C:\Users\Zandrio\AppData\Local\Programs\Python\Python38\lib\site-packages\aiohttp\client.py", line 380, in _request
    url = URL(str_or_url)
  File "C:\Users\Zandrio\AppData\Local\Programs\Python\Python38\lib\site-packages\yarl\__init__.py", line 149, in __new__
    raise TypeError("Constructor parameter should be str")
TypeError: Constructor parameter should be str

Code:

import asyncio
import csv
import aiohttp
import time
from bs4 import BeautifulSoup

urls = []

try:

 with open('C:\\Users\\Zandrio\\Documents\\Advanced Project\\input_links.csv','r', newline='') as csvIO:
    urls = list(csv.reader(csvIO))

except FileNotFoundError:
    pass


header = {
'Host': 'www.selexion.be',
'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Firefox/68.0',
'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
'Accept-Language': 'en-US,en;q=0.5',
'Accept-Encoding': 'gzip, deflate, br',
'Connection': 'keep-alive',
'Upgrade-Insecure-Requests': '1',
'Cache-Control': 'max-age=0',
'TE': 'Trailers'
}


async def worker(name, url, session):
    response = await session.get(url, headers=header)
    html = await response.read()
    soup = BeautifulSoup(html, features='lxml').select_one('.title-options span:first-of-type').text
    print(f'URL: {url} - {response.status} - {soup}')


async def main():
     async with aiohttp.ClientSession() as session:
            await asyncio.gather(*(worker(f'w{index}', url, session)
                            for index, url in enumerate(urls)))


if __name__ == '__main__':
    start = time.perf_counter()
    asyncio.run(main())
    elapsed = time.perf_counter() - start
    print(f'Executed in {elapsed:0.2f} seconds')

Answer 1

The error message says its a type error, meaning a function returns a parameter of type A, but type B was passed.

TypeError: Constructor parameter should be str

In the line

await asyncio.gather(*(worker(f'w{index}', url, session)

what is the type of url? You can find out with

type(url)

or by running your debugger.

What happens when you flip those two lines

 await asyncio.gather(*(worker(f'w{index}', url, session)
                            for index, url in enumerate(urls)))

I don't see where url comes from otherwise.