使用相同的 ClientSession 获取多个不同的 url

Question

Normally I code in requests so as a consequence I don't have much experience with aiohttp. But since requests is blocking I have to use aiohttp.

So what my code looks like in requests:

#Account gen code is here using requests 

r = requests.get(product_link)

watch_link = soup(r.text, "html.parser").find("div", {"id": "vi-atl-lnk"}).a["href"]

r = requests.get(watch_link)
r = requests.get(watch_link)   

So what this does is that it goes to an Ebay listing and then uses BS4 to scrape the watch link that is in that listing's source code. It then uses GET request to add the listing to the watch list. There have to be 2 GET requests on the add to watch list link, otherwise it won't actually add it.

Well that was in requests but now I need to write this in aiohttp. The closest I got is this:

session = aiohttp.ClientSession()

async def main():
    #Account gen code is here using aiohttp and session 
    async with session.get(product_link) as resp:
         r = await resp.text()
         watch_link = soup(r, "html.parser").find("div", {"id": "vi-atl-lnk"}).a["href"]
    async with session.get(watch_link) as respp:   
         time.sleep(.1)
    async with session.get(watch_link) as resp:   
         time.sleep(.1)

 loop = asyncio.get_event_loop()
 loop.run_until_complete(main())

I tried this and it ran for me, however it didn't add the item to the watch list. The code above this (not shown as it is not relevant to this problem AFAIK) ran perfectly and made the account. But it is not working when it comes to the watch list bit. What could be the reason for this?

Answer 1

I try so many times and finally found it got an issue on cookies. And you need to change your code to aiohttp.ClientSession(headers=headers) . btw The truth may be in the cookies , where convert ; to \\073

Not aiohttp.ClientSession(headers=headers,cookies=cookies)

There the code i sorted out.

import aiohttp
import asyncio
from bs4 import BeautifulSoup as soup

product_link = ""

cookies = {"Cookie":"_ga=GA1.2.808...."}
headers = {"Connection": "keep-alive"}
headers.update(cookies)

async def main():
    #Account gen code is here using aiohttp and session 
    async with aiohttp.ClientSession(headers=headers) as sessions:

        async with sessions.get(product_link) as resp:
            r = await resp.text()
            watch_link = soup(r, "lxml").find("div", {"id": "vi-atl-lnk"}).a.get("href")
            print(watch_link)

        async with sessions.get(watch_link) as resp:
            pass

        async with sessions.get(watch_link) as resp:
            pass


loop = asyncio.get_event_loop()
loop.run_until_complete(main())

Answer 2

        session = aiohttp.ClientSession()
        async with session.post(link,data=payload,headers=headers) as resp:
            print("Created account with email " + random_catchall)
        async with session.get(product_link,headers=headers) as response:
            r = await response.text()
            watch_link = soup(r, "html.parser").find("div", {"id": "vi-atl-lnk"}).a["href"]

            print("Connected to product")

        async with session.get(watch_link,headers=headers) as respp:
            print("Sucessfully added to watchlist")
        async with session.get(watch_link,headers=headers) as respp:
            print("Added to watch list")


        await session.close()

This ended up working for me. No need for cookies or anything of that sort :)