how to open different urls at the same time by using python selenium?

Question

is this possible to open 50 different urls at the same time using selenium with python? is this possible using threading?

If so, how would I go about doing this? If not, then what would be a good method to do so?

Answer 1

well, open 50 urls at the same time seems unreasonable and will require a LOT of processing, but, is possible. However i would recommend you to use a form of iteration opening one url at a time. 50 times.

list = ['list of urls here','2nd url'...]
driver = webdriver.Firefox()
for i in list:
  moving = driver.get(i)
  ...#rest of your code
driver.quit()

but... you can make one driver.get('url') for each url you want... using different drivers. Or tabs. But it will require a lot of processing.

Answer 2

You can try below to open 50 URLs one by one in new tab :

urls = ["http://first.com", "http://second.com", ...]

for url in urls:
    driver.execute_script('window.open("%s")' % url)

Answer 3

you can use celery (Distributed Task Queue) to open all these urls.

or you can use async and await with aiohttp on python >= 3.5 , which runs a single thread on a single process but concurrently(utilises the wait time on urls for fetching other urls) here is the code sample for the same. Loop takes care of scheduling these concurrent tasks.

#!/usr/local/bin/python3.5
import asyncio
from aiohttp import ClientSession

async def hello(url):
    async with ClientSession() as session:
        async with session.get(url) as response:
           response = await response.read()
           print(response)

loop = asyncio.get_event_loop()

loop.run_until_complete(hello("http://httpbin.org/headers"))