python - Get content from hundreds of webpages using requests

Question

I have a list of hundreds of domains, and I want to check if a string is in all of the domains on that list.

The problem is that I am getting something like a timeout on the requests (every 20 / 30 domains). I have implemented a try-except form, so that when it happens, the program sleeps for 90 seconds, because I have seen that this could solve the problem. It does solve the problem, but it takes to long.

Is there any more efficient way to do this?

Thank you!!

Answer 1

Instead of sleeping you could continue probing other domains and come back to the domain later.

remaining= domainlist
while(remaining):
    workset=remaining
    remaining=[]
    for domain in workset:
        try: 
            domain.probe()
        except Exception:
            remaining.append(domain)