What is the fastest way to do http requests in Python

Question

I am trying to build a web application fuzzer. It will take a wordlist and a url from the user and will do request to those urls. At the end, It will give output according to their responses' status codes.

I have written some code, it does ~600req/s in local (takes about 8 seconds to finish 4600 lines of wordlist) but since I'm using requests library I was thinking if there is a faster way to do so.

Only time consuming part as I analyzed is fuzz() and req() functions as they are doing the most job. I have also other functions but those that I've shown must be enough for you to understand (I didn't want to put so much code).

def __init__(self):
    self.statusCodes = [200, 204, 301, 302, 307, 403]
    self.session = requests.Session()
    self.headers = {
        'User-Agent': 'x',
        'Connection': 'Closed'
        }

def req(self, URL):
# request to only one url
    try:
        r = self.session.head(URL, allow_redirects=False, headers=self.headers, timeout=3)
        if r.status_code in self.statusCodes:
            if r.status_code == 301:
                self.directories.append(URL)
                self.warning("301", URL)
                return
            self.success(r.status_code, URL)
            return
        return
    except requests.exceptions.ConnectTimeout:
        return
    except requests.exceptions.ConnectionError:
        self.error("Connection error")
        sys.exit(1)

def fuzz(self):
    pool = ThreadPool(self.threads)
    pool.map(self.req, self.URLList)
    pool.close()
    pool.join()
    return

#self.threads is number of threads
#self.URLList is a list of full urls 

'__init__' ((<MWAF.MWAF instance at 0x7f554cd8dcb0>, 'http://localhost', '/usr/share/wordlists/seclists/Discovery/Web-Content/common.txt', 25), {}) 0.00362110137939453125 sec

#each req is around this
'req' ((<MWAF.MWAF instance at 0x7f554cd8dcb0>, 'http://localhost/webedit'), {}) 0.00855112075805664062 sec

'fuzz' ((<MWAF.MWAF instance at 0x7f554cd8dcb0>,), {}) 7.39054012298583984375 sec

Whole Program
[*] 7.39426517487

Answer 1

You may wish to combine multiple processes with multiple threads. As 400 threads in 20 processes outperform 400 threads in 4 processes while performing an I/O-bound task shows, there's an optimal number of threads per process -- the more the higher percentage of time they spend waiting for I/O.

On the higher order of vanishing, you can try reusing prepared requests to save on object creation time. (I'm not sure if that'll have an effect -- requests might eg treat them as immutable so it would create a new object each time anyway. But this may still cut on input validation time or something.)