我应该使用什么来加速此代码？ （多处理与多线程）

Question

Could you also show me how? Thanks in advance.

Here's the code:

def test():
    with open("proxies.txt", "r") as f:
        for line in f:
            try:
                proxy = line.split('\n', 1)[0]
                r = requests.get('http://www.icanhazip.com/', proxies={'http': 'http://' + proxy}, timeout=1)
                print(r.status_code)
            except (requests.exceptions.ProxyError, requests.exceptions.ConnectTimeout,
                requests.exceptions.ReadTimeout, requests.exceptions.ConnectionError):
                print("Doesn't work")

Multiprocessing or multithreading should only start when the function is called.

Answer 1

I would think threading would be best, it doesn't seem that you are performing large amount of computational work with each worker. Sub-processes take a decent amount of overhead to get started, and are therefore more suited for tasks requiring large amounts of computations.

Answer 2

Two observations:

1. You can try using the ThreadPoolExecutor or ProcessPoolExecutor from concurrent.futures library, so you can parallelize the execution.

2. You may want to see if creating explicit requests Session and reusing the session speed this up. This may save some cost in TLS renegotiation/handshake. Note that you may need to be careful with cookies as reused sessions will share a cookie jar by default.

Untested, quickly scratched together example:

session = requests.Session()
def do_request(line):
    proxy = line.split('\n', 1)[0]
    r = session.get('http://www.icanhazip.com/', proxies={'http': 'http://' + proxy}, timeout=1)
    return r.status_code

with ThreadPoolExecutor(max_workers=8) as executor, \
        open("proxies.txt", "r") as f:
    results = executor.map(do_request, f)
    for future in results:
        try:
            print(future.result())
        except (requests.exceptions.ProxyError, requests.exceptions.ConnectTimeout,
                requests.exceptions.ReadTimeout, requests.exceptions.ConnectionError):
            print("Doesn't work")