Python 多线程/多处理在 concurrent.futures 中非常慢

Question

I am trying to use multithreading and/or multiprocessing to speed up my script somewhat. Essentially I have a list of 10,000 subnets I read in from CSV, that I want to convert into an IPv4 object and then store in an array.

My base code is as follows and executes in roughly 300ms:

aclsConverted = []
def convertToIP(ip):
    aclsConverted.append(ipaddress.ip_network(ip))

for y in acls:
    convertToIP(y['srcSubnet'])

If I try with concurrent.futures Threads it works but is 3-4x as slow, as follows:

aclsConverted = []
def convertToIP(ip):
    aclsConverted.append(ipaddress.ip_network(ip))

with concurrent.futures.ThreadPoolExecutor(max_workers=20) as executor:
    for y in acls:
        executor.submit(convertToIP,y['srcSubnet'])

Then if I try with concurrent.futures Process it 10-15x as slow and the array is empty. Code is as follows

aclsConverted = []
def convertToIP(ip):
    aclsConverted.append(ipaddress.ip_network(ip))

with concurrent.futures.ProcessPoolExecutor(max_workers=20) as executor:
    for y in acls:
        executor.submit(convertToIP,y['srcSubnet'])

The server I am running this on has 28 physical cores.

Any suggestions as to what I might be doing wrong will be gratefully received!

Answer 1

If tasks are too small, then the overhead of managing multiprocessing / multithreading is often more expensive than the benefit of running tasks in parallel.

You might try following:

Just to create two processes ( not threads!!! ), one treating the first 5000 subnets, the other the the other 5000 subnets.

There you might be able to see some performance improvement. but the tasks you perform are not that CPU or IO intensive, so not sure it will work.

Multithreading in Python on the other hand will have no performance improvement at all for tasks, that have no IO and that are pure python code.

The reason is the infamous GIL (global interpreter lock). In python you can never execute two python byte codes in parallel within the same process.

Multithreading in python makes still sense for tasks, that have IO (performing network accesses), that perform sleeps, that call modules, that are implemented in C and that do release the GIL. numpy for example releases the GIL and is thus a good candidate for multi threading

Answer 2

Threading module is for IN/OUT operations, it can't accelarate conversion of strings to ip address instance in no way. When you create threads you spend time on them but do not get any bonus, that's why program without threads is faster.

Multiprocessing module is for heavy computations like find sum of square of all nambers in range from 0 to 1 000 000 000... You operation is not as heavy. Multiprocessing module need to serialize data to send it from one process to another, so it is expensive operation, in addition process spawn is also expenive as a result it is even slower then threading in the case.

I guess that the only way to speed up program is to read csv faster with something like pandas, in addition you can then convert string to ip inside pandas dataframe with apply method.