Python Why Is For-loop Performance Consistently Faster Compared to Using Multiprocessing?

Question

I am trying to learn the multiprocessing library in Python3.9. One thing I compared was the performance of a repeated computation of on a dataset composing of 220500 samples per dataset. I did this using the multiprocessing library and then using for loops.

Throughout my tests I am consistently getting better performance using for loops. Here is the code for the test I am running. I am computing the FFT of a signal with 220500 samples. My experiment involves running this process for a certain amount of times in each test. I am testing this out with setting the number of processes to 10, 100, and 1000 respectively.

 import time import numpy as np from scipy.signal import get_window from scipy.fftpack import fft import multiprocessing from itertools import product def make_signal(): # moved this code into a function to make threading portion of code clearer DUR = 5 FREQ_HZ = 10 Fs = 44100 # precompute the size N = DUR * Fs # get a windowing function w = get_window('hanning', N) t = np.linspace(0, DUR, N) x = np.zeros_like(t) b = 2*np.pi*FREQ_HZ*t for i in range(50): x += np.sin(b*i) return x*w, Fs def fft_(x, Fs): yfft = fft(x)[:x.size//2] xfft = np.linspace(0,Fs//2,yfft.size) return 2/yfft.size * np.abs(yfft), xfft if __name__ == "__main__": # grab the raw sample data which will be computed by the fft function x = make_signal() # len(x) = 220500 # create 5 different tests, each with the amount of processes below # array([ 10, 100, 1000]) tests_sweep = np.logspace(1,3,3, dtype=int) # sweep through the processes for iteration, test_num in enumerate(tests_sweep): # create a list of the amount of processes to give for each iteration fft_processes = [] for i in range(test_num): fft_processes.append(x) start = time.time() # repeat the process for test_num amount of times (eg 10, 100, 1000) with multiprocessing.Pool() as pool: results = pool.starmap(fft_, fft_processes) end = time.time() print(f'{iteration}: Multiprocessing method with {test_num} processes took: {end - start:.2f} sec') start = time.time() for fft_processes in fft_processes: # repeat the process the same amount of time as the multiprocessing method using for loops fft_(*fft_processes) end = time.time() print(f'{iteration}: For-loop method with {test_num} processes took: {end - start:.2f} sec') print('----------')

Here are the results of my test.

 0: Multiprocessing method with 10 processes took: 0.84 sec 0: For-loop method with 10 processes took: 0.05 sec ---------- 1: Multiprocessing method with 100 processes took: 1.46 sec 1: For-loop method with 100 processes took: 0.45 sec ---------- 2: Multiprocessing method with 1000 processes took: 6.70 sec 2: For-loop method with 1000 processes took: 4.21 sec ----------

Why is the for-loop method considerably faster? Am I using the multiprocessing library correctly? Thanks.

Answer 1

There is a nontrivial amount of overhead to starting a new process. In addition the data has to be copied from one process to another (again with some overhead compared to a normal memory copy).

Another aspect is that you should limit the number of processes to the number of cores you have. Going over will make you incurr process switching costs as well.

This, coupled with the fact that you have little computation per process makes the switch not worth while.

I think if you make the signal significantly longer (10x or 100x) you should start seeing some benefits from using multiple cores.

Also check if the operations you are running are already using some parallelism. They might be implemented with threads, which are significantly cheaper the processes (but historically didn't work well in python, dye to GIL).