在多处理池中运行的 sklearn linear_models.fit() 比在单进程 for 循环中慢

Question

I'v found that LinearRegression.fit() runs slower with python multiprocessing than simple for-loop, here is my code:

import multiprocessing as mp
from contextlib import contextmanager
from sklearn.linear_model import LinearRegression, Ridge, Lasso


def generate_random_data():
    n_samples = int(1e4)
    n_features = 100
    X = np.random.normal(size=(n_samples, n_features))
    beta = np.random.uniform(size=n_features)
    y = X @ beta + np.random.normal(size=n_samples)
    return X, y

@contextmanager
def timeit(tag, container=None):
    t0 = time.perf_counter()
    yield
    t1 = time.perf_counter()
    if container is None:
        print('[{}] : {:.2f} seconds'.format(tag, t1 - t0))
    else:
        container[tag] = t1 - t0

def test(i):
    X, y = generate_random_data()
    m = LinearRegression()    # Ridge, Lasso produce similar results
    with timeit('OLS.fit {}'.format(i)):
        m.fit(X, y)

print('===================== MultiProcessing =====================')
with mp.Pool(10) as pool:
    pool.map(test, range(10))

print('===================== For Loop =====================')
for i in range(10):
    test(i)

output:

===================== MultiProcessing =====================
[OLS.fit 5] : 0.76 seconds
[OLS.fit 9] : 0.98 seconds
[OLS.fit 7] : 1.54 seconds
[OLS.fit 0] : 1.58 seconds
[OLS.fit 4] : 1.67 seconds
[OLS.fit 1] : 1.75 seconds
[OLS.fit 8] : 1.80 seconds
[OLS.fit 2] : 1.96 seconds
[OLS.fit 6] : 2.02 seconds
[OLS.fit 3] : 2.09 seconds
===================== For Loop =====================
[OLS.fit 0] : 0.02 seconds
[OLS.fit 1] : 0.02 seconds
[OLS.fit 2] : 0.02 seconds
[OLS.fit 3] : 0.02 seconds
[OLS.fit 4] : 0.02 seconds
[OLS.fit 5] : 0.02 seconds
[OLS.fit 6] : 0.02 seconds
[OLS.fit 7] : 0.02 seconds
[OLS.fit 8] : 0.02 seconds
[OLS.fit 9] : 0.02 seconds

If I change model from LinearRegression to Lasso, Ridge, the result is almost same. But if chagne to other models, eg DecisionTreeRegressor, multiprocessing version almost consume same time as simple for-loop

my system version: - Ubuntu 18.04 - python 3.7 (Anaconda)

1. Why does this happen?
2. Is there any wrong with my usage of sklearn and/or multiprocessing?

Answer 1

Solved by https://github.com/scikit-learn/scikit-learn/issues/17139

It maybe because LinearRegression calls BLAS do linear algebra computations, which itself by default use all CPU threads, making n process from multiprocessing.Pool will lead to CPU oversubscription.

Should use joblib

Parallel(10, prefer='processes')(delayed(test)(i) for i in range(10))