Which operations in numpy/scipy are multithreaded?

Question

I'm working on an algorithm, and I've made no attempt to parallelize it other than just by using numpy/scipy. Looking at htop , sometimes the code uses all of my cores and sometimes just one. I'm considering adding parallelism to the single-threaded portions using multiprocessing or something similar.

Assuming that I have all of the parallel BLAS/MKL libraries, is there some rule of thumb that I can follow to guess whether a numpy/scipy ufunc is going to be multithreaded or not? Even better, is there some place where this is documented?

To try to figure this out, I've looked at: https://scipy.github.io/old-wiki/pages/ParallelProgramming , Python: How do you stop numpy from multithreading? , multithreaded blas in python/numpy .

Answer 1

You may try to take IDP package ( Intel® Distribution for Python ) which contains versions of NumPy*, SciPy*, and scikit-learn* with integrated Intel® Math Kernel Library.

This would give you threading of all Lapack routines automatically whether this make sense to do. Here you find out the list of threaded mkl's functions : https://software.intel.com/en-us/mkl-linux-developer-guide-openmp-threaded-functions-and-problems

Answer 2

The routines intrinsic to numpy and scipy allow single threads by default. You can change that if you so choose.

# encoding: utf-8
# module numpy.core.multiarray
# from /path/to/anaconda/lib/python3.6/site-packages/numpy/core/multiarray.cpython-36m-darwin.so
# by generator 1.145
# no doc
# no imports

# Variables with simple values

ALLOW_THREADS = 1

When compiling numpy , you can control threading by changing NPY_ALLOW_THREADS :

./core/include/numpy/ufuncobject.h:#if NPY_ALLOW_THREADS
./core/include/numpy/ndarraytypes.h:        #define NPY_ALLOW_THREADS 1

As for the external libraries, I've mostly found numpy and scipy to wrap around legacy Fortran code ( QUADPACK , LAPACK , FITPACK ... so on). All the subroutines in these libraries compute on single threads.

As for the MKL dependencies, the SO posts you link to sufficiently answer the question.

Answer 3

please try to set the global variable OMP_NUM_THREADS . It works for my scipy and numpy. The functions I use are,

ling.inv() and A.dot(B)