multiprocessing.cpu_count和os.cpu_count之间的区别

Question

Both os and multiprocessing modules define a cpu_count function.

os.cpu_count is documented as follows:

Return the number of CPUs in the system. Returns None if undetermined.

and multiprocessing.cpu_count 's documentation says:

Return the number of CPUs in the system. May raise NotImplementedError. See also os.cpu_count()

On my machine, they both return the same result:

>>> import os
>>> import multiprocessing as mp
>>> os.cpu_count()
8
>>> mp.cpu_count()
8

I would have thought that multiprocessing.cpu_count would be a mere reference to os.cpu_count , but it is not:

>>> os.cpu_count is mp.cpu_count
False

So what is the difference between them? Am I guaranteed that they'll always return the same result? Moreover, if I want to specify a number of processes to create for multiprocessing.Pool , should I use os or multiprocessing 's function?

Answer 1

The answer lies in multiprocessing.context , which defines BaseContext.cpu_count as follows:

# cpython/Lib/multiprocessing/context.py

class BaseContext(object):
    def cpu_count(self):
        '''Returns the number of CPUs in the system'''
        num = os.cpu_count()
        if num is None:
            raise NotImplementedError('cannot determine number of cpus')
        else:
            return num

Then, this cpu_count method is exposed by multiprocessing :

# cpython/Lib/multiprocessing/__init__.py

__all__ = [x for x in dir(context._default_context) if not x.startswith('_')]
globals().update((name, getattr(context._default_context, name)) for name in __all__)

So in the end, multiprocessing.cpu_count is nothing but a wrapper around os.cpu_count .