Is it possible to mock.patch
a function that will be run in a subprocess, which is created using the multiprocessing
module with the start method
set to spawn
?
If there is no solution to patching a child process which is not forked
, what would be the right solution to bypass this problem?
It's important to say that switching to use fork is not a solution to my problem. As of python3.8-macOS, the default behavior for the multiprocessing start method is spawn
.
From the multiprocessing
documentation:
Changed in version 3.8: On macOS, the spawn start method is now the default. The fork start method should be considered unsafe as it can lead to crashes of the subprocess. See bpo-33725.
.
Example code (Run on macos with python >= 3.8):
import multiprocessing
import unittest
from unittest import mock
def baz(i):
print(i)
def bar(i): # Middle function to avoid a pickeling problem with mocks
baz(i)
def foo():
with multiprocessing.Pool(2) as pool:
pool.map(bar, [i for i in range(10)])
class TestClass(unittest.TestCase):
@mock.patch(f'{__name__}.baz', return_value=None)
def test_case(self, mock):
# multiprocessing.set_start_method('fork', force=True)
foo()
The baz
function is not patched in the spawned processes (and hence, it prints) as can be seen in the following output. Changing the default start method (commented in code) solves the problem
Output:
============================================================================== test session starts ===============================================================================
platform darwin -- Python 3.8.0, pytest-5.4.3, py-1.8.2, pluggy-0.13.1
rootdir: /Users/alonmenczer/dev/path_test/proj
collected 1 item
mp_spawn.py 0
1
4
5
6
7
8
9
2
3
.
=============================================================================== 1 passed in 0.27s ================================================================================
I assume you would rather prefer to use mock
. But if the library doesn't work, mocking can be achieved directly. Eg:
def my_mock_baz(func):
def _mock_baz(i):
print(42)
def inner(*args, **kwargs):
global baz
backup = baz
baz = _mock_baz
res = func(*args, **kwargs)
baz = backup
return res
return inner
And then in the test:
class TestClass(unittest.TestCase):
@my_mock_baz
def test_case(self):
foo()
Of course, I other cases, mocking would be more difficult, but with the flexibility of Python it should be not too hard.
Side note: your problem doesn't reproduce in my Ubuntu 20.04, Python 3.8.2.
yes it definitely is it just depends how
from multiprocessing import Process
import os
def info(title):
print(title)
print('module name:', __name__)
print('parent process:', os.getppid())
print('process id:', os.getpid())
def f(name):
info('function f')
print('hello', name)
if __name__ == '__main__':
info('main line')
p = Process(target=f, args=('bob',))
p.start()
p.join()
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.