[英]How to execute python subTests in parallel?
Consider the following unittest.TestCase
, which implements two versions of the same test, the only difference being that one executes the subTest
s in parallel using multiprocessing
.考虑下面的unittest.TestCase
,它实现了同一个测试的两个版本,唯一的区别是一个使用multiprocessing
并行执行subTest
。
import multiprocessing as mp
from unittest import TestCase
class TestBehaviour(TestCase):
def _test_equals(self, val):
for target_val in [1, 2]:
with self.subTest(target=target_val, source=val):
self.assertEqual(val, target_val)
def test_equality_parallel(self):
with mp.Pool(processes=4) as pool:
pool.map(self._test_equals, [1, 2])
pool.join()
pool.close()
def test_equality(self):
for val in [1, 2]:
self._test_equals(val)
The serial version, test_equality
, works as expected and produces the following test failures:串行版本test_equality
按预期工作并产生以下测试失败:
======================================================================
FAIL: test_equality (temp.TestBehaviour) (target=2, source=1)
----------------------------------------------------------------------
Traceback (most recent call last):
File "temp.py", line 11, in _test_equals
self.assertEqual(val, target_val)
AssertionError: 1 != 2
======================================================================
FAIL: test_equality (temp.TestBehaviour) (target=1, source=2)
----------------------------------------------------------------------
Traceback (most recent call last):
File "temp.py", line 11, in _test_equals
self.assertEqual(val, target_val)
AssertionError: 2 != 1
On the other hand, test_equality_parallel
causes an error as the TestCase
cannot be pickled:另一方面, test_equality_parallel
会导致错误,因为TestCase
不能被pickle:
Traceback (most recent call last):
File "temp.py", line 15, in test_equality_parallel
pool.map(self._test_equals, [1, 2])
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/pool.py", line 364, in map
return self._map_async(func, iterable, mapstar, chunksize).get()
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/pool.py", line 768, in get
raise self._value
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/pool.py", line 537, in _handle_tasks
put(task)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/connection.py", line 206, in send
self._send_bytes(_ForkingPickler.dumps(obj))
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/reduction.py", line 51, in dumps
cls(buf, protocol).dump(obj)
TypeError: cannot pickle '_io.TextIOWrapper' object
Now I know that I could split out _test_equals
as a standalone function outside the class;现在我知道我可以将_test_equals
拆分为类之外的独立函数; however, I would like to keep the behaviour of subTest
to enable better logging (and subsequent debugging) of test failures.但是,我想保留subTest
的行为,以便更好地记录(和后续调试)测试失败。
How can I run the tests in parallel, but keep this subTest
functionality?如何并行运行测试,但保留此subTest
功能?
Update更新
I have also attempted this using pathos.multiprocessing.ProcessingPool
to circumvent the issues with TestCase
serialization;我也尝试过使用pathos.multiprocessing.ProcessingPool
来规避TestCase
序列化的问题; however, in this case pool.join()
raises ValueError: Pool is still running
.然而,在这种情况下pool.join()
引发ValueError: Pool is still running
。
from pathos.multiprocessing import ProcessingPool
...
def test_equality_parallel(self):
pool = ProcessingPool(processes=4)
pool.map(self._test_equals, [1, 2])
pool.join()
Update 2更新 2
This question is definitely relevant. 这个问题肯定是相关的。 The first solution proposed, creating a second class for the methods to be called from the child process, isn't appropriate as it wouldn't enable the use of subTest
.提出的第一个解决方案,为要从子进程调用的方法创建第二个类是不合适的,因为它不能使用subTest
。 The second, removing the unpickleable _Outcome
object from TestCase
seems hacky and, given that the child processes are running subTests
, also looks unsuitable.第二个,从TestCase
删除 unpickleable _Outcome
对象似乎很subTests
,并且考虑到子进程正在运行subTests
,看起来也不合适。
I'm the pathos
(and dill
and multiprocess
) author.我是pathos
(以及dill
和multiprocess
)作者。 You are still seeing a serialization error out of pathos
across processes.您仍然看到跨进程的pathos
序列化错误。 You could, try serializing across threads.您可以尝试跨线程序列化。 Thread parallel is probably appropriate for this level of function.线程并行可能适合这种级别的功能。
import multiprocess.dummy as mp
from unittest import TestCase
class TestBehaviour(TestCase):
def _test_equals(self, val):
for target_val in [1, 2]:
with self.subTest(target=target_val, source=val):
self.assertEqual(val, target_val)
def test_equality_parallel(self):
with mp.Pool(processes=4) as pool:
pool.map(self._test_equals, [1, 2])
pool.join()
pool.close()
def test_equality(self):
for val in [1, 2]:
self._test_equals(val)
Which yields:其中产生:
======================================================================
FAIL: test_equality (test_equaltiy.TestBehaviour)
----------------------------------------------------------------------
...[snip]...
AssertionError: 1 != 2
======================================================================
FAIL: test_equality_parallel (test_equaltiy.TestBehaviour)
----------------------------------------------------------------------
...[snip]...
AssertionError: 1 != 2
----------------------------------------------------------------------
Ran 2 tests in 0.108s
FAILED (failures=2)
This tells me that you might be able to use a serialization variant from dill
(ie within dill.settings
) to get around the serialization issue.这告诉我您可以使用dill
的序列化变体(即在dill.settings
)来解决序列化问题。 See: https://github.com/uqfoundation/multiprocess/issues/48 .请参阅: https : //github.com/uqfoundation/multiprocess/issues/48 。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.