QT function suppressing test failures in python

Question

Working on a complex unittest that is testing a python UI, and a QT function appears to be suppressing test failures. I believe I've been able to create a minimal file that repeats the behavior:

import pytest
from unittest import TestCase
from PySide2.QtCore import QTimer

def meaningless():
    return 'fire'


class TestClass(TestCase):

    def test_1(self):
        def inner_test_1():
            self.assertEqual(meaningless(),'x')
        
        inner_test_1()

    def test_2(self):
        def inner_test_2():
            self.assertEqual(meaningless(),'x')

        QTimer.singleShot(1, inner_test_2)


if __name__ == '__main__':
    import sys

    sys.exit(pytest.main([__file__]))

The first test fails as it should, but the second passes erroneously. In my more complex real-world unit test, the equivalent of "inner_test_2" does indeed fire, and the resulting assertion error can be seen in the test logs, but the test does not register as a failure. From what I can tell in the QT docs, this may have something to do with multi-threading? How do I get failures to fail?

Answer 1

This may have something to do with multi-threading?

Yes, exactly. Failing tests are detected via exceptions and exceptions in a different thread are not caught by the unittest framework, because it can not associate a thread with a test.

A minimal, reproducible example would be:

import pytest
from unittest import TestCase
import threading

class TestClass(TestCase):
    def test_2(self):
        def inner_test_2():
            self.assertTrue(False)
        myTestThread = threading.Thread(target=inner_test_2)
        myTestThread.start()
        myTestThread.join()

if __name__ == '__main__':
    import sys

    sys.exit(pytest.main([__file__]))

Note that while assertTrue(False) raises an exception we do not see a failing test. But what we do see (besides the stacktrace of the exception) is a warning from pytest :

    warnings.warn(pytest.PytestUnhandledThreadExceptionWarning(msg))

-- Docs: https://docs.pytest.org/en/stable/warnings.html

How do I get failures to fail?

You need to catch the exceptions from the thread and raise them in the main thread so the unittest framework can pick them up and understand what test failed. How to do that is a different topic, see Make Python unittest fail on exception from any thread .