TypeError: can't pickle _thread.RLock objects in python 3

Question

I have a very large project of a Web API using Flask and Python. It is used for testing some electronic hardware automatically. The program uses some threading in order to run a web UI while a server runs some services (SSH, serial, VISA) among others.

The program was originally coded in python 2.7 and works just fine with this version. Right now, I am trying to update it to python 3.8 for obvious reasons.

As I am updating the project, I'm having trouble with the copy library. It is supposed to serialize a _thread.RLock object and to send it to another thread, but it keeps giving me an error. Here is the traceback that I get:

Traceback (most recent call last):
  File "c:\git_files\[...]\nute\route_config\flask_api_testbench.py", line 208, in _hook_run
    super(FlaskAPITestbench, self).hook_run()
  File "c:\git_files\[...]\nute\core\testbench\base.py", line 291, in hook_run
    while self.state_machine():
  File "c:\git_files\[...]\nute\core\testbench\base.py", line 304, in state_machine
    on_input=self.state_testrun
  File "c:\git_files\[...]\nute\core\testbench\base.py", line 380, in wait_for_input_or_testrun
    self.hook_load_testrun(config_with_input)
  File "c:\git_files\[...]\nute\core\testbench\base.py", line 428, in hook_load_testrun
    self.interface.load_testrun(self.load_testrun(config))
  File "c:\git_files\[...]\nute\core\testbench\base.py", line 461, in load_testrun
    testrun = self.test_loader.load_testrun(config, context_type=self.TestRunContext)
  File "c:\git_files\[...]\nute\core\testrun\loader.py", line 89, in load_testrun
    testrun_template = process_all_loaders(self.batchers, _process_batcher)
  File "c:\git_files\[...]\nute\core\config\loader.py", line 127, in process_all_loaders
    return fn(loader)
  File "c:\git_files\[...]\nute\core\testrun\loader.py", line 85, in _process_batcher
    batcher.batch_testrun(testrun_template, config, context)
  File "c:\git_files\[...]\nute\batcher\python_module_batcher.py", line 21, in batch_testrun
    batch_module.main(testrun, context)
  File "C:\GIT_Files\[...]\pyscripts\script\patest\_batch.py", line 168, in main
    test.suite(ImpedanceTest)
  File "c:\git_files\[...]\nute\core\testrun\base.py", line 213, in suite
    testsuite = testsuite_instance_or_class()
  File "c:\git_files\[...]\nute\core\functions\helpers.py", line 233, in __new__
    cls._attach_nodes_to(template)
  File "c:\git_files\[...]\nute\core\functions\helpers.py", line 271, in _attach_nodes_to
    node = root.import_testcase(testcase)
  File "c:\git_files\[...]\nute\core\functions\specific.py", line 307, in import_testcase
    test_node = testcase.copy(cls=self.__class__)
  File "c:\git_files\[...]\nute\core\functions\base.py", line 645, in copy
    value = copy(value)
  File "c:\users\[...]\.conda\envs\py37\lib\copy.py", line 96, in copy
    rv = reductor(4)
TypeError: can't pickle _thread.RLock objects

It works fine in Python 2.7, but not with Python 3.x. I've tried it on 3.7.10, 3.8.9 and 3.9.6 with the same result.

Here's the implementation of my wrap method of copy :

from copy import copy

...

def copy(self, cls=None): # class method
    if cls is None:  
        cls = self.__class__
    new_self = cls()
    for key, value in self.__dict__.items():
        # if key == "multithread_lock":
        #     continue

        if self.should_copy_attribute(key, value):
            # Handle recursion by pointing to the new object instead of copying.
            if value is self:
                value = new_self
            else:
                value = copy(value)  # This is where it fails
            new_self.__dict__[key] = value
    return new_self

As you can see with the commented part, skipping the pickling of any _thread.RLock object makes the program work, but I need to refresh the web UI manually to see it running since the thread doesn't work.

Any idea why it's working on python 2.7 but not on newer versions? Thanks in advance.

Answer 1

So I found out that a _thread.RLock() object cannot be copied. I just added a condition to skip an object like this to be copied, and it works fine. For the web UI not refreshing, I changed to a lower version of Flask-SocketIO and it worked just fine.