How to deal with TypeError: Object of type 'Thread' is not JSON serializable?

Question

As I mentioned in the title, I'm having trouble setting the Thread object as JSON value. The program gives me TypeError . Here's my code:

from threading import Thread
import json
import time

# Set dict
threads = {}

while True:
    
    data = input('What do you want to call your Thread? ')
    
    # Target func
    def targetfunc(name):
        while True:
            print(f"I'm {name} Thread!")
            time.sleep(3)
            
    # Set thread
    threads[data] = Thread(target = targetfunc, args = [data])
    
    # Create threads file
    with open('threads.json', 'w') as rmd:
        json.dump(threads, rmd)

    # Start thread
    threads[data].start()

    # Print inside the dict
    print("INSIDE THE DICT:", threads)

This code is waiting for you to enter an input . The input you enter becomes Thread name. Here's the output:

What do you want to call your Thread? love

Traceback (most recent call last):
  File "C:\Users\WINDOWS 8.1\Desktop\MarthaAI\mTest\test3.py", line 23, in <modu
le>
    json.dump(threads, rmd)
  File "C:\Users\WINDOWS 8.1\AppData\Local\Programs\Python\Python37-32\lib\json\
__init__.py", line 179, in dump
    for chunk in iterable:
  File "C:\Users\WINDOWS 8.1\AppData\Local\Programs\Python\Python37-32\lib\json\
encoder.py", line 431, in _iterencode
    yield from _iterencode_dict(o, _current_indent_level)
  File "C:\Users\WINDOWS 8.1\AppData\Local\Programs\Python\Python37-32\lib\json\
encoder.py", line 405, in _iterencode_dict
    yield from chunks
  File "C:\Users\WINDOWS 8.1\AppData\Local\Programs\Python\Python37-32\lib\json\
encoder.py", line 438, in _iterencode
    o = _default(o)
  File "C:\Users\WINDOWS 8.1\AppData\Local\Programs\Python\Python37-32\lib\json\
encoder.py", line 179, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type Thread is not JSON serializable

As you can see I am getting TypeError: Object of type Thread is not JSON serializable . My expectation from the program is this:

What do you want to call your Thread? love
I'm love Thread!
INSIDE THE DICT: {'love': Thread(Thread-1, started 4060)}
What do you want to call your Thread?
I'm love Thread!
I'm love Thread!
...

How can I deal with this problem? I want to set Thread object as JSON value. Hope you help.

Answer 1

JSON is a very generic way of serializing all sorts of stuff. However, to do that, it uses a technique called "reflection". This will only work with objects that have a known structure, ie in Python it must be a Python object, in C# it must be a C# object.

As soon as you leave the programming language, this generic way of serialization is no longer possible. In this case, a thread is an object that is near the kernel, so it's likely implemented in C or C++ and thus not serializable. The same applies for other stuff that is optimized for performance, eg numpy, which also uses C++ for its implementation.

To deal with that issue, don't serialize the object directly but serialize all information that is needed to re-create the object from scratch again. For numpy, you would convert numbers into Python numbers. For your thread, serialize the name.

My expectation from the program is [...]

 INSIDE THE DICT: {'love': Thread(Thread-1, started 4060)}

At the time you first output the serialized object, it has not been started yet, so it cannot have a start time.

At this point you should have noticed that deserializing the thread will not work. If you would deserialize the thread, it would attempt to get the start time, but has not been started yet. And when you start it, it will get a new start time, making the deserialization of the start time useless.

Also, think of the thread ID. If a thread with that ID has already been assigned, you can't simply take it over by deserializing. It will necessarily have a different (like random) thread ID.

Answer 2

I don't know why you'd want to serialize a Thread object, but to do it you need to create a custom JSONEncoder that returns a dictionary representing the serialized object and pass it as the cls parameter to json.dump() or json.dumps() .

To get it back you need an object hook that recognizes the serialized object and regenerates the thread.

This is a very rough example. Note it digs into implementation details of the Thread but this is just for demonstration:

import json
import threading
import pickle

class ThreadEncoder(json.JSONEncoder):
    def default(self, obj):
        if isinstance(obj, threading.Thread):
            # serialize the function and arguments used by the thread.
            # the result must be a JSON-serializable object, in this case a "str".
            f = pickle.dumps(obj._target).decode('latin1')
            a = pickle.dumps(obj._args).decode('latin1')
            # return a JSON-serializable dictionary representing a Thread
            return {'thread':f, 'args': a}
        # Let the base class default method raise a TypeError for unsupported objects.
        return json.JSONEncoder.default(self, obj)

# This object hook recognizes a Thread object dictionary created by ThreadEncoder,
# retrieves the function and arguments, and creates a Thread object.
# Note this isn't a perfect reproduction of the Thread object...it doesn't restore other
# possible parameters to the constructor like "daemon" or "name".
def as_thread(dct):
    # assume a 'thread' key indicates a Thread dictionary object.
    if 'thread' in dct:
        f = pickle.loads(dct['thread'].encode('latin1'))
        a = pickle.loads(dct['args'].encode('latin1'))
        return threading.Thread(target=f, args=a)
    return dct # return unchanged if not a Thread.

def func(name):
    print(name)

name = 'mythread'
t = threading.Thread(target=func,args=(name,))

s = json.dumps(t,cls=ThreadEncoder) # serialize thread

del t  # destroy the object

j = json.loads(s,object_hook=as_thread) # restore the object

j.start() # run it.
j.join()

Output:

mythread