Python thread class object and storing via pickle

Question

Within Python, I have been using a thread to execute a pipeline from my main script:

thread = PipeLine(input)
thread.daemon = True
thread.start()
result = thread.join()

My class object in the other script looks this:

class PipeLine(threading.Thread):
    def __init__(self, input=()):
...

I call this code within a GUI, so using a Daemon thread seemed like a good idea to keep the GUI operative, while the process is busy. However, I also want to store the Pipeline class object somehow with pickle after the thread has finished. This is the code I used before.

pipeline = PipeLine(input)
with open(PATHOUT+subj_name+"\\PIPELINE", "wb" ) as f:
    pickle.dump(pipeline, f)  

How do I achieve this in Python 3 without changing what I have too much?

Thanks

Answer 1

First, a note or two. In your code you have:

thread = PipeLine(input)
                thread.daemon = True
                thread.start()
                result = thread.join()

1. The indentation is clearly off.
2. result = thread.join() -> method join() always returns None . What type of result are you looking for? And why are you calling join() on a daemon thread? The whole point of a daemon thread is that the program exists when there are no non-daemon threads left and so daemon threads do not have to be terminated for a program to be exited.
3. The name of your argument is input . You should try to avoid using names that are builtin function names.

Since it is a daemon thread, the intention is for it to run until program termination. But the thread is never "finished" in the normal sense when it is a daemon thread; it runs until there are no non-daemon threads left and then it is killed. Resources it held are not released and its current state is somewhat indeterminate. There is no way to be storing the daemon's thread's state at this point. One would think it might be possible to add to the Pipeline class a __del__ destructor that would do the pickling operation when the thread is garbage collected, but that garbage collection never happens for a daemon thread.

If you want to pickle the Pipeline object after it terminates, it will have to be a non-daemon thread that is joined at termination of the main thread. You will need to give the Pipeline thread the ability to check for a "stop condition" so that it will return from its run method in some consistent state such that a join call from the main thread will not hang.

Answer 2

Thanks for your helpful remarks. I solved the case by simply calling making the dump part of a function and then calling the function as a target. This way, the data is already dumped within the start procedure of the main function. Also noted your comment on using input, I will modify that.