Python：多线程复制-线程仍然存在

Question

Hello my problem is I have a multithreading copying class. And the copying works well, but the program is dont quit because the threads are still alive after copying. I tried to build in a thread event, but this has no effect. The t.join() is never ending because the threads are alive. I also turned them daemonic, but this is unwanted, because the program ends but the threads are still alive when the program stops. Has anyone an idea what is wrong here? The Input of the class is a dataframe with the file source in first column and file destination in the other column

import shutil as sh
from multiprocessing import Queue, Process, Value, Lock, cpu_count 
import threading, os, time,queue


class ThreadedCopy():


totalFiles = 0
copyCount = 0
lock = threading.Lock()

def __init__(self,srcDst):        
    #fileList = srcDst['srcCol']
    self.fileQueue = queue.Queue()
    self.totalFiles = srcDst.shape[0]

    print(str(self.totalFiles) + " files to copy.")
    self.threadWorkerCopy(srcDst)


def CopyWorker(self):
    while True:
    #while True:
        fileRow = self.fileQueue.get()
        sh.copyfile(fileRow[1], fileRow[2])

        self.fileQueue.task_done()
        with self.lock:
            self.copyCount += 1
            percent = (self.copyCount * 100) / self.totalFiles
            if (percent%10==0):
                print(str(percent) + " percent copied.")

def threadWorkerCopy(self, srcDst):
    threads=[]
    for fileRow in srcDst.itertuples():
        self.fileQueue.put(fileRow)
    for i in range(cpu_count()):
        t = threading.Thread(target=self.CopyWorker,name='CopyThread')            
        t.daemon = True
        t.start()
        #threads.append(t)

    self.fileQueue.join()

ThreadedCopy(scrDstDf)

EDIT

If I key interupt the program It is hanging here:

<ipython-input-14-8d9a9b84e73f> in threadWorkerCopy(self, srcDst)
    380         self.stop_event.set()
    381         for thread in threads:
--> 382             thread.join()
    383 
    384 #ThreadedCopy(scrDstDf)

/usr/lib/python3.5/threading.py in join(self, timeout)
   1052 
   1053         if timeout is None:
-> 1054             self._wait_for_tstate_lock()
   1055         else:
   1056             # the behavior of a negative timeout isn't documented, but

/usr/lib/python3.5/threading.py in _wait_for_tstate_lock(self, block, timeout)
   1068         if lock is None:  # already determined that the C code is done
   1069             assert self._is_stopped
-> 1070         elif lock.acquire(block, timeout):
   1071             lock.release()
   1072             self._stop()

KeyboardInterrupt: 

Answer 1

Your worker thread is blocked on self.fileQueue.get() , that's why it's not checking the stop event.

The most easiest way to solve this is to make the thread a daemon thread. That way they'll automatically get terminated when the main thread terminates.

If for some reason you don't want to/can't do this, then you'll need to wake up the worker thread by inserting a special marker value to the queue that your worker will check, if the worker sees this value from the queue, it should terminate itself.