Threaded python application not closing cleanly

Question

I have a small crawling application written in Python 2.7 that uses threads to fetch a lot of URLs. But it doesn't close cleanly or respond properly to a KeyboardInterrupt, although I tried to fix the latter issue with some advice I found here.

def main():
    ...
    for i in range(NUMTHREADS):
        worker = Thread(target=get_malware, args=(malq,dumpdir,))
        worker.setDaemon(True)
        worker.start()

    ...

    malq.join()


if __name__ == "__main__":
    try:
        main()
    except KeyboardInterrupt:
        sys.exit()

I need to make sure that it will exit properly when I hit Ctrl-C or when it completes its run rather than having to Ctrl-Z and kill the job.

Thanks!

Answer 1

There are discussions about how GIL could affect signal handling for Python applications with multiple threads that are IO bound. apparently IO bound threads cause the main thread starve for process time and not be able to handle signals as supposed to. I suggest looking at alternative parallel processing options (like subprocess module, or multiprocessing ) or asynchronous frameworks (like asyncoro )