Python sleep without blocking other processes

Question

I am running a python script every hour and I've been using time.sleep(3600) inside of a while loop. It seems to work as needed but I am worried about it blocking new tasks. My research of this seems to be that it only blocks the current thread but I want to be 100% sure. While the hourly job shouldn't take more than 15min, if it does or if it hangs, I don't want it to block the next one that starts. This is how I've done it:

import threading
import time


def long_hourly_job():
    # do some long task
    pass


if __name__ == "__main__":
    while True:
        thr = threading.Thread(target=long_hourly_job)
        thr.start()
        time.sleep(3600)

Is this sufficient?

Also, the reason i am using time.sleep for this hourly job rather than a cron job is I want to do everything in code to make dockerization cleaner.

Answer 1

The code will work (ie: sleep does only block the calling thread), but you should be careful of some issues. Some of them have been already stated in the comments, like the possibility of time overlaps between threads. The main issue is that your code is slowly leaking resources. After creating a thread, the OS keeps some data structures even after the thread has finished running. This is necessary, for example to keep the thread's exit status until the thread's creator requires it. The function to clear these structures (conceptually equivalent to closing a file) is called join . A thread that has finished running and is not join ed is termed a 'zombie thread'. The amount of memory required by these structures is very small, and your program should run for centuries for any reasonable amount of available RAM. Nevertheless, it is a good practice to join all the threads you create. A simple approach (if you know that 3600 s is more than enough time for the thread to finish) would be:

if __name__ == "__main__":
    while True:
        thr = threading.Thread(target=long_hourly_job)
        thr.start()
        thr.join(3600)  # wait at most 3600 s for the thread to finish
        if thr.isAlive(): # join does not return useful information
            print("Ooops: the last job did not finish on time")

A better approach if you think that it is possible that sometimes 3600 s is not enough time for the thread to finish:

if __name__ == "__main__":
    previous = []
    while True:
        thr = threading.Thread(target=long_hourly_job)
        thr.start()
        previous.append(thr)
        time.sleep(3600)
        for i in reversed(range(len(previous))):
            t = previous[i]
            t.join(0)
            if t.isAlive():
                print("Ooops: thread still running")
            else:
                print("Thread finished")
                previous.remove(t)

I know that the print statement makes no sense: use logging instead.

Answer 2

Perhaps a little late. I tested the code from other answers but my main process got stuck (perhaps I'm doing something wrong?). I then tried a different approach trying to emulate the QtCore.QTimer() behavior. I don't think it is extremely precise and it still contains a time.sleep() sentence, but much shorter, so it does not block the main process (at least not significantly):

import threading
import time
import traceback


class Timer:
    SNOOZE = 0
    ONEOFF = 1

    def __init__(self, timerType=SNOOZE):

        self._obj = self._Counter()
        self._kill = threading.Event()
        self._timerType = timerType
        self._msec = 0
        self._thread = None
        self._function = None

    class _Counter:

        def __init__(self):
            super().__init__()

        def run(self, quit_event: threading.Event(), msec: int, callback):
            endTime = round(time.time() * 1000) + msec
            while round(time.time() * 1000) < endTime and not quit_event.is_set():
                time.sleep(0.001)
            if not quit_event.is_set():
                callback()

    def start(self, msec, callback):
        self._function = callback
        self._msec = msec
        if self._function and hasattr(self._function, '__call__') and msec > 0:
            self._thread = threading.Thread(target=self._obj.run, args=(self._kill, self._msec, self._callback))
            self._thread.setDaemon(True)
            self._thread.start()

    def _callback(self):
        try:
            self._function()
            if self._timerType == self.SNOOZE:
                self.start(self._msec, self._function)
        except:
            tb_content = traceback.format_exc()
            print(tb_content)
            self.stop()

    def stop(self):
        self._kill.set()
        self._thread.join()


KEEP = True


def callback():
    global KEEP
    KEEP = False
    print("ENDED", time.strftime("%M:%S"))


if __name__ == "__main__":
    count = 0
    t = Timer(timerType=Timer.ONEOFF)
    t.start(5000, callback)
    print("START", time.strftime("%M:%S"))
    while KEEP:
        if count % 10000000 == 0:
            print("STILL RUNNING")
        count += 1

Notice the while loop and the time.sleep() sentence run in a separate thread, the elapsed time is measured by using time.time(), and uses a callback function to invoke when the time is over (in your case, this callback function would be used to check if the long running process has finished).