Using Threading and Queue in infinite loop does not close threads

Question

I'm struggling to have python periodically run some threaded function and close threads (or reuse the same threads without spawning new ones). When running the code below it spawns new threads every time the function is ran and eventually the program crashes/stops all together.

Is it possible to close threads when the task is done?

import threading
from Queue import Queue
import time
import socket

number_of_threads = 10

def check_server(address):
    s = socket.socket()
    s.settimeout(1)
    try:
        s.connect((address, 443))
        return True
    except:
        return False

def get_online_servers(subnet):
    def threader():
        while True:
            global online_servers
            ip = q.get()
            a = check_server(ip)
            if a:
                online_servers.append(ip)
            q.task_done()

    q = Queue()

    for x in range(number_of_threads):
        t = threading.Thread(target=threader)
        t.daemon = False
        t.start()

    for i in range(1, 20):
        ip = "{}.{}".format(subnet, i)
        q.put(ip)

    q.join()

    return online_servers

if __name__ == '__main__':
    while True:
        online_servers = []
        result = get_online_servers("10.0.0")
        print result
        time.sleep(5)

Answer 1

Threads automatically finish when they finish their function. (You do usually want to keep the thread objects around and join them somewhere, but that's not relevant to your problem here.)

So, if you want them to finish after each task, why did you put a while True: loop inside the function? Just that take out and everything will work as you requested.

It can be useful to have threads that run an infinite loop until the queue is done (and then break / return ), but that's a different design. For that design, you'd just want to start 10 threads at startup, rather than start a new batch of threads for each request. (Although in that case, you're just building a bog-standard thread pool, so it may be easier to use the already-existing one in the stdlib—or the slightly higher-level concurrent.futures.ThreadPoolExecutor .)

Finally, if what you want is really a new pool of 10 threads for each batch of requests, you can do that. It's basically all the complexity of both alternatives, without any advantages over either, but it's doable. Just as in the persistent thread pool, you'll need to change the thread function to exit when the queue is done. And you'll probably need to keep all 10 threads in a list and join them all after joining the queue. But that should work.