How do I propagate an error upward from a worker thread

Question

I'm trying to move some code from singlethread to multithread, but I'm having some trouble breaking the distributor thread if any worker fails. I've got a minimal reproducer below.

When run singlethreaded, an item is produced from the iterator, the item is processed, and if there's an exception, everything terminates with no further work done. However, when run multithreaded, when a worker raises an exception, nothing happens, as the exception dies with the worker. I'd like to capture and raise that exception immediately in the host thread. I've tried various ways to do this with events, but no matter what I try, the host is stuck getting the next item from the iterator.

Is there a way to break the host thread and have it immediately raise any exception seen on any worker?

import threading
q = queue.Queue()
s = set()
for i in range(5):
    q.put(i)
    s.add(i)


def yielder():
    while True:
        ret = q.get()
        if ret is None:
            return
        yield ret


def result(x):
    print(x)
    s.remove(x)
    if not s:
        q.put(None)


def do_work_fn(i):
    if i == 3:
        raise AssertionError("NO THREES!")
    return i


def do_work(work, result_fn):
    for i in work:
        result_fn(do_work_fn(i))



def do_pooled_work(work,result_fn):
    threads = []
    for i in work:
        t = threading.Thread(target=lambda x: result_fn(do_work_fn(x)), args=[i])
        t.start()
        threads.append(t)
    for i in threads:
        i.join()


#This will get stuck
do_pooled_work(yielder(), result)
#This won't
do_work(yielder(),result)

Answer 1

The only way I found for passing an exception from a thread to the main or host thread is by using an instance variable.

class Worker(Thread):
    def __init__(self):
        Thread.__init__(self)
        self.errors = []

    def run(self):
        # appends errors to self.errors

The host thread checks this variable and raises or handles the exceptions.

Sidenote: Set daemon for the worker threads to True so they automatically quit if the host stops