Python 3 concurrent.futures：如何將失敗的期貨添加回ThreadPoolExecutor？

Question

我有一個url列表可以通過concurrent.futures的ThreadPoolExecutor下載，但可能有一些超時URL我想在所有第一次嘗試結束后重新下載它們。 我不知道怎么做，這是我的嘗試，但失敗了無休止的打印'time_out_again'：

import concurrent.futures

def player_url(url):
    # here. if timeout, return 1. otherwise do I/O and return 0.
    ...

urls = [...]
time_out_futures = [] #list to accumulate timeout urls
with concurrent.futures.ThreadPoolExecutor(max_workers=10) as executor:
    future_to_url = (executor.submit(player_url, url) for url in urls)
    for future in concurrent.futures.as_completed(future_to_url):
        if future.result() == 1:
            time_out_futures.append(future)

# here is what I try to deal with all the timeout urls       
while time_out_futures:
    future = time_out_futures.pop()
    if future.result() == 1:
        print('time_out_again')
        time_out_futures.insert(0,future)   # add back to the list

那么，有什么方法可以解決這個問題嗎？

Answer 1

Future對象只能使用一次。 Future本身對返回結果的函數一無所知 - ThreadPoolExecutor對象負責創建Future ，返回它，並在后台運行函數：

def submit(self, fn, *args, **kwargs):
    with self._shutdown_lock:
        if self._shutdown:
            raise RuntimeError('cannot schedule new futures after shutdown')

        f = _base.Future()
        w = _WorkItem(f, fn, args, kwargs)

        self._work_queue.put(w)
        self._adjust_thread_count()
        return f

class _WorkItem(object):
    def __init__(self, future, fn, args, kwargs):
        self.future = future
        self.fn = fn
        self.args = args
        self.kwargs = kwargs

    def run(self):
        if not self.future.set_running_or_notify_cancel():
            return

        try:
            result = self.fn(*self.args, **self.kwargs)  # sefl.fn is play_url in your case
        except BaseException as e:
            self.future.set_exception(e)
        else:
            self.future.set_result(result)  # The result is set on the Future

如您所見，當函數完成時，結果將在Future對象上設置。 因為Future對象實際上並不知道有關提供結果的函數的任何信息，所以無法嘗試使用Future對象重新運行該函數。 您所能做的就是在超時發生時將url與1一起返回，然后將url重新submit給ThreadPoolExecutor ：

def player_url(url):
    # here. if timeout, return 1. otherwise do I/O and return 0.
    ...
    if timeout:
        return (1, url)
    else:
        return (0, url)

urls = [...]
with concurrent.futures.ThreadPoolExecutor(max_workers=10) as executor:
    while urls:
        future_to_url = executor.map(player_url, urls)
        urls = []  # Clear urls list, we'll re-add any timed out operations.
        for future in future_to_url:
            if future.result()[0] == 1:
                urls.append(future.result()[1]) # stick url into list