进程池执行器内部的线程池执行器运行时错误：线程中没有当前事件循环

Question

我有一个 processpoolexecutor，我向其中提交多个磁盘读/写调用。 我想在每个进程中创建一个线程池以提高性能。

下面是我尝试覆盖和修改 concurrent.futures process.py 的 _process_worker 方法以与 ProcessPoolExecutor 一起使用。 我正在尝试在 ThreadPoolExecutor 中运行 function -

from concurrent.futures import process as process_futures
class ProcessPoolExecutor(process_futures.ProcessPoolExecutor):
    """Override process creation to use our processes"""
    def _adjust_process_count(self):
        """This is copy-pasted from concurrent.futures to override the Process class"""
        for _ in range(len(self._processes), self._max_workers):
            p = Process(
                target=_process_worker,  
                args=(self._call_queue, self._result_queue, None, None))
            p.start()
            self._processes[p.pid] = p

def _process_worker(call_queue, result_queue):
    with ThreadPoolExecutor(max_workers=8) as executor: # starting a Threadpool
        while True:
            call_item = call_queue.get(block=True)
            if call_item is None:
                # Wake up queue management thread
                result_queue.put(os.getpid())
                return
            try:
                if 1: # my changes , problem with this code
                    future = executor.submit(call_item.fn, *call_item.args, **call_item.kwargs)
                    future.add_done_callback(
                        functools.partial(_return_result, call_item, result_queue))
                else: # original code with only processpool as in futures process.py
                    r = call_item.fn(*call_item.args, **call_item.kwargs)
            except BaseException as e:
                result_queue.put(process_futures._ResultItem(call_item.work_id,
                                             exception=e))
            else:
                result_queue.put(process_futures._ResultItem(call_item.work_id,
                                             result=r))

当我在 processpoolexecutor 中添加一个 threadpoolexecutor 时，出现以下错误

RuntimeError: There is no current event loop in thread '<threadedprocess._ThreadPoolExecutor object at 0x000001C5897B1FA0>_0'.

我知道事件循环不是在子线程上创建的，所以它抱怨没有当前事件循环。 所以，即使我添加新的事件循环 -

def _process_worker(call_queue, result_queue, a, b):
  
    try:
        import asyncio
        loop = asyncio.new_event_loop()
        asyncio.set_event_loop(loop)
        
    except Exception as e:
        logger.info("eexception {} ".format(e))
    with ThreadPoolExecutor(max_workers=8) as executor:
        while True:
            call_item = call_queue.get(block=True)
            if call_item is None:
                # Wake up queue management thread
                result_queue.put(os.getpid())
                return
            try:
                if 1: # my changes , problem with this code
                   job_func = functools.partial(call_item.fn, *call_item.args, **call_item.kwargs)
                    
                    try:
                        
                        loop.run_in_executor(executor, job_func)
                    except Exception as e:
                        logger.info("exception recvd {}".format(e))

                else: # original code with only processpool as in futures process.py
                        r = call_item.fn(*call_item.args, **call_item.kwargs)
            except BaseException as e:
                result_queue.put(process_futures._ResultItem(call_item.work_id,
                                             exception=e))
            else:
                result_queue.put(process_futures._ResultItem(call_item.work_id,
                                             result=r))

我收到一个新错误 -

concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.

我如何更改 _process_worker 以在线程池中运行工作？ 请有任何建议。

Answer 1

您可以使用asyncio.run(your_async_function_here())创建新的事件循环并将给定的 function 安排为该事件循环中的任务。