使用來自 celery 任務的多處理池引發異常

Question

對於那些閱讀本文的人：我決定改用 RQ，它在運行使用多處理模塊的代碼時不會失敗。 我建議你用那個。

我正在嘗試使用 celery 任務中的多處理池，使用 Python 3 和 redis 作為代理（在 Mac 上運行）。 但是，我似乎甚至無法從 Celery 任務中創建多處理池 object，而是。 我得到一個奇怪的異常，我真的不知道該怎么辦。

誰能告訴我如何做到這一點？

任務：

from celery import Celery
from multiprocessing.pool import Pool

app = Celery('tasks', backend='redis', broker='redis://localhost:6379/0')

@app.task
def test_pool():
    with Pool() as pool:
        # perform some task using the pool
        pool.close()
    return 'Done!'

我使用以下方法將其添加到 Celery：

celery -A tasks worker --loglevel=info

然后通過以下 python 腳本運行它：

import tasks

tasks.test_pool.delay()

返回以下 celery output：

[2015-01-12 15:08:57,571: INFO/MainProcess] Connected to redis://localhost:6379/0
[2015-01-12 15:08:57,583: INFO/MainProcess] mingle: searching for neighbors
[2015-01-12 15:08:58,588: INFO/MainProcess] mingle: all alone
[2015-01-12 15:08:58,598: WARNING/MainProcess] celery@Simons-MacBook-Pro.local ready.
[2015-01-12 15:09:02,425: INFO/MainProcess] Received task: tasks.test_pool[38cab553-3a01-4512-8f94-174743b05369]
[2015-01-12 15:09:02,436: ERROR/MainProcess] Task tasks.test_pool[38cab553-3a01-4512-8f94-174743b05369] raised unexpected: AttributeError("'Worker' object has no attribute '_config'",)
Traceback (most recent call last):
  File "/usr/local/lib/python3.4/site-packages/celery/app/trace.py", line 240, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/usr/local/lib/python3.4/site-packages/celery/app/trace.py", line 438, in __protected_call__
    return self.run(*args, **kwargs)
  File "/Users/simongray/Code/etilbudsavis/offer-sniffer/tasks.py", line 17, in test_pool
    with Pool() as pool:
  File "/usr/local/Cellar/python3/3.4.2_1/Frameworks/Python.framework/Versions/3.4/lib/python3.4/multiprocessing/pool.py", line 150, in __init__
    self._setup_queues()
  File "/usr/local/Cellar/python3/3.4.2_1/Frameworks/Python.framework/Versions/3.4/lib/python3.4/multiprocessing/pool.py", line 243, in _setup_queues
    self._inqueue = self._ctx.SimpleQueue()
  File "/usr/local/Cellar/python3/3.4.2_1/Frameworks/Python.framework/Versions/3.4/lib/python3.4/multiprocessing/context.py", line 111, in SimpleQueue
    return SimpleQueue(ctx=self.get_context())
  File "/usr/local/Cellar/python3/3.4.2_1/Frameworks/Python.framework/Versions/3.4/lib/python3.4/multiprocessing/queues.py", line 336, in __init__
    self._rlock = ctx.Lock()
  File "/usr/local/Cellar/python3/3.4.2_1/Frameworks/Python.framework/Versions/3.4/lib/python3.4/multiprocessing/context.py", line 66, in Lock
    return Lock(ctx=self.get_context())
  File "/usr/local/Cellar/python3/3.4.2_1/Frameworks/Python.framework/Versions/3.4/lib/python3.4/multiprocessing/synchronize.py", line 163, in __init__
    SemLock.__init__(self, SEMAPHORE, 1, 1, ctx=ctx)
  File "/usr/local/Cellar/python3/3.4.2_1/Frameworks/Python.framework/Versions/3.4/lib/python3.4/multiprocessing/synchronize.py", line 59, in __init__
    kind, value, maxvalue, self._make_name(),
  File "/usr/local/Cellar/python3/3.4.2_1/Frameworks/Python.framework/Versions/3.4/lib/python3.4/multiprocessing/synchronize.py", line 117, in _make_name
    return '%s-%s' % (process.current_process()._config['semprefix'],
AttributeError: 'Worker' object has no attribute '_config'

Answer 1

這是芹菜的已知問題 。 它源於台球依賴性中引入的一個問題。 一種解決方法是為當前進程手動設置_config屬性。 感謝用戶@martinth提供以下解決方法。

from celery.signals import worker_process_init
from multiprocessing import current_process

@worker_process_init.connect
def fix_multiprocessing(**kwargs):
    try:
        current_process()._config
    except AttributeError:
        current_process()._config = {'semprefix': '/mp'}

worker_process_init掛鈎將在工作進程初始化后執行代碼。 我們僅檢查_config存在，如果_config存在則對其進行設置。

Answer 2

通過戴維評論中鏈接到的Celery 問題報告中的有用評論，我能夠通過導入billiard模塊的Pool class 來解決這個問題。

代替

from multiprocessing import Pool

和

from billiard.pool import Pool

Answer 3

一種快速的解決方案是使用基於線程的“虛擬” multiprocessing實現。 更改

from multiprocessing import Pool  # or whatever you're using

至

from multiprocessing.dummy import Pool

但是，由於這種並行性是基於線程的，因此通常需要注意（GIL）。