將 dask 與多處理調度程序一起使用時出現 Loguru 錯誤

Question

嘗試使用啟用了enqueue的記錄器處理程序從dask.delayed function 記錄時，我收到了RuntimeError 。

這是一個最小的可重現示例

import sys, dask
from loguru import logger

logger.add(sys.stderr, enqueue=True)

@dask.delayed
def log():
    logger.info("Logging!")

dask.compute(*[log() for i in range(10)], scheduler="processes")

這會產生以下錯誤：

RuntimeError: SimpleQueue objects should only be shared between processes through inheritance

Answer 1

解決方案很簡單。 從delayed的 function 中導入記錄器：

In [1]: import sys, dask
   ...: from loguru import logger
   ...: 
   ...: logger.add(sys.stderr, enqueue=True)
   ...: 
   ...: @dask.delayed
   ...: def log():
   ...:     from loguru import logger
   ...:     logger.info("Logging!")
   ...: 
   ...: dask.compute(*[log() for i in range(10)], scheduler="processes")
2022-09-13 21:42:36.001 | INFO     | __main__:log:9 - Logging!
2022-09-13 21:42:36.001 | INFO     | __main__:log:9 - Logging!
2022-09-13 21:42:36.001 | INFO     | __main__:log:9 - Logging!
2022-09-13 21:42:36.001 | INFO     | __main__:log:9 - Logging!
2022-09-13 21:42:36.002 | INFO     | __main__:log:9 - Logging!
2022-09-13 21:42:36.002 | INFO     | __main__:log:9 - Logging!
2022-09-13 21:42:36.004 | INFO     | __main__:log:9 - Logging!
2022-09-13 21:42:36.004 | INFO     | __main__:log:9 - Logging!
2022-09-13 21:42:36.004 | INFO     | __main__:log:9 - Logging!
2022-09-13 21:42:36.004 | INFO     | __main__:log:9 - Logging!
Out[1]: (None, None, None, None, None, None, None, None, None, None)