[英]How to log to single file with multiprocessing.Pool.apply_async
我无法记录到使用 multprocess.Pool.apply_async 的单个文件。 我正在尝试改编 Logging Cookbook 中的这个示例,但它仅适用于multiprocessing.Process
。 将日志记录队列传递到apply_async
似乎没有效果。 我想使用一个池,以便我可以轻松管理并发线程的数量。
以下采用 multiprocessing.Process 的改编示例对我来说工作正常,除了我没有从主进程获取日志消息,而且我认为当我有 100 个大型作业时它不会很好地工作。
import logging
import logging.handlers
import numpy as np
import time
import multiprocessing
import pandas as pd
log_file = 'PATH_TO_FILE/log_file.log'
def listener_configurer():
root = logging.getLogger()
h = logging.FileHandler(log_file)
f = logging.Formatter('%(asctime)s %(processName)-10s %(name)s %(levelname)-8s %(message)s')
h.setFormatter(f)
root.addHandler(h)
# This is the listener process top-level loop: wait for logging events
# (LogRecords)on the queue and handle them, quit when you get a None for a
# LogRecord.
def listener_process(queue, configurer):
configurer()
while True:
try:
record = queue.get()
if record is None: # We send this as a sentinel to tell the listener to quit.
break
logger = logging.getLogger(record.name)
logger.handle(record) # No level or filter logic applied - just do it!
except Exception:
import sys, traceback
print('Whoops! Problem:', file=sys.stderr)
traceback.print_exc(file=sys.stderr)
def worker_configurer(queue):
h = logging.handlers.QueueHandler(queue) # Just the one handler needed
root = logging.getLogger()
root.addHandler(h)
# send all messages, for demo; no other level or filter logic applied.
root.setLevel(logging.DEBUG)
# This is the worker process top-level loop, which just logs ten events with
# random intervening delays before terminating.
# The print messages are just so you know it's doing something!
def worker_function(sleep_time, name, queue, configurer):
configurer(queue)
start_message = 'Worker {} started and will now sleep for {}s'.format(name, sleep_time)
logging.info(start_message)
time.sleep(sleep_time)
success_message = 'Worker {} has finished sleeping for {}s'.format(name, sleep_time)
logging.info(success_message)
def main_with_process():
start_time = time.time()
single_thread_time = 0.
queue = multiprocessing.Queue(-1)
listener = multiprocessing.Process(target=listener_process,
args=(queue, listener_configurer))
listener.start()
workers = []
for i in range(10):
name = str(i)
sleep_time = np.random.randint(10) / 2
single_thread_time += sleep_time
worker = multiprocessing.Process(target=worker_function,
args=(sleep_time, name, queue, worker_configurer))
workers.append(worker)
worker.start()
for w in workers:
w.join()
queue.put_nowait(None)
listener.join()
end_time = time.time()
final_message = "Script execution time was {}s, but single-thread time was {}s".format(
(end_time - start_time),
single_thread_time
)
print(final_message)
if __name__ == "__main__":
main_with_process()
但我无法让以下适应工作:
def main_with_pool():
start_time = time.time()
queue = multiprocessing.Queue(-1)
listener = multiprocessing.Process(target=listener_process,
args=(queue, listener_configurer))
listener.start()
pool = multiprocessing.Pool(processes=3)
job_list = [np.random.randint(10) / 2 for i in range(10)]
single_thread_time = np.sum(job_list)
for i, sleep_time in enumerate(job_list):
name = str(i)
pool.apply_async(worker_function,
args=(sleep_time, name, queue, worker_configurer))
queue.put_nowait(None)
listener.join()
end_time = time.time()
print("Script execution time was {}s, but single-thread time was {}s".format(
(end_time - start_time),
single_thread_time
))
if __name__ == "__main__":
main_with_pool()
我尝试了许多细微的变化,使用 multiprocessing.Manager、multiprocessing.Queue、multiprocessing.get_logger、apply_async.get(),但没有任何工作。
我认为会有一个现成的解决方案。 我应该试试芹菜吗?
谢谢
考虑使用两个队列。 第一个队列是您为工作人员放置数据的地方。 作业完成后,每个工作人员会将结果推送到第二个队列。 现在,使用第二个队列将日志写入文件。
实际上,这里有两个相互独立的问题:
multiprocessing.Queue()
对象作为参数传递给基于Pool的函数(可以将其传递给直接启动的工作程序,但不能传递给它的任何“进一步”对象)。 None
发送给侦听器进程。 要修复第一个,请替换:
queue = multiprocessing.Queue(-1)
有:
queue = multiprocessing.Manager().Queue(-1)
因为可以传递经理管理的Queue()
实例。
要解决第二个问题,请从每个异步调用收集每个结果,或者关闭池并等待它,例如:
pool.close()
pool.join()
queue.put_nowait(None)
或更复杂的:
getters = []
for i, sleep_time in enumerate(job_list):
name = str(i)
getters.append(
pool.apply_async(worker_function,
args=(sleep_time, name, queue, worker_configurer))
)
while len(getters):
getters.pop().get()
# optionally, close and join pool here (generally a good idea anyway)
queue.put_nowait(None)
(您还应该考虑将put_nowait
替换为等待版本的put
而不要使用无限长度的队列。)
[附录] 关于maxtasksperchild=1
你真的不需要它。 重复消息的原因是:您反复向子进程的根记录器添加queuehandlers
处理程序。 以下代码在添加另一个处理程序之前检查是否存在任何处理程序:
def worker_configurer(queue):
root = logging.getLogger()
# print(f'{root.handlers=}')
if len(root.handlers) == 0:
h = logging.handlers.QueueHandler(queue)
root.addHandler(h)
root.setLevel(logging.DEBUG)
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.