[英]Python threading: make the main thread report the progress
我並行運行一些作業,這可能會花費很長時間,因此我希望主線程報告進度。 例如,每小時。
以下是我想出的簡化版本。 該代碼將運行test_function
在2線從參數input_arguments
。 每5秒鍾將打印完成的%作業。
import threading
import queue
import time
def test_function(x):
time.sleep(4)
print("Finished ", x)
num_processes = 2
input_arguments = range(10)
# Define a worker which will continuously execute function taking input parameters from the queue
def worker():
while True:
x = q.get()
if x is None:
break
test_function(x)
q.task_done()
# Initialize queue and the threads
q = queue.Queue()
threads = []
for i in range(num_processes):
t = threading.Thread(target=worker)
t.start()
threads.append(t)
# Create a queue of input parameters for function
for item in input_arguments:
q.put(item)
# Report progress every 5 seconds
report_progress(q)
# stop workers
for i in range(num_processes):
q.put(None)
for t in threads:
t.join()
其中report_progress
定義如下
def report_progress(q):
qsize_init = q.qsize()
while not q.empty():
time.sleep(5)
portion_finished = 1 - q.qsize() / qsize_init
print("run_parallel: {:.1%} jobs are finished".format(portion_finished))
但是,我要每小時而不是5秒報告一次進度,如果所有作業都完成了,則該程序可能只是空閑了幾分鍾。
另一種可能性是以不同的方式定義report_progress
:
def report_progress(q):
qsize_init = q.qsize()
time_start = time.time()
while not q.empty():
current_time = time.time()
if current_time - time_start > 5:
portion_finished = 1 - q.qsize() / qsize_init
print("run_parallel: {:.1%} jobs are finished".format(portion_finished))
time_start = time.time()
我擔心不斷檢查這種情況會浪費CPU資源,但會占用很多時間。
有標准的處理方式嗎?
的Python:3.6
現在,我將使用@Andriy Maletsky的評論中建議的簡單解決方案。
如果q還不為空,主線程將每隔幾秒鍾檢查一次,如果自上次報告以來已超過1個小時,它將打印進度消息。
time_between_reports = 3600
time_between_checks = 5
def report_progress_until_finished(q):
qsize_init = q.qsize()
last_report_time = time.time()
while not q.empty():
time_elapsed = time.time() - last_report_time
if time_elapsed > time_between_reports:
portion_finished = 1 - q.qsize() / qsize_init
print("run_parallel: {:.1%} jobs are finished".format(portion_finished))
last_report_time = time.time()
time.sleep(time_between_checks)
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.