[英]I'm trying to use multiprocessing for compression in python
這就是我想要做的 -
import lz4.frame
import multiprocessing as mp
pool = mp.Pool(mp.cpu_count())
chunk_size = 64*1024
#screen is a pygame.Surface object converted to bytes
with lz4.frame.LZ4FrameCompressor(compression_level = 10, auto_flush = True) as compressor:
compressed = pool.map(compressor.compress, [screen[i : i + chunk_size] for i in range(0, len(screen), chunk_size)])
compressed = compressor.begin() + b''.join(compressed)
compressed += compressor.flush()
pool.close()
當我使用 map 而不是 pool.map 時,這工作得很好......使用 pool.map,沒有任何反應。 甚至沒有錯誤...
另外,壓縮函數中 block_size 參數的意義是什么? 我嘗試了參數 block_size(4 和 5)和 chunk_size(64k 和 256k)的不同組合,但似乎沒有太大區別。
如果不是 fork 安全,在上下文中進行多處理可能會很麻煩。
在文檔( https://python-lz4.readthedocs.io/en/stable/intro.html )中,我讀到:
調用底層 LZ4 庫時,綁定會刪除 GIL,並且是線程安全的。
您是否嘗試過多線程?
import lz4.frame
from concurrent.futures import ThreadPoolExecutor
import os
nb_chunk = 1024
chunk_size = 64*1024
screen = os.urandom(nb_chunk*chunk_size)
context = lz4.frame.create_compression_context()
chunks = [screen[i : i + chunk_size] for i in range(0, len(screen), chunk_size)]
contexts = [context] * len(chunks)
compressed = lz4.frame.compress_begin(context)
with ThreadPoolExecutor(max_workers=os.cpu_count()) as executor:
compressed_chunks = executor.map(lz4.frame.compress_chunk, contexts, chunks)
compressed += b''.join(compressed_chunks)
compressed = lz4.frame.compress_flush(context)
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.