Here's what I'm trying to do-
import lz4.frame
import multiprocessing as mp
pool = mp.Pool(mp.cpu_count())
chunk_size = 64*1024
#screen is a pygame.Surface object converted to bytes
with lz4.frame.LZ4FrameCompressor(compression_level = 10, auto_flush = True) as compressor:
compressed = pool.map(compressor.compress, [screen[i : i + chunk_size] for i in range(0, len(screen), chunk_size)])
compressed = compressor.begin() + b''.join(compressed)
compressed += compressor.flush()
pool.close()
This works just fine when I use map instead of pool.map... With pool.map, nothing happens. Not even an error...
Also, what is the point of block_size argument in the compress function? I tried different combinations of argument block_size (4 and 5) and chunk_size (64k and 256k), but it didn't seem to make much difference.
Doing multiprocessing inside of a context can be hasardous if it is not fork safe.
In the doc ( https://python-lz4.readthedocs.io/en/stable/intro.html ) I read:
The bindings drop the GIL when calling in to the underlying LZ4 library, and is thread safe.
Have you tried multithreading instead ?
import lz4.frame
from concurrent.futures import ThreadPoolExecutor
import os
nb_chunk = 1024
chunk_size = 64*1024
screen = os.urandom(nb_chunk*chunk_size)
context = lz4.frame.create_compression_context()
chunks = [screen[i : i + chunk_size] for i in range(0, len(screen), chunk_size)]
contexts = [context] * len(chunks)
compressed = lz4.frame.compress_begin(context)
with ThreadPoolExecutor(max_workers=os.cpu_count()) as executor:
compressed_chunks = executor.map(lz4.frame.compress_chunk, contexts, chunks)
compressed += b''.join(compressed_chunks)
compressed = lz4.frame.compress_flush(context)
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.