多处理中的Python巨大文件读取

Question

I have binary file that contains invariable number of images (size of each image 1024*768). I put each image to JoinableQueue and analyzed it in multiprocessing, and it works perfect with small files, but I get Memory Error when I try to read huge files. Anybody know how can i store big files to bufer/Queue(as string)? (unfortunately i can't use Manager or Pool)

Answer 1

Did you have a look at the module io.BytesIO? You can find it here: https://docs.python.org/release/3.1.3/library/io.html#binary-io You can set your Buffer size, that solved a memory problem for me once.

Answer 2

1. You can read about buffer here .
2. If your memory if small, you can try force gc like that:

import gc

SIZE = 1024*768  
MEMOSIZE = 1024  # your memory size
with open('xxx', 'rb') as fp:  # open the file
    i = 0  # remember the number to gc in time
    queue = []
    while True:
        if (i*(SIZE-1) < MEMOSIZE):
            x = fp.read(SIZE)  # if your image is single channel
            queue.append(x)
            # do something
        else:
            del queue
            gc.collect()