Python读取大二进制文件的一部分

Question

I have large binary file ( size ~2.5Gb ). It contains header (size 336 byte) and seismic signal data (x, y and z channels) with type int32. Count of discrete is 223 200 000. I need read part of signal. For example, I want get part of signal in interval of discrete [216 000 000, 219 599 999]. I wrote the function:

def reading(path, start_moment, end_moment):
    file_data = open(path, 'rb')
    if start_moment is not None:
        bytes_value = start_moment * 4 * 3
        file_data.seek(336 + bytes_value)
    else:
        file_data.seek(336)

    if end_moment is None:
        try:
            signals = np.fromfile(file_data, dtype=np.int32)
        except MemoryError:
            return None
        finally:
            file_data.close()
    else:
        moment_count = end_moment - start_moment + 1
        try:
            signals = np.fromfile(file_data, dtype=np.int32,
                                  count=moment_count * 3)
        except MemoryError:
            return None
        finally:
            file_data.close()
    channel_count = 3
    signal_count = signals.shape[0] // channel_count
    signals = np.reshape(signals, newshape=(signal_count, channel_count))
    return signals

If I run script with the function in PyCharm IDE I get error:

Traceback (most recent call last): File "D:/AppsBuilding/test/testReadBaikal8.py", line 41, in signal_2 = reading(path=path, start_moment=216000000, end_moment=219599999) File "D:/AppsBuilding/test/testReadBaikal8.py", line 27, in reading count=moment_count * 3) OSError: obtaining file position failed

But if I run script with parameters: start_moment=7200000, end_moment=10799999 all ok. On my PC was installed Windows7 32bit. Memory size is 1.95Gb Please, help me resolve this problem.

Answer 1

Divide the file into small segments, freeing memory after each small piece of content is processed

def read_in_block(file_path):
    BLOCK_SIZE = 1024
    with open(file_path, "r") as f:
        while True:
            block = f.read(BLOCK_SIZE)  
            if block:
                yield block
            else:
                return  

        print block

Answer 2

I don't use Numpy but I don't see anything obviously wrong with your code. However, you say the file is approximately 2.5 GB in size. A triplet index of 219,599,999 requires a file at least 2.45 GB in size:

$ calc
; 219599999 * 4 * 3
    2635199988
; 2635199988 / 1024^3
    ~2.45422123745083808899

Are you sure your file is really that large?

I also don't use MS Windows but the following toy programs work for me. The first creates a data file that mimics the structure of yours. The second shows that it can read the final data triplet. What happens if you run these on your system?

fh = open('x', 'wb')
fh.write(b'0123456789')
for i in range(0, 1000):
    s = bytes('{:03d}'.format(i), 'ascii')
    fh.write(b'a' + s + b'b' + s + b'c' + s)

Read the data from file x :

fh = open('x', 'rb')
triplet = 999
fh.seek(10 + triplet * 3 * 4)
data = fh.read(3 * 4)
print(data)