Python如何读取巨大的二进制文件（> 25GB）？

Question

I have N-body simulation data and have to read that file in python.

Its size is over 25GB so file.read() is not work by lack of memory.

So I wrote the code like this

with open("fullFoF_merger.cbin.z0.Run1", "rb") as mergertree:
    def param(data):
        result = {"nowhid":data[0], "nexthid":data[2],"zi":data[10], 
                  "zip1":data[11], "F":data[4], "mass":data[9], 
                  "dlnM":data[5],"dM":data[12], "dlnJ":data[6],"dJ":data[13],
                  "dlnspin": data[7], "spin":data[8],
                  "G":data[14], "overden":data[15]}
        return result

    num = 0

    while 1:
        num +=1

        binary_data = mergertree.read(4)

        if not binary_data : break

        n_max = struct.unpack('I', binary_data)


        binary_data = mergertree.read(64*n_max[0])

        Halo = [None]*n_max[0]


        for i in range(1,n_max[0]+1):
            data = struct.unpack("4i12f", binary_data[64*(i-1):64*(i)])
            Halo[i-1] = param(data)

        MergerQ = []+Halo


print(MergerQ)

print(num)

print("\n Run time \n --- %d seconds ---" %(time.time()-start_time))

In this process while loop calculate 45470522 times in this code. But when I print MergerQ in python it shows only one dictionary data like this

[{'nowhid': 53724, 'nexthid': 21912952, 'zi': 0.019874930381774902, 'zip1': -1.6510486602783203e-05, 'F': inf, 'mass': 67336740864.0, 'dlnM': 0.0, 'dM': 0.0, 'dlnJ': 0.1983184665441513, 'dJ': 8463334768640.0, 'dlnspin': 0.19668935239315033, 'spin': 0.012752866372466087, 'G': inf, 'overden': 1.0068886280059814}]

I think it caused by lack of memory or memory limit of python's variables.

How can I solve this problem?

Is there any way to read whole data and save in python variables?

Parallel computing can be the solution of this code?

I will waiting for your comment. Thank you.

Answer 1

This line is your problem:

MergerQ = []+Halo

You clear MergerQ , put it outside of your loop instead:

num = 0
MergerQ = []

while 1:
    ...
    MergerQ += Halo

But don't expect to have the amount of memory you need to store the entire thing if your file is that big, you'll need a lot of memory and a lot of time.

Edit

Its very possible that you'll be able to successfully run your code without as much physical RAM would be needed as your OS will likely store it in your hard disk fetching it when needed, but this will massively increase run time.

Try running this code snippet and seeing what happens ( forewarning: if you leave this running too long your machine will become unresponsive and most likely need physically reset )

a = []
while 1:
    a = [a, a]

Expect your script to react similarly.