View data structures of .pkl file

Question

I try to view the data structures of ExtremeNet_250000.pkl( size about 780M) by the code:

import pickle
pth=open(r'E:/ExtremeNet_250000.pkl','rb')
pkl=pickle.load(pth)
print(pkl)

but it return the int just like

RESTART: C:/Users/cwc888888/AppData/Local/Programs/Python/Python37/111.py =
119547037146038801333356

could you give me some suggestion?

Answer 1

For your specific case:

Based on your file name, I'm guessing you're playing around with the ExtremeNet demo project . Tracing through the code and dependencies, it looks like that cache file is the result of a torch.save call from the PyTorch package. Simplest solution for this case is probably just to install PyTorch , and use torch.load to load it .

---

General solution:

It's possible to save multiple pickled objects in a file in an unstructured manner. That is, rather than saving a list as a list with a single:

 with open(filename, 'wb') as f:
     pickle.dump(mylist, f)

the creator could dump the individual values one by one with:

 with open(filename, 'wb') as f:
     for x in mylist:
         pickle.dump(x, f)
     # Or equivalently and usually slightly faster creating a single reusable Pickler:
     pickler = pickle.Pickler(f)
     for x in mylist:
         pickler.dump(x)

removing the list structure from the output.

If you've got such a file, you have to read the values out in a similar way. The simplest approach is to make a generator function that will yield successive values when iterated, eg:

def iterunpickle(f):
    unpickler = pickle.Unpickler(f)
    try:
        while True:
            yield unpickler.load()
    except EOFError:
        pass

allowing you to iterate the contents of your file with:

with open(r'E:/ExtremeNet_250000.pkl','rb') as pth:
    for item in iterunpickle(pth):
        print(item)

or load the whole thing into a list (if sufficient memory available) with:

with open(r'E:/ExtremeNet_250000.pkl','rb') as pth:
    items = list(iterunpickle(pth))

For a file that large, I'd recommend iterating live to preserve memory rather than converting to a list up front; if it's nothing but largish integers in the range of that first value, the memory cost to load the entire thing into a list would be roughly three times the disk size, depending on pickle protocol, whether your interpreter is 32 or 64 bit, etc. Regardless, you'd been looking a multiple GB of RAM for the data alone, before you did anything with it.