有没有办法在现有二进制文件周围包装 numpy `ndarray` 接口？

Question

I have a binary.network capture ( .pcapng ) file that contains video data. I am parsing the .pcapng with scapy and I can extract the data, but the video sequences I am working with are very large and the operations I want to perform quickly grind my machine to a halt if I load very much data at once. One approach to deal with this would be to extract all the data and save it into a mmap file, or better yet, HDF5. However, before I sign up for making copies of all the data, I wanted to see if it is possible to memory map the existing files in place. Is there a way to make a discontinuous mmap into an existing file that tells an ndarray object where to find memory associated with a given index, when that memory may be in arbitrary locations within the file? I haven't found any good analogs in mmap, which assumes a contiguous file is available. I imagine some ndarray subclass that loads up the file, scans the file for the boundaries of all the relevant imagery data within the .pcapng file, and provides a custom implementation of ndarray __index__ method that can return the appropriate file offset(s) for a given index or slice. Is this bonkers, or is there a better (already solved) method for doing this?

Answer 1

Construct a mmap.mmap object from the file and pass that to the numpy.ndarray constructor. For example:

import os
import numpy

with open("filename", "rb") as f:
    mm = mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ)

nd = numpy.ndarray(1000000,buffer=mm)