numpy.save 以存储 3D Numpy 阵列与 ZD304BA20E96D87414Z588EEABAC850E3

Question

I would like to write Numpy arrays with shape (3, 225, 400) into a binary file.

These arrays are basically generated by using a screen buffer, and each screen has a label. My goal is to save each screen with its label.

numpy.save receives only two arguments: file pointer and array to be saved. The only option seems to be appending labels to arrays as follows:

with open(file, 'wb') as f:
   np.save(f, np.append(buffer, [label]) )

However, I would not prefer this. Another approach might be saving only the array and then writing " \t label " like regular binary writing:

with open(file, 'wb') as f:
   np.save(f, buffer)
   f.write("\t" + label)

I am not sure whether np.save moves the file pointer to new line after saving.

Considering the fact that I will save hundreds of thousands of array-label pairs in a high frequency, what would you suggest in terms of efficiency?

Answer 1

One option is to save to a numpy (NPZ) file. I have included an example below. np.savez and np.savez_compressed allow one to save multiple arrays to one file.

import numpy as np

# Create fake data.
rng = np.random.RandomState(0)
buffer = rng.normal(size=(3, 225, 400))
label = "this is the label"

# Save. Can use np.savez here instead.
np.savez_compressed("output.npz", buffer=buffer, label=label)

# Load.
npzfile = np.load("output.npz")

np.testing.assert_equal(npzfile["buffer"], buffer)
np.testing.assert_equal(npzfile["label"], label)

---

Another option is to use HDF5 using h5py . The organization of an HDF5 file is similar to a filesystem (root is / and datasets can be created with names like /data/buffers/dataset1 ). One way of organizing the buffers and labels is to create a dataset for each buffer and set a label attribute on it.

import h5py
import numpy as np

# Create fake data.
rng = np.random.RandomState(0)
buffer = rng.normal(size=(3, 225, 400))
label = "this is the label"

this_dataset = "/buffers/0"

# Save to HDF5.
with h5py.File("output.h5", "w") as f:
    f.create_dataset(this_dataset, data=buffer, compression="lzf")
    f[this_dataset].attrs.create("label", label)

# Load.
with h5py.File("output.h5", "r") as f:
    loaded_buffer = f[this_dataset]
    loaded_label = f[this_dataset].attrs["label"]