排序和解压缩大字节数组的最快方法是什么？

Question

I have a large binary file that needs to be converted into hdf5 file format.

I am using Python3.6. My idea is to read in the file, sort the relevant information, unpack it and store it away. My information is stored in a way that the 8 byte time is followed by 2 bytes of energy and then 2 bytes of extra information, then again time, ... My current way of doing it, is the following (my information is read as an bytearray, with the name byte_array):

for i in range(0, len(byte_array)+1, 12):

    if i == 0:
        timestamp_bytes = byte_array[i:i+8]
        energy_bytes = byte_array[i+8:i+10]
        extras_bytes = byte_array[i+10:i+12]
    else:
        timestamp_bytes += byte_array[i:i+8]
        energy_bytes += byte_array[i+8:i+10]
        extras_bytes += byte_array[i+10:i+12]


timestamp_array = np.ndarray((len(timestamp_bytes)//8,), '<Q',timestamp_bytes)
energy_array = np.ndarray((len(energy_bytes) // 2,), '<h', energy_bytes)
extras_array = np.ndarray((len(timestamp_bytes) // 8,), '<H', extras_bytes)

I assume there is a much faster way of doing this, maybe by avoiding to loop over the the whole thing. My files are up to 15GB in size so every bit of improvement would help a lot.

Answer 1

You should be able to just tell NumPy to interpret the data as a structured array and extract fields:

as_structured = numpy.ndarray(shape=(len(byte_array)//12,),
                              dtype='<Q, <h, <H',
                              buffer=byte_array)
timestamps = as_structured['f0']
energies = as_structured['f1']
extras = as_structured['f2']

This will produce three arrays backed by the input bytearray. Creating these arrays should be effectively instant, but I can't guarantee that working with them will be fast - I think NumPy may need to do some implicit copying to handle alignment issues with these arrays. It's possible (I don't know) that explicitly copying them yourself with .copy() first might speed things up.

Answer 2

You can use numpy.frombuffer with a custom datatype:

import struct
import random

import numpy as np


data = [
    (random.randint(0, 255**8), random.randint(0, 255*255), random.randint(0, 255*255))
    for _ in range(20)
    ]

Bytes = b''.join(struct.pack('<Q2H', *row) for row in data)
dtype = np.dtype([('time', np.uint64), 
                  ('energy', np.uint16), # you may need to change that to `np.int16`, if energy can be negative
                  ('extras', np.uint16)])

original = np.array(data, dtype=np.uint64)
result = np.frombuffer(Bytes, dtype)

print((result['time'] == original[:, 0]).all())
print((result['energy'] == original[:, 1]).all())
print((result['extras'] == original[:, 2]).all())

print(result)

Example output:

True
True
True
[(6048800706604665320, 52635, 291) (8427097887613035313, 15520, 4976)
 (3250665110135380002, 44078, 63748) (17867295175506485743, 53323, 293)
 (7840430102298790024, 38161, 27601) (15927595121394361471, 47152, 40296)
 (8882783920163363834, 3480, 46666) (15102082728995819558, 25348, 3492)
 (14964201209703818097, 60557, 4445) (11285466269736808083, 64496, 52086)
 (6776526382025956941, 63096, 57267) (5265981349217761773, 19503, 32500)
 (16839331389597634577, 49067, 46000) (16893396755393998689, 31922, 14228)
 (15428810261434211689, 32003, 61458) (5502680334984414629, 59013, 42330)
 (6325789410021178213, 25515, 49850) (6328332306678721373, 59019, 64106)
 (3222979511295721944, 26445, 37703) (4490370317582410310, 52413, 25364)]

Answer 3

I'm not an expert on numpy, but here's my 5 cents: You have lots of data, and probably it's more than your RAM. This points to the simplest solution - don't try to fit all data in your program. When you read a file into a variable - then the X GB is being read into RAM. If it's more than available RAM, then swapping is done by your OS. Swapping slows you down, since not only do you have disk operations for reading from source file, now you also have writing to disk to dump RAM contents into swap file. Instead of that write the script so that it uses parts of the input file as necessary (in your case you read the file along anyways and don't go back or jump far ahead).

Try opening the input file as memory mapped data structure (please note differences in usage between Unix and windows environments)

Then you can do simple read([n]) bytes at a time and append that to your arrays. behind the scenes data is read into RAM page by page as needed and will not exceed the available memory, also, leaving more space for your arrays to grow.

Also consider the fact that your resultant arrays can also outgrow RAM, which will cause similar slowdown as reading of a big file.