Python-将HDF5数据集读入列表与numpy数组

Question

I've been exploring HDF5 and its python interface (h5py) so I tried to read a HDF5 file (one dimensional array of 100 million integers) into: a normal list and another time to a numpy array. Converting the dataset to a numpy was very fast comparing to when I tried to convert it to a normal python list (actually doing it with a list took a very long time that I had to kill it before it finished).

Can any one help me understand what happens internally that makes converting HDF5 dataset to a numpy array extremely faster than doing it with a normal list? Does it has to do with h5py compatibility with numpy?

import numpy as np
import hdf5

def readtolist(dataset):
    return "normal list count = {0}".format(len(list(dataset['1'])))

def readtonp(dataset):
    n1 = np.array(dataset)
    return "numpy count = {0}".format(len(n1))

f = h5py.File(path, 'r')    
readtolist(f['1'])
readtonp(f['1'])

Thanks for the help!

Answer 1

Using a test file that I recently created:

In [78]: f = h5py.File('test.h5')
In [79]: list(f.keys())
Out[79]: ['x']
In [80]: f['x']
Out[80]: <HDF5 dataset "x": shape (2, 5), type "<i8">
In [81]: x = f['x'][:]
In [82]: x
Out[82]: 
array([[0, 2, 4, 6, 8],
       [1, 3, 5, 7, 9]])
In [83]: alist = x.tolist()
In [84]: alist
Out[84]: [[0, 2, 4, 6, 8], [1, 3, 5, 7, 9]]

Data storage in HDF5 is similar to numpy arrays. h5py uses compiled code ( cython ) to interface with HDF5 base code. It loads the datasets as numpy arrays.

To get a list, then, you have to convert the array to a list. For a 1d array, list(x) sort of works, but it is slow and incomplete. tolist() is the correct way.

list() iterates over the first dimension of the array:

In [85]: list(x)
Out[85]: [array([0, 2, 4, 6, 8]), array([1, 3, 5, 7, 9])]
In [86]: list(f['x'])
Out[86]: [array([0, 2, 4, 6, 8]), array([1, 3, 5, 7, 9])]

1211:~/mypy$ h5dump test.h5
HDF5 "test.h5" {
GROUP "/" {
   DATASET "x" {
      DATATYPE  H5T_STD_I64LE
      DATASPACE  SIMPLE { ( 2, 5 ) / ( 2, 5 ) }
      DATA {
      (0,0): 0, 2, 4, 6, 8,
      (1,0): 1, 3, 5, 7, 9
      }
   }
}
}

I should add that a Python list is a unique data structure. It contains pointers to objects elsewhere in memory, and thus can hold all kinds of objects - numbers, other lists, dictionaries, strings, custom classes, etc. A HDF5 dataset, like a numpy array, has to have a uniform data type (the DATATYPE in the dump). It can't, for example, store an object dtype array. If you want to save a list to a HDF5 , you first have to convert it an array.

Answer 2

HDF5 is a file format intended for storing large quantities of scientific array data. It can store multiple datasets and it offer multiple on-the-fly compression models, enabling data with repeated patterns to be stored more efficiently.

Usually parsing it using Pandas or Numpy would be much faster as they handle that compression in a vectorized form, while native Python list handles that via nesting, which is significantly slower.

This is basically it in a nutshell.

The following is an experiment using pandas and Numpy under the hood to generate a file of 100000 entries, store it in HDF5 and then parse it again.

Generating the Data

frame = pd.DataFrame({'a': np.random.randn(100000)})
store = pd.HDFStore('mydata.h5')
frame.to_hdf('mydata.h5', 'obj1', format='table')
store.close()

Timing the Parsing

%%timeit
df = pd.read_hdf('mydata.h5', 'obj1')

Output

9.14 ms ± 240 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

Very speedy compared to list .

Answer 3

I'm not sure the difference you are seeing is specific to hdf5. A similar effect can be had using just a string as a data source:

>>> import numpy as np
>>> import string, random
>>> 
>>> from timeit import timeit    
>>> kwds = dict(globals=globals(), number=1000)
>>> 
>>> a = ''.join(random.choice(string.ascii_letters) for _ in range(1000000))
>>> 
>>> timeit("A = np.fromstring(a, dtype='S1')", **kwds)
0.06803569197654724
>>> timeit("L = list(a)", **kwds)
6.131339570041746

Simplifying slightly, for homogeneous data, numpy stores them 'as they are', as a block of memory, whereas a list creates a python object for each item. A python object in this case consists of the value, a pointer to its type object and a ref count. So, in summary, numpy can essentially just copy a block of memory whereas list has to allocate and create all those objects and the list container.

The flipside of this is that accessing individual elments is faster from lists because - amongst other things - now the array has to create a python object whereas the list can simply return the one it has stored:

>>> L = list(a)
>>> A = np.fromstring(a, dtype='S1')
>>> 
>>> kwds = dict(globals=globals(), number=100)
>>> 
>>> timeit("[L[i] for i in range(len(L))]", **kwds)
5.607562301913276
>>> timeit("[A[i] for i in range(len(A))]", **kwds)
13.343806453049183