PyTables, Creating a table without opening an hdf5 file

Question

is it possible to create a PyTables table without opening or creating an hdf5 file? What I mean, and what I need, is to create a table (well actually very many tables) in different processes, work with these tables and store the tables into an hdf5 file only in the end after some calculations (and ensuring only one process at a time performs the storage).

In principle I could do all the calculations on normal Python data (arrays strings etc) and perform the storage in the end. However, why I would appreciate to work on PyTables right from the start are the sanity checks. I want to always ensure that the data I work with fits into the predefined tables and does not violate shape constraints etc (and since PyTables checks for those problems I don't need to implement it all by myself).

Thanks a lot and kind regards, Robert

Answer 1

You are looking for pandas which has great Pytables integration . You will be working with tables all the way and in the end you will be able to save to hdf5 in the easiest possible way.

Answer 2

You can create a numpy-array with given shape and datatype.

my_array = num.empty(shape=my_shape, dtype=num.float)

If you need indexing by name have a look at numpy record-arrays (nee numpy recarray )

But if you work directly with the PyTable-Object it can be faster (see benchmark here ).