Create HDF5 file using pytables with table format and data columns

Question

I want to read a h5 file previously created with PyTables .

The file is read using Pandas , and with some conditions, like this:

pd.read_hdf('myH5file.h5', 'anyTable', where='some_conditions')

From another question, I have been told that, in order for a h5 file to be "queryable" with read_hdf's where argument it must be writen in table format and, in addition, some columns must be declared as data columns .

I cannot find anything about it in PyTables documentation.

The documentation on PyTable's create_table method does not indicate anything about it.

So, right now, if I try to use something like that on my h5 file createed with PyTables I get the following:

>>> d = pd.read_hdf('test_file.h5','basic_data', where='operation==1')
C:\Python27\lib\site-packages\pandas\io\pytables.py:3070: IncompatibilityWarning: 
where criteria is being ignored as this version [0.0.0] is too old (or
not-defined), read the file in and write it out to a new file to upgrade (with
the copy_to method)

  warnings.warn(ws, IncompatibilityWarning)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Python27\lib\site-packages\pandas\io\pytables.py", line 323, in read_hdf
    return f(store, True)
  File "C:\Python27\lib\site-packages\pandas\io\pytables.py", line 305, in <lambda>
    key, auto_close=auto_close, **kwargs)
  File "C:\Python27\lib\site-packages\pandas\io\pytables.py", line 665, in select
    return it.get_result()
  File "C:\Python27\lib\site-packages\pandas\io\pytables.py", line 1359, in get_result
    results = self.func(self.start, self.stop, where)
  File "C:\Python27\lib\site-packages\pandas\io\pytables.py", line 658, in func
    columns=columns, **kwargs)
  File "C:\Python27\lib\site-packages\pandas\io\pytables.py", line 3968, in read
    if not self.read_axes(where=where, **kwargs):
  File "C:\Python27\lib\site-packages\pandas\io\pytables.py", line 3196, in read_axes
    values = self.selection.select()
  File "C:\Python27\lib\site-packages\pandas\io\pytables.py", line 4482, in select
    start=self.start, stop=self.stop)
  File "C:\Python27\lib\site-packages\tables\table.py", line 1567, in read_where
    self._where(condition, condvars, start, stop, step)]
  File "C:\Python27\lib\site-packages\tables\table.py", line 1528, in _where
    compiled = self._compile_condition(condition, condvars)
  File "C:\Python27\lib\site-packages\tables\table.py", line 1366, in _compile_condition
    compiled = compile_condition(condition, typemap, indexedcols)
  File "C:\Python27\lib\site-packages\tables\conditions.py", line 430, in compile_condition
    raise _unsupported_operation_error(nie)
NotImplementedError: unsupported operand types for *eq*: int, bytes

EDIT:

The traceback mentions something about IncompatibilityWarning and version [0.0.0], however if I check my versions of Pandas and Tables I get:

>>> import pandas
>>> pandas.__version__
'0.15.2'
>>> import tables
>>> tables.__version__
'3.1.1'

So, I am totally confused.

Answer 1

I had the same issue, and this is what I have done.

1. Create a HDF5 file by PyTables;

2. Read this HDF5 file by pandas.read_hdf and use parameters like "where = where_string, columns = selected_columns"

3. I got the warning message like below and other error messages:

   D:\\Program Files\\Anaconda3\\lib\\site-packages\\pandas\\io\\pytables.py:3065: IncompatibilityWarning: where criteria is being ignored as this version [0.0.0] is too old (or not-defined), read the file in and write it out to a new file to upgrade (with the copy_to method)

   warnings.warn(ws, IncompatibilityWarning)

4. I tried commands like this:

   hdf5_store = pd.HDFStore(hdf5_file, mode = 'r')

   h5cpt_store_new = hdf5_store.copy(hdf5_new_file, complevel=9, complib='blosc') h5cpt_store_new.close()

5. And run the command exactly like step 2, it works.

   pandas. version '0.17.1'

   tables. version '3.2.2'