pandas dataframe 与包含行 'nan' 的字符串索引在保存到 hd5 和从 hd5 读取时被转换为 NaN

Question

I have a simple dataframe with a string index:

>>> df = pd.DataFrame(dict(x=['a','nan', 'NA', 'na', 'NaN'],
                           y=[1,2,3,4,5])).set_index('x')
>>> df
     y
x
a    1
nan  2
NA   3
na   4
NaN  5

It properly sets the index as strings.

>>> df.index.isna()
array([False, False, False, False, False])

However, when I save it to a hdf5 file, and read again, it somehow changes the index entry 'nan' to NaN.

>>> df.to_hdf('test.h5', key='test')
>>> df2=pd.read_hdf('test.h5')
>>> df2.index.isna()
array([False,  True, False, False, False])

Is there a way to avoid this conversion? In my actual code, the index is based on Drosophila gene nan and I don't want it to be converted.

>>> pd.show_versions()

INSTALLED VERSIONS
------------------
commit           : f2c8480af2f25efdbd803218b9d87980f416563e
python           : 3.8.8.final.0
python-bits      : 64
OS               : Linux
OS-release       : 3.10.0-1160.45.1.el7.x86_64
Version          : #1 SMP Wed Oct 13 17:20:51 UTC 2021
machine          : x86_64
processor        : x86_64
byteorder        : little
LC_ALL           : POSIX
LANG             : en_US.UTF-8
LOCALE           : None.None

pandas           : 1.2.3
numpy            : 1.20.2
pytz             : 2021.1
dateutil         : 2.8.1
pip              : 21.0.1
setuptools       : 52.0.0.post20210125
Cython           : 0.29.23
pytest           : 6.2.3
hypothesis       : None
sphinx           : 4.0.1
blosc            : None
feather          : None
xlsxwriter       : 1.3.8
lxml.etree       : 4.6.3
html5lib         : 1.1
pymysql          : None
psycopg2         : None
jinja2           : 2.11.3
IPython          : 7.22.0
pandas_datareader: None
bs4              : 4.9.3
bottleneck       : 1.3.2
fsspec           : 0.7.4
fastparquet      : None
gcsfs            : None
matplotlib       : 3.4.1
numexpr          : 2.7.3
odfpy            : None
openpyxl         : 3.0.7
pandas_gbq       : None
pyarrow          : None
pyxlsb           : None
s3fs             : 0.4.2
scipy            : 1.6.2
sqlalchemy       : 1.4.6
tables           : 3.6.1
tabulate         : None
xarray           : None
xlrd             : 2.0.1
xlwt             : 1.3.0
numba            : 0.53.1
>>>

Answer 1

I think that's an issue with read_hdf module.I could be wrong.

but one work around is to not set x as index when you save it as hd5 but after you read it back from hdf,set the index to x:

import pandas as pd

df = pd.DataFrame(dict(x=['a', 'nan', 'NA', 'na', 'NaN'],
                       y=[1, 2, 3, 4, 5]))

df.to_hdf('test.h5', key='test')

df2 = pd.read_hdf('test.h5')
df2 = df2.set_index('x')

test:

>>> df2.index.isna()
array([False, False, False, False, False])

or if you want to keep the index on your original dataframe, just reset the index when you want to save the hdf:

import pandas as pd

df = pd.DataFrame(dict(x=['a', 'nan', 'NA', 'na', 'NaN'],
                       y=[1, 2, 3, 4, 5])).set_index('x')

df.reset_index().to_hdf('test.h5', key='test')

df2 = pd.read_hdf('test.h5')
df2 = df2.set_index('x')