Pandas：选择带有unicode字符的字符串

Question

I am trying to select rows by specifying the value of one of the columns. That works perfectly well, as long as the value selected is pure ascii. If however, it contains non-ascii characters, I cannot get it to work no matter how I encode the value.

Simplified example to illustrate the problem:

>>> from __future__ import (absolute_import, division, 
                            print_function, unicode_literals)
>>> import pandas as pd
>>> df = pd.DataFrame([[1, 'Stuttgart'], [2, 'München']], columns=['id', 'city'])
>>> df['city'] = df['city'].map(lambda x: x.encode('latin-1'))
>>> store = pd.HDFStore('test_store.h5')
>>> store.append('test_key', df, data_columns=True)
>>> store['test_key']
   id       city
0   1  Stuttgart
1   2    M�nchen

Note that the non-asci string is indeed properly stored:

>>> store['test_key']['city'][1]
'M\xfcnchen'

Selecting for asci value works just fine:

>>> store.select('test_key', where='city==%r' % 'Stuttgart')
   id       city
0   1  Stuttgart

But selecting for the non-ascii value fails to return the row:

>>> store.select('test_key', where='city==%r' % 'München')
Empty DataFrame
Columns: [id, city]
Index: []

>>> store.select('test_key', where='city==%r' % 'München'.encode('latin-1'))
Empty DataFrame
Columns: [id, city]
Index: []

Clearly I am doing something wrong... How does one solve this issue?

Answer 1

Oddly, selection seems to work fine if the encoding is utf-8 instead of latin-1:

from __future__ import (absolute_import, division, 
                        print_function, unicode_literals)

import pandas as pd

df = pd.DataFrame([[1, 'Stuttgart'], [2, 'München']], columns=['id', 'city'])
df['city'] = df['city'].map(lambda x: x.encode('utf-8'))
store = pd.HDFStore('/tmp/test_store.h5', 'w')
store.append('test_key', df, data_columns=True)
print(store.select('test_key', where='city==%r' % 'Stuttgart'.encode('utf-8')))
#    id       city
# 0   1  Stuttgart

print(store.select('test_key', where='city==%r' % 'München'.encode('utf-8')))
#    id     city
# 1   2  München

store.close()

Answer 2

It looks like PyTables 3.1.1 may not support unicode columns. I'm not a PyTables user but this bug report suggests it is a known problem postponed until version 3.2. This other issue is maybe relevant.