通过日期时间索引过滤pandas DataFrame时结果不同

Question

I'm trying to filter a pandas DataFrame and I'm getting different results using a test case and the real data. Using real data I'm getting NaN values, while on the test case I'm getting what I expect.

Test case:

The test case I created has following code:

import pandas as pd
df1 = pd.DataFrame([
["2014-08-06 12:10:00", 19.85,  299.96, 17.5,   228.5,  19.63,  571.43],
["2014-08-06 12:20:00", 19.85,  311.55, 17.85,  248.68, 19.78,  547.21],
["2014-08-06 12:30:00", 20.06,  355.27, 18.35,  224.82, 19.99,  410.68],
["2014-08-06 12:40:00", 20.14,  405.95, 18.49,  247.33, 20.5,   552.79],
["2014-08-06 12:50:00", 20.14,  352.87, 18.7,   449.33, 20.86,  616.44],
["2014-08-06 13:00:00", 20.28,  356.96, 18.92,  307.57, 21.15,  471.18]],
columns=["date_time","t1", "1", "t4", "4", "t6", "6"])
df1 = df1.set_index(["date_time"])
df1 = pd.to_datetime(df1)

filter1 = pd.DataFrame(["2014-08-06 12:20:00","2014-08-06 13:00:00"])
df1_filtered = df1.ix[filter1[filter1.columns[0]][0:2]]

As you may expect, the result is:

>>> df1_filtered
                        t1       1     t4       4     t6       6
2014-08-06 12:20:00  19.85  311.55  17.85  248.68  19.78  547.21
2014-08-06 13:00:00  20.28  356.96  18.92  307.57  21.15  471.18

Using real data:

Real data comes from a txt file and looks like this:

Fecha_hora  t1  1   t4  4   t6  6
2014-08-06 12:10:00 19.85   299.96  17.5    228.5   19.63   571.43
2014-08-06 12:20:00 19.85   311.55  17.85   248.68  19.78   547.21
2014-08-06 12:30:00 20.06   355.27  18.35   224.82  19.99   410.68
2014-08-06 12:40:00 20.14   405.95  18.49   247.33  20.5    552.79
2014-08-06 12:50:00 20.14   352.87  18.7    449.33  20.86   616.44
2014-08-06 13:00:00 20.28   356.96  18.92   307.57  21.15   471.18

However when I read the real data, and use same filter as before this way:

df2 = pd.read_csv(r"D:/tmp/data.txt", sep='\t', parse_dates=True, index_col=0)
df2_filtered = df2.ix[filter1[filter1.columns[0]][0:2]]

I get following results with values as NaN :

>>> df2_filtered
                     t1   1  t4   4  t6   6
2014-08-06 12:20:00 NaN NaN NaN NaN NaN NaN
2014-08-06 13:00:00 NaN NaN NaN NaN NaN NaN

But I can still get the values from a certain row like this:

>>> df2.ix["2014-08-06 12:20:00"]
t1     19.85
1     311.55
t4     17.85
4     248.68
t6     19.78
6     547.21
Name: 2014-08-06 12:20:00

Question:

How can I filter my real data in order to get same results as in my test case? May there be a better way to achieve what I'm looking for?

Note : My pandas version is 0.9.0 used under python 2.5 . Means I have no loc function.

Note 2 : I even tried this using python 2.7 under pythonanywhere.com with same different results. However if I check for df1==df2 I get True for every single value.

Answer 1

Hopefully goes without saying, but if at all possible, upgrade your python/pandas!

In this case, on a recent version ( 0.20.3 ) I get missing values in both cases - I need to convert the lookup keys to datetimes and I'm guessing it will work for you too.

The convenience string based date indexing only works with scalars / slices.

In [174]: lookup = pd.to_datetime(filter1[filter1.columns[0]][0:2])

In [175]: df2.ix[lookup]
Out[175]: 
                        t1       1     t4       4     t6       6
Fecha_hora                                                      
2014-08-06 12:20:00  19.85  311.55  17.85  248.68  19.78  547.21
2014-08-06 13:00:00  20.28  356.96  18.92  307.57  21.15  471.18