问题？ 从熊猫0.17.1中的DataFrame中选择数据ween_time

Question

I am having an issue selecting data from a pandas DataFrame with between_time. When the start and end dates of the query are between two days the result is empty. I am using pandas 0.17.1 (python 2.7)

I have the following data frame:

mydf = pd.DataFrame.from_dict({'azi': {Timestamp('2015-05-12 00:00:14.348000'): 109.801,
Timestamp('2015-05-12 00:00:36.125000'): 109.994,
Timestamp('2015-05-12 00:00:57.599000'): 109.60299999999999,
Timestamp('2015-05-12 00:01:14.576000'): 100.2},
'ele': {Timestamp('2015-05-12 00:00:14.348000'): 180.001,
Timestamp('2015-05-12 00:00:36.125000'): 179.999,
Timestamp('2015-05-12 00:00:57.599000'): 179.999,
Timestamp('2015-05-12 00:01:14.576000'): 180.001}})

Which results in:

                            azi     ele
2015-05-12 00:00:14.348     109.801     180.001
2015-05-12 00:00:36.125     109.994     179.999
2015-05-12 00:00:57.599     109.603     179.999
2015-05-12 00:01:14.576     100.200     180.001

The following query fails :

mydf['azi'].between_time(datetime(2015, 5, 11, 23, 59, 59, 850000), datetime(2015, 5, 12, 0, 1, 59, 850000))

resulting in:

Series([], Name: azi, dtype: float64)

However the following query works :

mydf2['azi'].between_time(datetime(2015, 5, 11, 0, 0, 0, 0), datetime(2015, 5, 12, 0, 1, 59, 850000))

with the right answer:

 2015-05-12 00:00:14.348    109.801
 2015-05-12 00:00:36.125    109.994
 2015-05-12 00:00:57.599    109.603
 2015-05-12 00:01:14.576    100.200
 Name: azi, dtype: float64

Questions :

1. I am missing something in the functionality of the function, or is this a real bug?
2. Is there a workaround for this? The background is that I really need to process data in 1 minute chunks which limits are not always coinciding with 00:00:00

Answer 1

You could find a lot of information how to work with datetime index from docs . For you case you could try loc :

In [147]: mydf['azi'].loc[datetime(2015, 5, 11, 23, 59, 59, 850000): datetime(2015, 5, 12, 0, 1, 59, 850000)]
Out[147]: 
2015-05-12 00:00:14.348    109.801
2015-05-12 00:00:36.125    109.994
2015-05-12 00:00:57.599    109.603
2015-05-12 00:01:14.576    100.200
Name: azi, dtype: float64

It was about your 2) bullet. About 1) you could see explanation from @Jeff

Answer 2

The doc-string says it all.

between_time selects all TIMES.

In [67]: mydf.between_time?
Signature: mydf.between_time(start_time, end_time, include_start=True, include_end=True)
Docstring:
Select values between particular times of the day (e.g., 9:00-9:30 AM)

Parameters
----------
start_time : datetime.time or string
end_time : datetime.time or string
include_start : boolean, default True
include_end : boolean, default True

Returns
-------
values_between_time : type of caller
File:      ~/pandas/pandas/core/generic.py
Type:      instancemethod

In [68]: mydf
Out[68]: 
                             azi      ele
2015-05-12 00:00:14.348  109.801  180.001
2015-05-12 00:00:36.125  109.994  179.999
2015-05-12 00:00:57.599  109.603  179.999
2015-05-12 00:01:14.576  100.200  180.001

In [70]: mydf.between_time('00:00:30','00:01:00')
Out[70]: 
                             azi      ele
2015-05-12 00:00:36.125  109.994  179.999
2015-05-12 00:00:57.599  109.603  179.999

You can separately use partial-string indexing, see here to select based on dates (these can be strings or datetimelikes).

In [73]: mydf.loc['20150512 00:00:30':'20150512 00:01:00']
Out[73]: 
                             azi      ele
2015-05-12 00:00:36.125  109.994  179.999
2015-05-12 00:00:57.599  109.603  179.999

I think .between_time should actually raise on non .time / string convertible objects, but IIRC this was done for ease of implementation.