Python Pandas-将数据框中的2行结合起来-有条件

Question

I have a Pandas data frame that looks like that:

A       B     C    Stime    Etime    
1220627 a   10.0 18:00:00 18:09:59
1220627 a   12.0 18:15:00 18:26:59
1220683 b   3.0  18:36:00 18:38:59
1220683 a   3.0  18:36:00 18:38:59
1220732 a   59.0 18:00:00 18:58:59
1220760 A   16.0 18:24:00 18:39:59
1220760 a   16.0 18:24:00 18:39:59
1220760 A   19.0 18:40:00 18:58:59
1220760 b   19.0 18:40:00 18:58:59
1220760 a   19.0 18:40:00 18:58:59
1220775 a   3.0  18:03:00 18:05:59

Stime and Etime cols are from type datetime.

C is the number of minutes between Stime and Etime.

A col is household ID and B col is person ID in the household.

(so that cols A and B together represent a unique person).

What I need to do is to update the table such that if, for a certain person, the Stime comes right after the end time - I will unit the 2 lines and I will update C.

for example here, for person a in HH 1220760 the first Etime is 18:39:59

and the second Stime is 18:40:00 - which comes right after 18:39:59, so I would like to unit the lines and update C for this person to be 35 (16+19).

I tried to use groupby but I don't know how to add the condition that Stime will come right after Etime .

Answer 1

If we add one second to Etime then we can find rows to be joined by grouping by ['A', 'B'] and then for each group comparing shifted Etime s with the next Stime :

df['Etime'] += pd.Timedelta(seconds=1)
df = df.sort_values(by=['A', 'B', 'Stime'])
df['keep'] = df.groupby(['A','B'])['Etime'].shift(1) != df['Stime']
#           A  B     C               Etime               Stime   keep
# 0   1220627  a  10.0 2016-05-29 18:10:00 2016-05-29 18:00:00   True
# 1   1220627  a  12.0 2016-05-29 18:27:00 2016-05-29 18:15:00   True
# 3   1220683  a   3.0 2016-05-29 18:39:00 2016-05-29 18:36:00   True
# 2   1220683  b   3.0 2016-05-29 18:39:00 2016-05-29 18:36:00   True
# 4   1220732  a  59.0 2016-05-29 18:59:00 2016-05-29 18:00:00   True
# 5   1220760  A  16.0 2016-05-29 18:40:00 2016-05-29 18:24:00   True
# 7   1220760  A  19.0 2016-05-29 18:59:00 2016-05-29 18:40:00  False
# 12  1220760  a   0.0 2016-05-29 18:10:00 2016-05-29 18:00:00   True
# 6   1220760  a  16.0 2016-05-29 18:40:00 2016-05-29 18:24:00   True
# 9   1220760  a  19.0 2016-05-29 18:59:00 2016-05-29 18:40:00  False
# 11  1220760  a  11.0 2016-05-29 19:10:00 2016-05-29 18:59:00  False
# 8   1220760  b  19.0 2016-05-29 18:59:00 2016-05-29 18:40:00   True
# 10  1220775  a   3.0 2016-05-29 18:06:00 2016-05-29 18:03:00   True

We will want to keep rows where keep is True and remove rows where keep is False, except that we will also want to update the Etime s as appropriate.

It would be nice if we could assign a "group number" to each row so that we could group by ['A', 'B', 'group_number'] -- and in fact we can. All we need to do is apply cumsum to the keep column:

df['group_number'] = df.groupby(['A','B'])['keep'].cumsum()
#           A  B     C               Etime               Stime   keep  group_number
# 0   1220627  a  10.0 2016-05-29 18:10:00 2016-05-29 18:00:00   True           1.0
# 1   1220627  a  12.0 2016-05-29 18:27:00 2016-05-29 18:15:00   True           2.0
# 3   1220683  a   3.0 2016-05-29 18:39:00 2016-05-29 18:36:00   True           1.0
# 2   1220683  b   3.0 2016-05-29 18:39:00 2016-05-29 18:36:00   True           1.0
# 4   1220732  a  59.0 2016-05-29 18:59:00 2016-05-29 18:00:00   True           1.0
# 5   1220760  A  16.0 2016-05-29 18:40:00 2016-05-29 18:24:00   True           1.0
# 7   1220760  A  19.0 2016-05-29 18:59:00 2016-05-29 18:40:00  False           1.0
# 12  1220760  a   0.0 2016-05-29 18:10:00 2016-05-29 18:00:00   True           1.0
# 6   1220760  a  16.0 2016-05-29 18:40:00 2016-05-29 18:24:00   True           2.0
# 9   1220760  a  19.0 2016-05-29 18:59:00 2016-05-29 18:40:00  False           2.0
# 11  1220760  a  11.0 2016-05-29 19:10:00 2016-05-29 18:59:00  False           2.0
# 8   1220760  b  19.0 2016-05-29 18:59:00 2016-05-29 18:40:00   True           1.0
# 10  1220775  a   3.0 2016-05-29 18:06:00 2016-05-29 18:03:00   True           1.0

Now the desired result can be found by grouping by ['A', 'B', 'group_number'] , and finding the minimum Stime and maximum Etime for each group:

result = df.groupby(['A','B', 'group_number']).agg({'Stime':'min', 'Etime':'max'})

                                     Stime               Etime
A       B group_number                                        
1220627 a 1.0          2016-05-29 18:00:00 2016-05-29 18:10:00
          2.0          2016-05-29 18:15:00 2016-05-29 18:27:00
1220683 a 1.0          2016-05-29 18:36:00 2016-05-29 18:39:00
        b 1.0          2016-05-29 18:36:00 2016-05-29 18:39:00
1220732 a 1.0          2016-05-29 18:00:00 2016-05-29 18:59:00
1220760 A 1.0          2016-05-29 18:24:00 2016-05-29 18:59:00
        a 1.0          2016-05-29 18:00:00 2016-05-29 18:10:00
          2.0          2016-05-29 18:24:00 2016-05-29 19:10:00
        b 1.0          2016-05-29 18:40:00 2016-05-29 18:59:00
1220775 a 1.0          2016-05-29 18:03:00 2016-05-29 18:06:00

---

Putting it all together,

import numpy as np
import pandas as pd

df = pd.DataFrame(
    {'A': [1220627, 1220627, 1220683, 1220683, 1220732, 1220760, 1220760,
           1220760, 1220760, 1220760, 1220775, 1220760, 1220760],
     'B': ['a', 'a', 'b', 'a', 'a', 'A', 'a', 'A', 'b', 'a', 'a', 'a', 'a'], 
     'C': [10.0, 12.0, 3.0, 3.0, 59.0, 16.0, 16.0, 19.0, 19.0, 19.0, 3.0, 11.0, 0], 
     'Stime': ['18:00:00', '18:15:00', '18:36:00', '18:36:00', '18:00:00',
               '18:24:00', '18:24:00', '18:40:00', '18:40:00', '18:40:00', 
               '18:03:00', '18:59:00', '18:00:00'],
     'Etime': ['18:09:59', '18:26:59', '18:38:59', '18:38:59', '18:58:59',
               '18:39:59', '18:39:59', '18:58:59', '18:58:59', '18:58:59', 
               '18:05:59', '19:09:59', '18:09:59'],})
for col in ['Stime', 'Etime']:
    df[col] = pd.to_datetime(df[col])
df['Etime'] += pd.Timedelta(seconds=1)
df = df.sort_values(by=['A', 'B', 'Stime'])
df['keep'] = df.groupby(['A','B'])['Etime'].shift(1) != df['Stime']
df['group_number'] = df.groupby(['A','B'])['keep'].cumsum()
result = df.groupby(['A','B', 'group_number']).agg({'Stime':'min', 'Etime':'max'})
result = result.reset_index()
result['C'] = (result['Etime']-result['Stime']).dt.total_seconds() / 60.0
result = result[['A', 'B', 'C', 'Stime', 'Etime']]
print(result)

yields

         A  B     C               Stime               Etime
0  1220627  a  10.0 2016-05-29 18:00:00 2016-05-29 18:10:00
1  1220627  a  12.0 2016-05-29 18:15:00 2016-05-29 18:27:00
2  1220683  a   3.0 2016-05-29 18:36:00 2016-05-29 18:39:00
3  1220683  b   3.0 2016-05-29 18:36:00 2016-05-29 18:39:00
4  1220732  a  59.0 2016-05-29 18:00:00 2016-05-29 18:59:00
5  1220760  A  35.0 2016-05-29 18:24:00 2016-05-29 18:59:00
6  1220760  a  10.0 2016-05-29 18:00:00 2016-05-29 18:10:00
7  1220760  a  46.0 2016-05-29 18:24:00 2016-05-29 19:10:00
8  1220760  b  19.0 2016-05-29 18:40:00 2016-05-29 18:59:00
9  1220775  a   3.0 2016-05-29 18:03:00 2016-05-29 18:06:00

---

One of the advantages of using half-open intervals of the form [start, end) instead of fully-closed intervals [start, end] is that when two interval abut, the end of one equals the start of the next.

Another advantage is that the number of minutes in a half-open interval equals end-start . With a fully-closed interval, the formula becomes end-start+1 .

Python's builtin range and list slicing syntax use half-open intervals for these same reasons . So I would recommend using half-open intervals [Stime, Etime) in your DataFrame too.

Answer 2

what about this approach?

In [68]: df.groupby(['A','B', df.Stime - df['Etime'].shift() <= pd.Timedelta('1S')], as_index=False)['C'].sum()
Out[68]:
         A  B     C
0  1220627  a  22.0
1  1220683  a   3.0
2  1220683  b   3.0
3  1220732  a  59.0
4  1220760  A  35.0
5  1220760  a  35.0
6  1220760  b  19.0
7  1220775  a   3.0

Answer 3

Ok I think have a solution, but it is very crude and I'm sure someone can improve upon it.

assuming df = the data you have provided above:

df['Stime'] = pd.to_datetime(df['Stime'], format='%H:%M:%S') # needs to be converted to datetime
df['Etime'] = pd.to_datetime(df['Etime'], format='%H:%M:%S') # needs to be converted to datetime

df = df.sort_values(['A','B','Stime']) # data needs to be sorted by unique person : Stime
df = df.reset_index(drop=True)
df = df.reset_index() 

def new_person(row):
    if row.name > 0:
        if row['A'] != df.ix[row.name-1][1] or row['B'] != df.ix[row.name-1][2]:
            return 'Yes'

def update(row):
    if row.name > 0:
        if row['B'] == df.ix[row.name-1][2]:
            if df.ix[row.name][4] - df.ix[row.name-1][5] >= pd.Timedelta(seconds=0) and df.ix[row.name][4] - df.ix[row.name-1][5] < pd.Timedelta(seconds=2):
                return df.groupby(['A','B'])['C'].cumsum().ix[row.name]

def rewrite(row):
    if row['update'] > 0:
        return row['update']
    else:
        return row['C']

df['new_person'] = df.apply(new_person, axis=1) # adds column where value = 'Yes' if person is not the same as row above
df['update'] = df.apply(update,axis=1) # adds a column 'update' to allow for a cumulative sum rewritten to 'C' in rewrite function
print df

df['Stime'] = pd.to_datetime(df['Stime'], format='%H:%M:%S').dt.time # removes date from datetime
df['Etime'] = pd.to_datetime(df['Etime'], format='%H:%M:%S').dt.time # removes date from datetime
df['C'] = df.apply(rewrite,axis=1) # rewrites values for 'C' column

# hacky way of combining idxmax and indices of rows where the person is 'new'
updated = df.groupby(['A','B'])['C'].agg(pd.Series.idxmax).values
not_updated = df['new_person'].isnull().tolist()

combined = [x for x in df.index if (x in updated or x in not_updated)]

df = df.iloc[combined]
df = df.drop(['new_person','update','index'],axis=1)
print df

Apologies for the extremely hacky answer, but I think it should achieve what you need. Not sure how well it will work if your dataframe is very large though.

Resulting dataframe:

          A  B   C     Stime     Etime
0   1220627  a  10  18:00:00  18:09:59
1   1220627  a  12  18:15:00  18:26:59
2   1220683  a   3  18:36:00  18:38:59
3   1220683  b   3  18:36:00  18:38:59
4   1220732  a  59  18:00:00  18:58:59
6   1220760  A  35  18:40:00  18:58:59
9   1220760  a  46  18:59:00  18:09:59
10  1220760  b  19  18:40:00  18:58:59
11  1220775  a   3  18:03:00  18:05:59