根据数据框的条件在python中创建记录

Question

I have a dataframe A like this.

    Timestamp   A   B   C   D
1/1/2018 0:00   10  10  10  10
1/1/2018 0:10           10  25
1/1/2018 0:20           10  25
1/1/2018 0:30               25
1/1/2018 0:40               25
1/1/2018 0:50   25  25  25  25
1/1/2018 1:00   30  30  30  30
1/1/2018 1:10   42  42  42  42
1/1/2018 1:20               
1/1/2018 1:30               
1/1/2018 1:40   40      40  40
1/1/2018 1:50           35  35
1/1/2018 2:00               37
1/1/2018 2:10               49
1/1/2018 2:20   51  51      51

I want to delete some rows based on the following dataframe as follows.

  StartTime       EndTime       Comment
1/1/2018 1:20  1/1/2018 1:30   to be removed
1/1/2018 2:00  1/1/2018 2:20   to be removed

To get the dataframe A without the above timestamps as

      Timestamp     A   B   C   D
    1/1/2018 0:00   10  10  10  10
    1/1/2018 0:10           10  25
    1/1/2018 0:20           10  25
    1/1/2018 0:30               25
    1/1/2018 0:40               25
    1/1/2018 0:50   25  25  25  25
    1/1/2018 1:00   30  30  30  30
    1/1/2018 1:10   42  42  42  42
    1/1/2018 1:40   40      40  40
    1/1/2018 1:50           35  35

And I want results like as follows:

StartTime       EndTime       Column    Comment
1/1/2018 0:10   1/1/2018 0:40   A        NULL
1/1/2018 1:50   1/1/2018 1:50   A        NULL
1/1/2018 0:10   1/1/2018 0:40   B        NULL
1/1/2018 1:40   1/1/2018 1:50   B        NULL
1/1/2018 0:30   1/1/2018 0:40   C        NULL

Thanks for the help in advance.

Answer 1

This is harder than what I thought , but following will do the job

v=np.concatenate([pd.date_range(x['StartTime'],x['EndTime'],freq='10Min') for _,x in remove.iterrows()])
df=df[~df.Timestamp.isin(v)]
df
Out[36]:
             Timestamp   A   B   C   D
0  1900-01-01 00:00:00  10  10  10  10
1  1900-01-01 00:10:00          10  25
2  1900-01-01 00:20:00          10  25
3  1900-01-01 00:30:00              25
4  1900-01-01 00:40:00              25
5  1900-01-01 00:50:00  25  25  25  25
6  1900-01-01 01:00:00  30  30  30  30
7  1900-01-01 01:10:00  42  42  42  42
10 1900-01-01 01:40:00  40      40  40
11 1900-01-01 01:50:00          35  35


v2=df.set_index('Timestamp').stack().swaplevel(1,0).sort_index(level=0)

v3=v2[v2==''].to_frame().reset_index(level=1)
v3['New']=v3.Timestamp.diff().astype('timedelta64[m]').ne(10).cumsum()

v3.groupby([v3.index.get_level_values(level=0),v3.New]).Timestamp.agg(['first','last']).reset_index()
Out[74]:
  level_0  New               first                last
0       A    1 1900-01-01 00:10:00 1900-01-01 00:40:00
1       A    2 1900-01-01 01:50:00 1900-01-01 01:50:00
2       B    3 1900-01-01 00:10:00 1900-01-01 00:40:00
3       B    4 1900-01-01 01:40:00 1900-01-01 01:50:00
4       C    5 1900-01-01 00:30:00 1900-01-01 00:40:00

Answer 2

Let's use IntervalIndex and boolean indexing:

Create Interval index for df_b

df_b.index = pd.IntervalIndex.from_arrays(df_b['StartTime'],df_b['EndTime'], closed='both')

df_a[~df_a[['Timestamp']].apply(lambda x: df_b.index.contains(x.values), axis=1)]

Output:

             Timestamp     A     B     C     D
0  2018-01-01 00:00:00  10.0  10.0  10.0  10.0
1  2018-01-01 00:10:00   NaN   NaN  10.0  25.0
2  2018-01-01 00:20:00   NaN   NaN  10.0  25.0
3  2018-01-01 00:30:00   NaN   NaN   NaN  25.0
4  2018-01-01 00:40:00   NaN   NaN   NaN  25.0
5  2018-01-01 00:50:00  25.0  25.0  25.0  25.0
6  2018-01-01 01:00:00  30.0  30.0  30.0  30.0
7  2018-01-01 01:10:00  42.0  42.0  42.0  42.0
10 2018-01-01 01:40:00  40.0   NaN  40.0  40.0
11 2018-01-01 01:50:00   NaN   NaN  35.0  35.0