日志条目的第一次和最后一次出现

Question

I am aware that there are already many similar questions on stackoverflow, but I just could not find anything, nor could I come up with my own solution... Here we go:

I run consistency checks on a changing set of data X and with every run I might find some violations. I can identify the cases of the violations with a unique key . Once a violation is resolved in the original data set X , the violation obviously disappears from the checks. A violation can reappear at a later date, and should then be considered new.

Every time I run the checks, I create a log file, which records the date and key of the violation.

From this logfile, I would like to extract like how many cases / violations were in status open, and how many cases have been cumulatively closed at any date in the log file:

This is another transformation of the left table which might help understand the result (numbers in (.) refer to the corresponding line in the left table):

AAA is opened on 5/1/2020 (1) and closed on 5/4/2020 because there is no entry for AAA on 5/4/2020, but we know tests were run on that date (6).

AAA is opened again after it was closed on 5/5/2020, and closed on 5/7/2020, because tests were run on that date, and AAA did not show up anymore (8). 5/6/2020 never showed.

BBB was opened on 5/2/2020, and appeared throughout 5/7/2020, so never closed.

Here is the skeleton code:

df = pd.DataFrame(
    {
        "date": [
            "5/1/2020",
            "5/2/2020",
            "5/3/2020",
            "5/5/2020",
            "5/2/2020",
            "5/3/2020",
            "5/4/2020",
            "5/5/2020",
            "5/7/2020",
        ],
        "key": ["AAA"] * 4 + ["BBB"] * 5,
    }
)
df['date'] = df['date'].astype("datetime64")

I believe I have to work with the date ladder ( date_ladder = df[['date']].drop_duplicates().sort_values(by='date') ) and do an outer merge with df, to get values for each key on all dates, and then continue from there. But I already fail there to create that merge.

Answer 1

So, here is an attempt after sleeping over it... First, we will add a dummy variable and then pivot over the dataframe and melt it back again:

df['v'] = True
pvt = df.pivot_table(index='date', columns='key', values='v').fillna(False)
df2 = pvt.reset_index().melt(id_vars='date')
"""
df2:
         date  key  value
0  2020-05-01  AAA   True
1  2020-05-02  AAA   True
2  2020-05-03  AAA   True
3  2020-05-04  AAA  False
4  2020-05-05  AAA   True
5  2020-05-07  AAA  False
6  2020-05-01  BBB  False
7  2020-05-02  BBB   True
8  2020-05-03  BBB   True
9  2020-05-04  BBB   True
10 2020-05-05  BBB   True
11 2020-05-07  BBB   True
"""

Now we will shift the dataframe by 1 and then check if there is a switch from True to False (or vice versa), indicating that a record in the original log file appeared or dissappeared:

df2['switch'] = (df2['value'] ^ df2.shift()['value']) |( (df2['key']!=df2.shift()['key']) & df2['value'])
df2['start'] = df2['switch'] & df2['value']
df2['end'] = df2['switch'] & ~df2['value']

df2['status'] = ''
df2.loc[df2['start'], 'status'] = 'Start'
df2.loc[df2['end'], 'status'] = 'End'
"""
         date  key  value  switch  start    end status
0  2020-05-01  AAA   True    True   True  False  Start
1  2020-05-02  AAA   True   False  False  False       
2  2020-05-03  AAA   True   False  False  False       
3  2020-05-04  AAA  False    True  False   True    End
4  2020-05-05  AAA   True    True   True  False  Start
5  2020-05-07  AAA  False    True  False   True    End
6  2020-05-01  BBB  False   False  False  False       
7  2020-05-02  BBB   True    True   True  False  Start
8  2020-05-03  BBB   True   False  False  False       
9  2020-05-04  BBB   True   False  False  False       
10 2020-05-05  BBB   True   False  False  False       
11 2020-05-07  BBB   True   False  False  False       
"""

We can condense this with pd.melt() :

df3 = df2.loc[df2['status']!=''].melt(id_vars=['key', 'status'], value_vars='date', value_name='date').drop('variable', axis=1)
"""
   key status       date
0  AAA  Start 2020-05-01
1  AAA    End 2020-05-04
2  AAA  Start 2020-05-05
3  AAA    End 2020-05-07
4  BBB  Start 2020-05-02
"""

The next step is to sort the data by date, because we want to know how many items are open or closed at any time.

df4 = df2.sort_values(by='date')
df4['opened'] = df4['start'].cumsum()
df4['closed'] = df4['end'].cumsum()
"""
         date  key  value  switch  ...    end  status opened  closed
0  2020-05-01  AAA   True    True  ...  False   Start      1       0
6  2020-05-01  BBB  False   False  ...  False              1       0
1  2020-05-02  AAA   True   False  ...  False              1       0
7  2020-05-02  BBB   True    True  ...  False   Start      2       0
2  2020-05-03  AAA   True   False  ...  False              2       0
8  2020-05-03  BBB   True   False  ...  False              2       0
3  2020-05-04  AAA  False    True  ...   True     End      2       1
9  2020-05-04  BBB   True   False  ...  False              2       1
4  2020-05-05  AAA   True    True  ...  False   Start      3       1
10 2020-05-05  BBB   True   False  ...  False              3       1
5  2020-05-07  AAA  False    True  ...   True     End      3       2
11 2020-05-07  BBB   True   False  ...  False              3       2
"""

Finally, we will run a groupby() and calculate how many items are open on a specific date and plot the result:

result = df4.groupby('date').agg(opened=pd.NamedAgg(column='opened', aggfunc='max'), closed = pd.NamedAgg(column='closed', aggfunc='max'))
result['open_on'] = result['opened']-result['closed']
result[['closed', 'open_on']].plot.area()