计算记录每天在特定状态下花费的时间

Question

Could anybody please help me with the next task?

Here is a problem: we have a table of history (status changes of processes) and we need to calculate how much time (in hours per day) a process was in a particular status. Here is the structure of history table:

ID| ProcessId| CreatedDate         | Status 
------------------------------------------- 
1 | Process1 | 2016-01-09 06:30:00 | UP
2 | Process1 | 2016-01-09 12:30:00 | UP
3 | Process1 | 2016-01-09 18:30:00 | DOWN
4 | Process1 | 2016-01-10 00:30:00 | UP 
5 | Process2 | 2016-01-08 18:30:00 | UP
6 | Process2 | 2016-01-09 00:30:00 | DOWN
7 | Process2 | 2016-01-09 06:30:00 | DOWN
8 | Process2 | 2016-01-09 12:30:00 | DOWN
9 | Process2 | 2016-01-09 18:30:00 | DOWN
10| Process2 | 2016-01-10 00:30:00 | UP
11| Process2 | 2016-01-10 06:30:00 | UP
12| Process2 | 2016-01-10 12:30:00 | DOWN
13| Process2 | 2016-01-10 18:30:00 | DOWN
14| Process2 | 2016-01-11 00:30:00 | DOWN
15| Process2 | 2016-01-11 06:30:00 | DOWN

As a result we need to create a view / table like:

ProcessId | Status | Date        | TimeSpentInStatusInDays
----------------------------------------------------------
Process1  | UP     | 2016-01-09  | 12h 00m
Process1  | DOWN   | 2016-01-09  | 05h 30m
Process1  | UP     | 2016-01-10  | 00h 00m
Process1  | DOWN   | 2016-01-10  | 00h 30m
Process2  | UP     | 2016-01-08  | 05h 30m
Process2  | DOWN   | 2016-01-08  | 00h 00m
Process2  | UP     | 2016-01-09  | 24h 00m
Process2  | DOWN   | 2016-01-09  | 00h 00m
Process2  | UP     | 2016-01-10  | 12h 00m
Process2  | DOWN   | 2016-01-10  | 12h 00m
Process2  | UP     | 2016-01-11  | 00h 00m
Process2  | DOWN   | 2016-01-11  | 06h 30m

Values are for example (they are not connected to actual data set).

The code needs to be in mySQL. Any help would be much appreciated.Thanks.

Answer 1

I'm not promising this is a good way to do this in MySQL or that it's fast.

I take your history table and append rows where necessary for the end of each day (except the last day per process.) The added rows contain the status of the final row per process per day. This could indeed result in an instantaneous status change at midnight if such a row already existed. (I tried to handle this scenario later.)

Since MySQL doesn't have lead/lag functions I'm matching up each row of two identical copies of the above to find the next time in sequence (which may be the logical status row added for end of day.) After that it's just a matter of grouping.

Since I'm not as familiar with MySQL date functions I just went with time_to_sec since the span can never be more the a day. The only complication is that midnight has to be treated specially. I'll let you deal with converting the seconds value to an appropriate output format.

http://sqlfiddle.com/#!9/b0f3279/44

select
    ProcessId,
    date(CreatedDate) as `Date`,
    Status,
    sum(
        case
            when time_to_sec(NextDate) = 0 then 86400
            else time_to_sec(NextDate)
        end - time_to_sec(CreatedDate)
    ) as TimeSpentSeconds
from
    (
    select
        h1.ProcessId, h1.CreatedDate, h1.Status,
        min(
            h2.CreatedDate
            --case
            --    when date(h2.CreatedDate) > date(h1.CreatedDate)
            --    then date_add(date(h1.CreatedDate), interval 1 day)
            --    else h2.CreatedDate
            --end 
        ) as NextDate
    from
        (
        select ProcessId, CreatedDate, Status from history
        union
        select
            ProcessId,
            date_add(date(CreatedDate), interval 1 day),
            substring(
                max(
                    concat(
                        date_format(CreatedDate, get_format(datetime, 'ISO')),
                        Status
                    )
                ), 20, 10) as LastStatus
        from history h0
        where date(CreatedDate) <
            (
                select max(date(CreatedDate)) from history hm
                where hm.ProcessId = h0.ProcessId
            )
        group by ProcessId, date(CreatedDate)
        ) h1
            inner join
        (
        select ProcessId, CreatedDate, Status from history
        union
        select
            ProcessId,
            date_add(date(CreatedDate), interval 1 day),
            substring(
                max(
                    concat(
                        date_format(CreatedDate, get_format(datetime, 'ISO')),
                        Status
                    )
                ), 20, 10) as LastStatus
        from history h0
        where date(CreatedDate) <
            (
                select max(date(CreatedDate)) from history hm
                where hm.ProcessId = h0.ProcessId
            )
        group by ProcessId, date(CreatedDate)
        ) h2
            on      h2.ProcessId   = h1.ProcessId
                and h1.CreatedDate < h2.CreatedDate
                and h2.CreatedDate <= date_add(date(h1.CreatedDate), interval 1 day)
    group by h1.ProcessId, h1.CreatedDate, h1.Status
    ) hx
group by ProcessId, date(CreatedDate), Status
order by ProcessId, `Date`, Status desc, TimeSpentSeconds

I believe this second option would handle my instantaneous/duplicate status scenario mentioned above. It was already a little complicated but this feels a lot more messy. I added a sort of sequence number to facilitate a tie break and tweaked the time difference expression. Finally I included a having clause to eliminate rows with zero accumulation from being spit out. Refer to ProcessX in the fiddle's sample data:

select
    ProcessId,
    date(CreatedDate) as `Date`,
    Status,
    sum(
        case
            when NextDate > CreatedDate and time_to_sec(NextDate) = 0 then 86400
            else time_to_sec(NextDate)
        end - time_to_sec(CreatedDate)
    ) as TimeSpentSeconds
from
    (
    select
        h1.ProcessId, h1.CreatedDate, h1.Status,
        min(
            h2.CreatedDate,
            --case
            --    when date(h2.CreatedDate) > date(h1.CreatedDate)
            --    then date_add(date(h1.CreatedDate), interval 1 day)
            --    else h2.CreatedDate
            --end 
        ) as NextDate
    from
        (
        select 1 as Sequence, ProcessId, CreatedDate, Status from history
        union all
        select
            0,
            ProcessId,
            date_add(date(CreatedDate), interval 1 day),
            substring(
                max(
                    concat(
                        date_format(CreatedDate, get_format(datetime, 'ISO')),
                        Status
                    )
                ), 20, 10) as LastStatus
        from history h0
        where date(CreatedDate) <
            (
                select max(date(CreatedDate)) from history hm
                where hm.ProcessId = h0.ProcessId
            )
        group by ProcessId, date(CreatedDate)
        ) h1
            inner join
        (
        select 1 as Sequence, ProcessId, CreatedDate, Status from history
        union all
        select
            0,
            ProcessId,
            date_add(date(CreatedDate), interval 1 day),
            substring(
                max(
                    concat(
                        date_format(CreatedDate, get_format(datetime, 'ISO')),
                        Status
                    )
                ), 20, 10) as LastStatus
        from history h0
        where date(CreatedDate) <
            (
                select max(date(CreatedDate)) from history hm
                where hm.ProcessId = h0.ProcessId
            )
        group by ProcessId, date(CreatedDate)
        ) h2
            on      h2.ProcessId   = h1.ProcessId
                and (
                        h1.CreatedDate <  h2.CreatedDate
                    and h2.CreatedDate <= date_add(date(h1.CreatedDate), interval 1 day)
                    or
                        h1.CreatedDate =  h2.CreatedDate
                    and h1.Sequence    <  h2.Sequence
                )   
    group by h1.ProcessId, h1.CreatedDate, h1.Status
    ) hx
group by ProcessId, date(CreatedDate), Status
having TimeSpentSeconds > 0 /* MySQL shortcut reference */
order by ProcessId, `Date`, Status desc, TimeSpentSeconds

http://sqlfiddle.com/#!9/b582b2/10

I just kind of realized that my expressions for NextDate don't need to check for midnight overrun so I commented that out. I didn't change the fiddles though. And I also forgot to mention that I am assuming there's at least one status report per day for each process. Maybe this is a starting point to play around with other MySQL options like temp tables (for speed) or variables (for lead/lag.)

Answer 2

I liked your question because it gave me a reason to play around with SQL, which I didn't have a chance to do in a while.

Here is my take on your question.

First, we prepare a temporary table TempStatusLog , where for each day we add a record at 00:00:01 with the status equal to the earliest reading of that day, and a record at 23:59:59 with the latest reading of that day. We also number all the rows using a variable @rownumvar . Assuming, that the original table is called StatusLog , the temporary table is created using this SELECT statement:

SELECT @rownumvar := @rownumvar + 1 AS `rowNo`,
       `t`.`ProcessId`, `t`.`CreatedDate`, `t`.`Status`
FROM (SELECT `ProcessId`, `CreatedDate`, `Status`
      FROM   `StatusLog`

      UNION

      SELECT `ProcessId`,
             STR_TO_DATE(CONCAT(`OnDate`, ' 23:59:59'),
                         '%Y-%m-%d %H:%i:%s') AS `CreatedDate`,
            (SELECT `Status`
             FROM   `StatusLog` AS `l`
             WHERE  `l`.`ProcessId` = `t1`.`ProcessId` AND
                    `l`.`CreatedDate`
                      = STR_TO_DATE(CONCAT(`t1`.`OnDate`, ' ', `t1`.`LastStatus`),
                                    '%Y-%m-%d %H:%i:%s')) AS `Status`
      FROM (SELECT `ProcessId`,
                   DATE_FORMAT(`CreatedDate`, '%Y-%m-%d') AS `OnDate`,
                   DATE_FORMAT(MAX(TIME(`CreatedDate`)), '%H:%i:%s') AS `LastStatus`
            FROM   `StatusLog`
            GROUP BY DATE(`OnDate`), `ProcessId`
            ORDER BY `ProcessId`, DATE(`OnDate`)) AS `t1`

      UNION

      SELECT `ProcessId`,
             STR_TO_DATE(CONCAT(`OnDate`, ' 00:00:01'),
                         '%Y-%m-%d %H:%i:%s') AS `CreatedDate`,
            (SELECT `Status`
             FROM   `StatusLog` AS `l`
             WHERE  `l`.`ProcessId` = `t2`.`ProcessId` AND
                    `l`.`CreatedDate`
                      = STR_TO_DATE(CONCAT(`t2`.`OnDate`, ' ', `t2`.`FirstStatus`),
                                    '%Y-%m-%d %H:%i:%s')) AS `Status`
      FROM (SELECT `ProcessId`,
                   DATE_FORMAT(`CreatedDate`, '%Y-%m-%d') AS `OnDate`,
                   DATE_FORMAT(MIN(TIME(`CreatedDate`)), '%H:%i:%s') AS `FirstStatus`
            FROM   `StatusLog`
            GROUP BY DATE(`OnDate`), `ProcessId`
            ORDER BY `ProcessId`, DATE(`OnDate`)) AS `t2`) AS `t`,
     (SELECT @rownumvar := 0) AS `r`
ORDER BY `t`.`ProcessId`, `t`.`CreatedDate` ASC

Now it is relatively easy to calculate for how long each process was in each state every day. We select a running window of two rows (this is where numbered rows come into play) and calculate the time differences between each two readings, which are then summed up:

SELECT `p`.`ProcessId`,
       DATE_FORMAT(`q`.`CreatedDate`, '%Y-%m-%d') AS `Day`,
       DATE_FORMAT(
         SEC_TO_TIME(
           SUM(
             TIME_TO_SEC(
               TIMEDIFF(TIME(`q`.`CreatedDate`),
                        TIME(`p`.`CreatedDate`))
             )
           )
         ),
         '%H:%i:%s'
       ) AS `Elapsed`,
       `p`.`Status`
FROM   `TempStatusLog` AS `p`,
       `TempStatusLog` AS `q`
WHERE  `q`.`rowNo` = `p`.`rowNo` + 1 AND
       DATE(`q`.`CreatedDate`) = DATE(`p`.`CreatedDate`)
GROUP BY `Day`, `Status`, `ProcessId`
ORDER BY `Day` ASC, `ProcessId` ASC, `Status` ASC

There are two minor issues with this solution:

1. It loses 2 seconds every day. Ie, if a process was up the whole day, it will say it was up for 23:59:58.
2. If the process was up the whole day, there will be no record saying that it was down for 00:00:00 (and vice-versa)

To me, both issues seem to be too minor to bother about.

Here you can take a look at a live demo: http://www.sqlfiddle.com/#!9/0a79cc/1

Note that SQLFiddle does not allow to create temporary tables, so I created a normal table for that purpose.

PS: It was considerably harder to solve this in MySQL than it would have been in almost any other RDBMS, for MySQL does not support many features of SQL. For one, it does not support CTE, which is a part of ANSI SQL specs. This forces users to create temporary tables or find other similar workarounds. Many RDBMSs (Oracle, SQL Server) also support some variations of ROW_NUMBER() function, which I had to work around using a variable.

Answer 3

Just for fun. Go for Postgres :)

select 
  ProcessId, CreatedDate, Status, 
  to_char( CreatedDate - lag( CreatedDate ) over ( order by CreatedDate, ProcessId ), 'HH24:MI' ) as diff
from history
order by ProcessId, ID;

http://sqlfiddle.com/#!15/83cb0/9