R：如何找到重叠数据点且没有缺失数据的最长周期？

Question

I have a very large time series dataset of electricity load from a substation which has been cleaned to have consistent time intervals of 15 minutes, however there are still large periods of missing data. The substation is split into individual feeders so is in the form:

Feeder <- c("F1","F1","F1","F1","F1", "F2","F2","F2","F2","F2", "F3","F3","F3","F3","F3")
Load <- c(3.1, NA, 4.0, 3.8, 3.6, 2.1, NA, 2.6, 2.9, 3.0, 2.4, NA, 2.3, 2.2, 2.5)

start <- as.POSIXct("2016-01-12 23:15:00")
end <- as.POSIXct("2016-01-13 00:15:00")
DateTimeseq <- seq(start, end, by = "15 min")
DateTime <- c(DateTimeseq, DateTimeseq, DateTimeseq)

dt <- data.frame(Feeder, Load, DateTime)

My actual data spans over a period of multiple years but I have condensed it down so it is easily replicable. As you can see, there are missing values. My actual dataset has large periods of missing data. In order to perform effective analysis, I need to find periods where there are no missing load data points for all feeders (ie. longest overlapping periods). If possible, I would like to generate a list of the longest overlapping periods without any NA values with the minimum being around 24 hours (I know this is not possible for the example I give but if you could show me how that would be great.). You could use a minimum of 15 minutes or something in this example.

As you can see from the simple data, the longest period would be 30 minutes between 2016-01-12 23:45:00 and 2016-01-13 00:15:00. However, in this example the second longest period would be 15 minutes but is inside the longest period. If possible, I would like to run it so it doesn't replicate values. If so, the second longest period in this case would be the overlapping point at 2016-01-12 23:15:00.

Feel free to play around with it and add more values if it would make it easier. It may be beneficial to create individual columns for the different feeders. I usually use pipes from dplyr but this is not essential. If you require anymore information do not hesitate to ask.

Thanks!

Answer 1

Perhaps, this will give you a start. For each Feeder you can create groups between NA values., calculate their first and last value and create a 15-minute sequence between them. You can then count which interval occur the most in the data.

library(dplyr)

dt %>%
  group_by(Feeder) %>%
  group_by(grp = cumsum(is.na(Load)), .add = TRUE) %>%
  #Use add = TRUE in old dplyr
  #group_by(grp = cumsum(is.na(Load)), add = TRUE) %>%
  summarise(start = first(DateTime), 
            end = last(DateTime)) %>%
  ungroup %>%
  mutate(datetime = purrr::map2(start, end, seq, by = '15 mins')) %>%
  tidyr::unnest(datetime) %>%
  select(-start, -end) %>%
  count(datetime, sort = TRUE)

Answer 2

Base R solution:

# Strategy 1 contiguous period classification:
data.frame(do.call("rbind", lapply(split(dt, dt$Feeder), function(x){
    y <- with(x, x[order(DateTime),])
    y$category <- paste0(y$Feeder, ":", cumsum(is.na(y$Load)) + 1)
    tmp <- y[!(is.na(y$Load)),]
    cat_diff <- do.call("rbind", lapply(split(tmp, tmp$category), 
                function(z){
                  data.frame(category = unique(z$category), 
                    max_diff = difftime(max(z$DateTime),
                                        min(z$DateTime), 
                                        units = "hours"))}))
    y$max_diff <- cat_diff$max_diff[match(y$category, cat_diff$category)] 
    return(y)
      }
    )
  ), row.names = NULL
)

Answer 3

Here is another option to cast into a wide table and check for consecutive rows without any NAs:

library(data.table)

wDT <- dcast(setDT(dt)[, na := +is.na(Load)], DateTime ~ Feeder, value.var="na")

wDT[, c("ri", "rr") := {
    ri <- rleid(rowSums(.SD)==0L)
    .(ri, rowid(ri))
}, .SDcols=names(wDT)[-1L]]
range(wDT[ri %in% ri[rr==max(rr)]]$DateTime)
#[1] "2016-01-12 23:45:00 +08" "2016-01-13 00:15:00 +08"

Answer 4

I might have a nice 3 lines of code solution for you:

1. First bringt the data into wide format, that each Feeder is a column
2. Check row wise (which is now timestamp wise), that all Feeders are non-NA. This gives something like 12:15 TRUE, 12:30 TRUE, 12:45 FALSE,... FALSE in this context means all Feeders are available for this timestamp
3. Do a run length encoding on the resulting True,True,False,False,... series - this enables finding what you call consecutive overlapping periods

Code:

 library("tidyr")
 library("dplyr")
 # Into wide format
 dt_wide <- dt %>% pivot_wider(names_from = Feeder, values_from = Load)

 # Check if complete row is available
  dt_anyna <- apply(y,1, anyNA)
 
 # Now we need to find the longest FALSE runs
  rle(dt_anyna)

This gives you a run length encoding, that looks the following

  Run Length Encoding
  lengths: int [1:3] 1 1 3
  values : logi [1:3] FALSE TRUE FALSE

Meaning at the beginning you have 1 False in a row, next 1 TRUE in a row, next 3 FALSE in a row.

You can now easily work with this results. You probably want to filter out the TRUE runs, because you are only looking for the longest run, where all data is available (these are the FALSE runs). Then you can look for the max() run and you can also look for eg runs > 4 (which would be 1h for your 15 mins data).

additional code for the question from Ellis

rle <- rle(dt_anyna)
x <- data.frame(  value = rle$values, duration = rle$lengths)
x$start <- dt_wide$DateTime[(cumsum(x$duration)- x$duration)+1]
x$end <-  dt_wide$DateTime[cumsum(x$duration)]
x$duration_s <-  x$end - x$start
ordered <- x[order(x$duration, decreasing = TRUE),]  
filtered <- filter(ordered, value == FALSE)
filtered

So just resuming where we ended before - you can add yourself start / end times / duration / sort and filter with this code. (you now must also call library("dplyr") in the beginning)

The results would looks like this:

value  duration   start                end                 duration_s
FALSE        3    2016-01-12 23:45:00 2016-01-13 00:15:00  1800 secs
FALSE        1    2016-01-12 23:15:00 2016-01-12 23:15:00     0 secs

This would give you a data.frame ordered by duration of consecutive non-NA segments with start and end times.