在一个图上显示多个时间序列中的缺失值

Question

I have five different time-series for continuous three years. Now I want to show the missing values in these series by the gaps on a plot. So, I thought that I will create another data frame corresponding to these series and where ever I have a value, I will replace that with one and leave NA's as such. Such a dummy data frame is as:

# create sample time index
timeindex <- seq(as.POSIXct("2014-01-01"),as.POSIXct("2016-12-31"),by="1 mins")
# create 5 sample series of same length as of time index
sequence_1 <- sample(seq(from = 0, to = 1, by = 1), size =  length(timeindex), replace = TRUE)
sequence_2 <- sample(seq(from = 0, to = 1, by = 1), size =  length(timeindex), replace = TRUE)
sequence_3 <- sample(seq(from = 0, to = 1, by = 1), size =  length(timeindex), replace = TRUE)
sequence_4 <- sample(seq(from = 0, to = 1, by = 1), size =  length(timeindex), replace = TRUE)
sequence_5 <- sample(seq(from = 0, to = 1, by = 1), size =  length(timeindex), replace = TRUE)
# create data frame of sequences
df <- data.frame(sequence_1,sequence_2,sequence_3,sequence_4,sequence_5)
df <- ifelse(df==0,NA,1) # replace 0 with NA to show missing data values
df_with_time <- data.frame(timeindex,df) # attach timestamp to sequences

Now the question is how to show missing values (gaps) in one graph. I melted my data frame and thought of using geom_line() with facet_grid() , but it seems that my computer hangs for an indefinite time. The code is:

library(ggplot2)
df_melt <- reshape2::melt(df_with_time,id.vars="timeindex") # melt for ggplot
ggplot(df_melt,aes(timeindex,value,variable)) +  geom_line() + facet_grid(variable~.)
#ggplot(df_melt,aes(timeindex,value,variable)) +  geom_area() + facet_grid(variable~.)

Now I have two questions:

1. Although ggplot fails to plot this huge data on a machine with 8GB RAM, 2.6 GHZ processor. Is there any other way to plot such huge data?
2. Is there any other way to show gaps(missing values) in the data?

UPDATE I want a plot something like this:

The missing data points are shown as gaps.

Answer 1

If aggregating NAs across series is no go, I would suggest to perform a time-based binning of your data. Briefly, you can count how many NA you have in 30-min or 60-min windows and plot the counts using ggplot. I show an example below.

# Binning
head(df_with_time)
time.gap <- 60 # bin by hour
idx <- seq(1, nrow(df_with_time), by = time.gap) 
na.counts <- lapply(idx[-length(idx)], (function(i){
  tmp <- df_with_time[i:(i+(time.gap-1)),]
  counts <- apply(tmp[,-1], 2, (function(y){ sum(is.na(y)) }))
  counts
}))
na.counts <- data.frame(time=df_with_time[idx[-length(idx)],]$timeindex, 
                        do.call(rbind, na.counts), 
                        stringsAsFactors = FALSE,
                        row.names = NULL)
head(na.counts)

# Convert to suitable df and then plot (color tracks with NA count)
df_melt <- reshape2::melt(na.counts,id.vars="time") # melt for ggplot
df_melt$y <- as.integer(as.factor(df_melt$variable))
df_melt <- df_melt[order(df_melt$value - median(df_melt$value)), ]

ggplot(df_melt,aes(x=time, y=y)) +  
  geom_point(aes(colour = value), shape = 124, alpha = 0.75, size = 2.5) + 
  scale_colour_gradient2(low = "#01665e", mid = "#f5f5f5", high = "#8c510a", midpoint = median(df_melt$value))

This is the result.

Alternatively, you may want to get rid of values too close to the median and only plot your 'outliers'. Since this removes a lot of data, the plot will be generated quickly.

df_melt2 <- df_melt[abs(df_melt$value - median(df_melt$value)) > 8, ]

ggplot(df_melt2,aes(x=time, y=y)) +  
  geom_point(aes(colour = value), shape = 124, alpha = 0.75, size = 4.5) + 
  scale_colour_gradient2(low = "#01665e", mid = "#f5f5f5", high = "#8c510a", midpoint = median(df_melt$value))

PS: I assumed you are interested in those values that are far from the median. If you care about total NA counts, use scale_colour_gradient() instead.

Answer 2

If you reshape it into long format:

data <- reshape2::melt(df_with_time,
                       id.vars="timeindex", 
                       variable.name = 'Sequence', 
                       value.name = 'Data')

you can plot it in ggplot like you want:

ggplot(data, 
       aes(x = timeindex, 
           y = Sequence, 
           size = Data)) + 
geom_line()

This is for a single month, to keep things smaller: