在时间序列中插入缺失值并使用 R 创建一个时间序列 model，如协整、ARIMA(X) 和 SARIMA(X)

Question

I have a dataset with 18 variables and 30k+ observations, with data being collected from sensors every 15min. Some of the variables collect the temperature and these will be used as independent variables to predict the state of a structure. During this period some data is missing due to the sensors being offline and some other observations were removed because they were outliers.

Is there a way to interpolate the missing values in a fairly accurate way? I've been trying to create a dataset with all time values every 15min and then matching with the dataset with all values

df$Hour <- substr(df$Time,15,16)
df$Hour <- replace(df$Hour, df$Hour > 0 & df$Hour < 15, "00")
df$Hour <- replace(df$Hour, df$Hour > 15 & df$Hour < 30, 15)
df$Hour <- replace(df$Hour, df$Hour > 30 & df$Hour < 45, 30)
df$Hour <- replace(df$Hour, df$Hour > 45 & df$Hour < 60, 45)
library(lubridate)
df$Time <- ymd_hm(paste(substr(df$Time,1,14),df$Hour,sep=""))
allDates <- seq(ISOdate(2018,4,27,14,15), ISOdate(2018,9,10,19,45), by = "15 min")
allValues <- merge(data.frame(Time=allDates),df,all.x=TRUE)

After this I try to use the Amelia package but it return an error: "There are observations in the data that are completely missing" amelia(df, m = 5, p2s = 1, frontend = FALSE, idvars = "id", ts = "Time", cs = NULL, polytime = NULL, splinetime = NULL, intercs = FALSE, lags = 2:17, leads = 2:17, startvals = 0, tolerance = 0.0001, logs = NULL, sqrts = NULL, lgstc = NULL, noms = NULL, ords = NULL, incheck = TRUE, collect = FALSE, arglist = NULL, empri = 0.1*nrow(DadosTS), priors = NULL, autopri = 0.05, emburn = c(0,0), bounds = NULL, max.resample = 100, overimp = NULL, boot.type = "ordinary", ncpus = getOption("amelia.ncpus", 1L), cl = NULL) amelia(df, m = 5, p2s = 1, frontend = FALSE, idvars = "id", ts = "Time", cs = NULL, polytime = NULL, splinetime = NULL, intercs = FALSE, lags = 2:17, leads = 2:17, startvals = 0, tolerance = 0.0001, logs = NULL, sqrts = NULL, lgstc = NULL, noms = NULL, ords = NULL, incheck = TRUE, collect = FALSE, arglist = NULL, empri = 0.1*nrow(DadosTS), priors = NULL, autopri = 0.05, emburn = c(0,0), bounds = NULL, max.resample = 100, overimp = NULL, boot.type = "ordinary", ncpus = getOption("amelia.ncpus", 1L), cl = NULL)

Any method to deal with this problem / use the available data to create a time-series model would be very much appreciated

Answer 1

If you want time series interpolation / imputation you could use the imputeTS package.

library(imputeTS)
x <- na_interpolation(df)

That's it already.

But some additional hints: The package is doing the imputation (missing data replacement) based on the inter-time correlations of each variable. So this won't account for inter-variable correlations (dependent on the data this might improve the imputation). On the other hand Amelia like you used it only looks for inter-variable correlations. If one row is completely NA like in your case (NA,NA,NA,NA) it fails, since then there is no inter-variable correlation to estimate the missing value.

In imputeTS you also have different time series imputation algorithms to choose from. The code above is for imputation by linear interpolation. You could also do spline interpolation:

library(imputeTS)
x <- na_interpolation(df, option = "spine)

Another more advanced option, wich is especially useful if you know your time series has seasonality is using imputation by Kalman Smoothing on state-space-models / state space representation of an ARIMA model.

library(imputeTS)
x <- na_kalman(df)

There are also other imputation / interpolation functions available in the package (see this paper )

In general if you expect not too much inter-variable correlation, these listed methods from imputeTS will very likely be a good choice. If you do expect a strong inter-variable correlation, then also take a look at other approaches. Eg could also use imputeTS to impute just one column of your dataset and then use AMELIA again (then the error should disappear, since you do not have rows being completely NA any more).