How to calculate rolling correlations between several columns and one column in rollapply?

Question

Problem

If I want to calculate the rolling correlation between each of my 39 stocks in their columns in stock_returns (xts object) and the market_return (separate xts object, only one column with market returns) with rollapply:

rolling_3yearcor <- rollapply(stock_returns,width=750,FUN=cor,y=market_return)

I get this error:

Error in FUN(.subset_xts(data, (i - width + 1):i, j), ...) : incompatible dimensions

Even if I subset the single column in the market_return with

rolling_3yearcor <- rollapply(stock_returns,width=750,FUN=cor,y=market_return$market)

I get the error as well, even though they have the same dimensions?! (1 col, same number of rows).

What I'd like to have:

I want an xts object with the correlations of stock[i] with the market in each of the 39 stock-columns in a rolling 750 days window instead of the daily returns in stock_returns.

Shouldn't rollapply do exactly that?

EDIT 1: Data sample for problem of one day backwards shifting

Returns StockA
1997-01-03 -0.0054065397
1997-01-06  0.0024139001
1997-01-07 -0.0030085614
1997-01-08  0.0054329941
1997-01-09 -0.0005990317
1997-01-10 -0.0102205387
...

with code:

ind <- market_return
ind[] <- seq_along(market_return)
rolling_3yearcor <- function(x,y,ind){
  rollapply(ind,width=5,function(i) cor(x[i],y[i]))
}
rollcor_3year <- lapply(stock_returns,rolling_3yearcor,market_return,ind)
rollcor_3year <- as.data.frame(rollcor_3year,col.names=names(stock_returns))
colnames(rollcor_3year) <- colnames(stock_returns)
rollcor_3year <- as.xts(rollcor_3year)

gives me:

dput(head(rollcor_3year$StockA.N))
structure(c(NA, NA, NA, NA, 0.30868769358199, 0.576490782746284
), .indexCLASS = c("POSIXct", "POSIXt"), tclass = c("POSIXct", 
"POSIXt"), .indexTZ = "", tzone = "", class = c("xts", "zoo"), index = 
structure(c(852246000, 
852505200, 852591600, 852678000, 852764400, 852850800), tzone = "", tclass = 
c("POSIXct", 
"POSIXt")), .Dim = c(6L, 1L), .Dimnames = list(NULL, "StockA.N"))

then with:

indexTZ(rollcor_3year) <- "UTC"
dput(head(rollcor_3year$StockA.N))

structure(c(NA, NA, NA, NA, 0.30868769358199, 0.576490782746284
), .indexCLASS = c("POSIXct", "POSIXt"), tclass = c("POSIXct", 
"POSIXt"), .indexTZ = c(TZ = "UTC"), tzone = c(TZ = "UTC"), class = c("xts", 
"zoo"), index = structure(c(852246000, 852505200, 852591600, 
852678000, 852764400, 852850800), tzone = c(TZ = "UTC"), tclass = 
c("POSIXct", 
"POSIXt")), .Dim = c(6L, 1L), .Dimnames = list(NULL, "StockA.N"))

it gives me :

head(rollcor_3year$StockA.N)
1997-01-02 23:00:00        NA
1997-01-05 23:00:00        NA
1997-01-06 23:00:00        NA
1997-01-07 23:00:00        NA
1997-01-08 23:00:00 0.3086877
1997-01-09 23:00:00 0.5764908

Answer 1

The issue is that you are trying to calculate correlation between vectors of different length. Try cor(1:10, 1:9) to see this directly. rollapply only rolls its first argument, so market_return$market is used in its entirety.

One method for dealing with this is to roll over an index of the vectors (assuming both are originally the same length). I don't have access to your data, so some data:

set.seed(2)
df1 <- as.data.frame(replicate(5, runif(10), simplify=FALSE))
names(df1) <- paste0("V", 1:5)
vec2 <- runif(10)

Looking at the first column of the frame, we can show the proof-of-concept:

rollapply(seq_along(vec2), 3, function(i) cor(df1$V1[i], vec2[i]))
# [1]  0.2873624 -0.8522555 -0.9859923 -0.6394554 -0.4626926  0.4939377  0.5590373  0.9994124

To easily apply this to all columns of the frame, we can make a helper function:

rollcor <- function(v1,v2) {
  rollapply(seq_along(v1), 3, function(i) cor(v1[i], v2[i]))
}

lapply(df1, rollcor, vec2)
# $V1
# [1]  0.2873624 -0.8522555 -0.9859923 -0.6394554 -0.4626926  0.4939377  0.5590373  0.9994124
# $V2
# [1]  0.79602807  0.16857013 -0.24970680  0.01997719  0.96922386 -0.99937633 -0.32920929
# [8] -0.34819538
# $V3
# [1]  0.78978134 -0.08632500 -0.13991114 -0.26078798 -0.05284222  0.24405994 -0.68231437
# [8] -0.48694537
# $V4
# [1]  0.9850739  0.9823811  0.9743629  0.8470096  0.7337313 -0.9617746 -0.7033091 -0.4968143
# $V5
# [1] -0.6696637 -0.8672182 -0.9074534 -0.7671002 -0.3954844 -0.9864078 -0.2806075 -0.5689732

EDIT

Since you say it's a time-series, while we still need to use the indices (and not the time-series vector itself), we can preserve the time-series with two techniques:

1. Use zoo:::rollapply.ts on the unmodified code (before this edit). This is slightly discouraged since it is relying on an unexported function. I think it is generally safe, but it's not good form in the long run.

2. Apply the same time series to the indices we'll roll over.

    ind <- vec2 ind[] <- seq_along(vec2) rollapply(ind, 3, function(i) cor(df1$V1[i], vec2[i])) rollcor <- function(v1,v2,ind) { rollapply(ind, 3, function(i) cor(v1[i], v2[i])) } lapply(df1, rollcor, vec2, ind) 

Answer 2

Use rollapplyr with the indicated function and by.column = FALSE .

# test data
stock_returns <- xts(anscombe[6:8], as.Date("2000-01-01") + seq(0, length=nrow(anscombe)))
market <- xts(anscombe[, 5], time(stock_returns))

x <- cbind(market, stock_returns)   
rollapplyr(x, 5, function(x) cor(x[, 1], x[, -1]), by.column = FALSE)

giving:

2000-01-01         NA          NA         NA
2000-01-02         NA          NA         NA
2000-01-03         NA          NA         NA
2000-01-04         NA          NA         NA
2000-01-05  0.6912899 -0.19831742  0.8437913
2000-01-06 -0.0904641 -0.08067339  0.3773026
2000-01-07  0.3714166 -0.05974574  0.3604551
2000-01-08  0.9013902  0.90672036 -0.6537459
2000-01-09  0.9059692  0.91388127 -0.7673776
2000-01-10  0.7996265  0.89299770 -0.7032847
2000-01-11  0.7812519  0.89427224 -0.6959074