在时差内匹配观测值的最快方法

Question

I'm calculating price differences between trades that have a specific time difference (say 60 seconds). I need this to be done with several assets and several trades. However, I could not figure a way to do this without an eternal for-loop.

Let's create some random prices:

library(birk)
library(tictoc)
library(dplyr)

initial.date <- as.POSIXct('2018-10-27 10:00:00',tz='GMT')
last.date <- as.POSIXct('2018-10-28 17:00:00',tz='GMT')

PriorityDateTime=seq.POSIXt(from=initial.date,to = last.date,by = '30 sec')
TradePrice=seq(from=1, to=length(PriorityDateTime),by = 1)

ndf<- data.frame(PriorityDateTime,TradePrice)
ndf$InstrumentSymbol <- rep_len(x = c('asset1','asset2'),length.out = length(ndf$PriorityDateTime))
ndf$id <- seq(1:length(x = ndf$InstrumentSymbol))

My main function is the following: For each trade (at the TradePrice column) I need to find closest trade that falls in the 60-second interval.

calc.spread <- function(df,c=60){
  n<-length(df$PriorityDateTime)
  difft <- dspread <- spread <- rep(0,n)
  TimeF <- as.POSIXct(NA)
  for (k in 1:n){
    diffs <- as.POSIXct(df$PriorityDateTime) - as.POSIXct(df$PriorityDateTime[k])
    idx <- which.closest(diffs,x=c)  
    TimeF[k]<- as.POSIXct(df$PriorityDateTime[idx])
    difft[k] <- difftime(time1 = TimeF[k],time2 = df$PriorityDateTime[k], units = 'sec')
    dspread[k] <- abs(df$TradePrice[k] - df$TradePrice[idx])
    spread[k] <- 2*abs(log(df$TradePrice[k]) - log(df$TradePrice[idx]))

  }

  df <- data.frame(spread,dspread,difft,TimeF,PriorityDateTime=df$PriorityDateTime,id=df$id)
}

The function which.closest is just a wrapper for which.min(abs(vec - x)). As I have a data frame with multiple assets, I run:

c=60
spreads <- ndf %>% group_by(InstrumentSymbol) %>% do(calc.spread(.,c=c))

The problem is that I need to run this for 3-million row data frames. I have searched on the forum but couldn't find a way to run this code faster. Ddply is a little bit slower than using dplyr.

Is there any suggestion?

Answer 1

You might have made a mistake in the sense that you are not looking for the minimum difference within 60 secs difference as described, but instead you are looking for a trade which took place as close as possible to 60secs in past or future:

idx <- which.closest(diffs,x=c)

Using this a trade which took place 1 sec ago would be discarded for a trade that happened closer to 60 secs away, I don't think that this is what you want. You probably want the lowest price difference for all trades within 60 secs which can be done by:

res$idx[i] <<-  which.min(pricediff)[1]

See the code below:

library(lubridate)
library(dplyr)
ndf$datetime <- ymd_hms(ndf$PriorityDateTime)
res <- ndf %>% data.frame(stringsAsFactors = F)
res$dspread <- res$idx <- res$spread <- NA
sapply(1:nrow(res),function(i){
  within60 <- abs(difftime(ndf$datetime[i],ndf$datetime,"secs"))<=60
  samesymbol <- res$InstrumentSymbol[i]==res$InstrumentSymbol
  isdifferenttrade <- 1:nrow(res)!=i 
  pricediff <- ifelse(within60&samesymbol&isdifferenttrade,abs(res$TradePrice[i]-res$TradePrice), Inf)

  res$dspread[i] <<-  min(pricediff)
  res$idx[i] <<-  which.min(pricediff)[1] #in case several elements have same price 
  res$spread[i] <<-  2*abs(log(res$TradePrice[i])-log(res$TradePrice[res$idx[i]]))
} )
head(res)

What I used was apply which is similar to (and can be even slower than) for loops. If this is any faster for your real data, it is because I did the operations in a way which needed less steps.

Let me know, otherwise you can try the same in a for loop, or we'd have to try with data.table which I am less familiar with. These are generally time consuming of course because you need to define conditions based on each row of data.

     PriorityDateTime TradePrice InstrumentSymbol id            datetime    spread idx
1 2018-10-27 10:00:00          1           asset1  1 2018-10-27 10:00:00 2.1972246   3
2 2018-10-27 10:00:30          2           asset2  2 2018-10-27 10:00:30 1.3862944   4
3 2018-10-27 10:01:00          3           asset1  3 2018-10-27 10:01:00 2.1972246   1
4 2018-10-27 10:01:30          4           asset2  4 2018-10-27 10:01:30 1.3862944   2
5 2018-10-27 10:02:00          5           asset1  5 2018-10-27 10:02:00 1.0216512   3
6 2018-10-27 10:02:30          6           asset2  6 2018-10-27 10:02:30 0.8109302   4
  dspread
1       2
2       2
3       2
4       2
5       2
6       2

Answer 2

Being quite unsatisfied by my own previous answer, I asked here for help and turns out there is at least one way in data.table which is clearly faster. Also made a dplyr-related question here

s <- Sys.time()
initial.date <- as.POSIXct('2018-10-27 10:00:00',tz='GMT')
last.date <- as.POSIXct('2018-12-28 17:00:00',tz='GMT')
PriorityDateTime=seq.POSIXt(from=initial.date,to = last.date,by = '30 sec');length(PriorityDateTime)
TradePrice=seq(from=1, to=length(PriorityDateTime),by = 1)
ndf<- data.frame(PriorityDateTime,TradePrice)
ndf$InstrumentSymbol <- rep_len(x = c('asset1','asset2'),length.out = length(ndf$PriorityDateTime))
ndf$id <- seq(1:length(x = ndf$InstrumentSymbol))
ndf$datetime <- ymd_hms(ndf$PriorityDateTime)
res <- ndf %>% data.table()
res2 <- setDT(res)
res2 <- res2[, `:=` (min_60 = datetime - 60, plus_60 = datetime + 60, idx = .I)][
  res2,  on = .(InstrumentSymbol = InstrumentSymbol, datetime >= min_60, datetime <= plus_60), allow.cartesian = TRUE][
    idx != i.idx, .SD[which.min(abs(i.TradePrice - TradePrice))], by = id][
      , .(id, minpricewithin60 = i.TradePrice, index.minpricewithin60 = i.idx)][
        res, on = .(id)][, `:=` (min_60 = NULL, plus_60 = NULL, idx = NULL)]
res2[]
e <- Sys.time()
> e-s
Time difference of 1.23701 mins

You can then apply your calc.spread function directly to the minpricewithin60 column.