Using IBrokers and R, what is the appropriate way to retrieve live trading data for multiple stocks?

Question

Obviously, a trader can easily be monitoring 100s of stocks in a live trading session.

Here is code to just retrieve live data for 30+ stock symbols:

tws <- twsConnect()
sym <- c("AAPL", "MSFT", "GOOGL", "TWTR", "MSFT", "BABA","JD", "WLTW", "DIA",
         "QQQ", "VOO", "MOMO","IBB", "VOOG", "VYM","VNQ", "BIDU", "BA",
         "AYX", "ROKU", "UBER","ACB", "CRON", "TTD","BAM", "AZO", "JPM",
         "WMT", "GRUB", "TSN")
contracts <- lapply(sym, function(x) twsEquity(x, 'SMART','ISLAND'))

system.time(
last_prices <-   lapply(contracts, function(x) reqMktData(tws, 
                                                          Contract = x,
                                                          snapshot = TRUE))

)

here is the performance metrics:

2 -1 2104 Market data farm connection is OK:usfarm 
2 -1 2106 HMDS data farm connection is OK:ushmds 
2 -1 2158 Sec-def data farm connection is OK:secdefil 
   user  system elapsed 
  0.408   0.367  46.727 
Warning message:
In .Internal(gc(verbose, reset, full)) :
  closing unused connection 6 (->localhost:7496)

This is obviously unacceptable. It should not take over 40+ seconds to retrieve live data for merely 30+ stocks.

Using quantmod::getSymbols, this operation is essentially instant.

What am I doing wrong?

I have an 8-core XEON processor. One of the latest models too. No way can it be my hardware.

Answer 1

Ibrokers is single threaded only. Which is why some of the ibrokers functions keep running until you interrupt the R session. This is an basically an R issue, as R is (for now) single threaded. You could simulate this by creating new threads, as ib can handle this, but that requires an asynch languange, and that is not R. For example, socketConnection can't handle parallel (asych) requests, you would need to set up a complete new connection for each thread. Possible, but then tracking the messages returning from ib correctly becomes more difficult.

The reqMktData function accepts only 1 ticker (independent of language) and you need to loop over the tickers. The speed of of this tends to be an issue in other languages as well, overhead, speed of the ib servers, latency etc etc etc. But comparing this to the yahoo retrieve you are doing to with quantmod is not exactly the same. There are a lot of checks going on on the ib side, like is it a correct message, and the message being send back needs to be decoded before it shows you the data. All of that adds overhead.

Normally you would request market data streams with reqMktData (snapshot = FALSE) up to 100 tickers at the same time, and check the stream realtime for the last price, but then you need to use python, c# or c++ (or java). Also you would need to distinguish between tickers, which means using requestids or other identifying info. For python see the ib_insync library. Info here .

For more info on ib api issues, check the twsapi group for the latest bugs / questions regarding the api. R not much represented there as this is still on a very old version of the api. Also check the interactive-brokers tag on stackoverflow. This might also give you some ideas / help.