Bloomberg API: how do I associate request with response on IntradayBarRequest?

Question

Unfortunately I am led on the forums to believe (but without 100% certainty) that the Bloomberg Desktop API does not allow more than one IntradayBarRequest or IntradayTickRequest at a time, which is different from HistoricalDataRequest or Subscriptions, where multiple simultaneous requests are allowed.

This question is thus probably moot, unless somebody tells me that the above is not true

If true, then the only way to handle the question below, is to send each new request only after the previous one has been handled.

---

I am using the Python Bloomberg Desktop API to access subscription (live updating) and historical daily data for financial securities. In both of those cases I can send multiple simultaneous requests, and when the responses come (not necessarily in the order the requests were sent), I can find out which security the response is associated with, using msg.getElement("securityData").getElementAsString("security") in the case of historical data, or in the case of subscription data, by querying the correlationId that I had set beforehand (at subscription request time) using msg.correlationIds()[0].value().

However I don't know how to do this for IntradayBarResponse requests (and WAPI docs are not helpful). These do not seem to have a setable correlationId, nor do they have the above "securityData" field. If I send multiple intradayBarRequests, how can I find out for which security the responses are?

Here is my code (adapted from the API Python examples).

import blpapi  # interface to bloomberg
import time    # will need this for time parsing
from optparse import OptionParser
import pdb     # debugger, when necessary
import csv     # for csv reading
import string  # for string parsing
from pymongo import MongoClient
import inspect
from datetime import datetime
from bson.son import SON


def parseCmdLine():
    parser = OptionParser(description="Retrieve realtime data.")
    parser.add_option("-a",
                      "--ip",
                      dest="host",
                      help="server name or IP (default: %default)",
                      metavar="ipAddress",
                      default="localhost")
    parser.add_option("-p",
                      dest="port",
                      type="int",
                      help="server port (default: %default)",
                      metavar="tcpPort",
                      default=8194)
    parser.add_option("--me",
                      dest="maxEvents",
                      type="int",
                      help="stop after this many events (default: %default)",
                      metavar="maxEvents",
                      default=100000000000)
    parser.add_option("--mongohost",
                      dest="mongohost",
                      default="192.168.1.30")
    parser.add_option("--mongoport",
                      dest="mongoport",
                      type="int",
                      default=27017)

    (options, args) = parser.parse_args()

    return options


def main():
    options = parseCmdLine()

    # connect to MongoDB MONGO MONGO MONGO MONGO ----------------
    print "Connecting to MongoDB"
    print options.mongohost
    print options.mongoport
    client = MongoClient(options.mongohost, options.mongoport) # connect to MongoDB
    db = client.bb # connect to the DB database
    bbsecs = db.bbsecs
    bbticks = db.bbticks
    # now get the securities list


    # Fill SessionOptions
    sessionOptions = blpapi.SessionOptions()
    sessionOptions.setServerHost(options.host)
    sessionOptions.setServerPort(options.port)

    print "connecting to Bloomberg"
    print "Connecting to %s:%d" % (options.host, options.port)

    # Create a Session
    session = blpapi.Session(sessionOptions)

    # Start a Session
    if not session.start():
        print "Failed to start session."
        return

    # open the market data subscription service
    if not session.openService("//blp/mktbar"):
        print "Failed to open //blp/mktbar"
        return
    if not session.openService("//blp/refdata"):
        print "Failed to open //blp/refdata"
        return



    # now startup the subscription list
    # Now open the secs.dat file and read it, append each to subscription list
    maxtimes = bbticks.aggregate([{'$group': {'_id':'$ticker', 'maxtime':{'$max': '$time'}}}]) # get the last updates by ticker
    refDataService = session.getService("//blp/refdata") # start the ref
    for i in maxtimes["result"]:
        ticker = i["_id"]
        tstamp = i["maxtime"]
        request = refDataService.createRequest("IntradayBarRequest")
        request.set("security", ticker)
        request.set("eventType", "TRADE")
        request.set("interval", 1)
        request.set("startDateTime", tstamp)
        request.set("endDateTime", datetime.now())
        print "Sending Request:", ticker
        session.sendRequest(request)

    subscriptions = blpapi.SubscriptionList()
    secdic = dict() # a new dictionary
    for post in bbsecs.find():
        print(post["ticker"])
        # subscribe tick
        #subscriptions.add(str(post["ticker"]), "LAST_PRICE", [], blpapi.CorrelationId("TICK:" + str(post["ticker"])))
        #subscribe 1 minute bars
        subscriptions.add("//blp/mktbar/ticker/"+str(post["ticker"]), 
                          "LAST_PRICE", 
                          "interval=1.0",
                          blpapi.CorrelationId(str(post["ticker"])))
        # setup the dictionary
        secdic[post["bbsecnum"]] = post["ticker"]
    if not session.openService("//blp/refdata"):
        print "Failed to open //blp/refdata"
        return
    # now subscribe
    session.subscribe(subscriptions)


    # HISTORICALHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH
    # Obtain previously opened service
    #refDataService = session.getService("//blp/refdata")
    # Create and fill the request for the historical data
    #request = refDataService.createRequest("HistoricalDataRequest")
    #for post in bbsecs.find():  
    #    request.getElement("securities").appendValue(str(post["ticker"]))
    #request.getElement("fields").appendValue("LAST_PRICE")
    #request.set("periodicityAdjustment", "ACTUAL")
    #request.set("periodicitySelection", "DAILY")
    #request.set("startDate", "20100101")
    #request.set("endDate", "20121231")
    #request.set("maxDataPoints", 2000)
    #print "Sending Request:", request
    # Send the request
    #session.sendRequest(request)
    #hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh


    try:
        # Process received events
        eventCount = 0
        while(True):
            # We provide timeout to give the chance to Ctrl+C handling:
            event = session.nextEvent(500)
            for msg in event:
                if event.eventType() == blpapi.Event.SUBSCRIPTION_STATUS:
                    #print "%s - %s" % (msg.correlationIds()[0].value(), msg)
                    print "subscription status"
                elif event.eventType() == blpapi.Event.SUBSCRIPTION_DATA:
                    key = msg.correlationIds()[0].value()
                    if msg.messageType() == "MarketBarStart":
                        open = msg.getElementAsFloat("OPEN")
                        high = msg.getElementAsFloat("HIGH")
                        low = msg.getElementAsFloat("LOW")
                        close = msg.getElementAsFloat("CLOSE")
                        btstamp = msg.getElementAsDatetime("TIME")
                        tstamp = datetime.now()
                        print "bar", key, close, tstamp
                        bbticks.insert({"type": "BAR", "ticker": key, "value": close, \
                                "open": open, "high": high, "low": low, "close": close, \
                                "time": tstamp})
                    elif msg.messageType() == "MarketBarUpdate":
                        close = msg.getElementAsFloat("CLOSE")
                        #print "tick", close,
                        #bbticks.insert({"type": "TICK", "ticker": key, "value": close, "time": tstamp})

                    #if etype == "TRADE":
                    #    if msg.hasElement("LAST_TRADE"):
                    #        key = msg.correlationIds()[0].value(),
                    #        keytype = key[:(key.index(":"))]
                    #        key = key[(key.index(":") + 1):]
                    #        value = msg.getElementAsString("LAST_TRADE")
                    #        timestamp = msg.getElementAsDatetime("TRADE_UPDATE_STAMP_RT")
                    #        print key, value, 
                    #        bbticks.insert({"ticker": key, "value": value, "timestamp": timestamp})

                else:
                    if msg.messageType() == "HistoricalDataResponse":
                        securityData = msg.getElement("securityData")
                        security = securityData.getElementAsString("security")
                        fieldDataArray = securityData.getElement("fieldData")
                        for j in range(0, fieldDataArray.numValues()):
                            fieldData = fieldDataArray.getValueAsElement(j)
                            field = fieldData.getElement(0)
                            tstamp = field.getValueAsDatetime()
                            tstamp = datetime(tstamp.year, tstamp.month, tstamp.day)
                            field = fieldData.getElement(1)
                            close = field.getValueAsFloat()
                            #print "history", security, close, 
                            #bbticks.insert({"type": "DAILY", "ticker": security, "value": close, "close": close, \
                            #        "time": tstamp})
                    elif msg.messageType() == "IntradayBarResponse":
                        print "IntradayBarResponse"
                        data = msg.getElement("barData").getElement("barTickData")
                        numvals = data.numValues()
                        print numvals
                        if numvals > 0:
                            print data.getValueAsElement(1).getElement(1).getValueAsFloat()



            if event.eventType() == blpapi.Event.SUBSCRIPTION_DATA:
                eventCount += 1
                if eventCount >= options.maxEvents:
                    break
    finally:
        # Stop the session
        session.stop()

if __name__ == "__main__":
    try:
        main()
    except KeyboardInterrupt:
        print "Ctrl+C pressed. Stopping..."

I have looked at the eidData and that is empty, even if I ask for it to be returned. I am looking at currencies, rather than equities, so there is no exchange entitlement required.

> (Pdb) print eid._Element__dataHolder IntradayBarResponse = {
>     barData = {
>         eidData[] = {
>         }
>         barTickData[] = {
>             barTickData = {
>                 time = 2013-08-02T18:36:00.000
>                 open = 4.233100
>                 high = 4.233600
>                 low = 4.233100
>                 close = 4.233400
>                 volume = 0
>                 numEvents = 119
>                 value = 0.000000
>             }
>             barTickData = {
>                 time = 2013-08-02T18:37:00.000
>                 open = 4.233400
>                 high = 4.233700
>                 low = 4.233100
>                 close = 4.233500
>                 volume = 0
>                 numEvents = 96
>                 value = 0.000000
>             }
>             barTickData = {
>                 time = 2013-08-02T18:38:00.000
>                 open = 4.233500
>                 high = 4.233600
>                 low = 4.233300
>                 close = 4.233500
>                 volume = 0
>                 numEvents = 135
>                 value = 0.000000
>             }
>             barTickData = {
>                 time = 2013-08-02T18:39:00.000

I am still looking for a way to associate requests with responses, without having to do the inefficient request....wait for response.... request etc. I am not sure if Bloomberg provides this functionality, however. This limitation appears to exist for historical tick data as well.

Answer 1

Thomas, you should always use correlationId of a response to find which request it is associated with. IIRC, reference data and historical data requests support multiple securities, but market data and intraday bars may have only one security per request.

That is why there is additional "securityData" field in refdata and historical responses. For market data and intraday bars correlationId is enough to identify the request and hence the security.

Hope it helps.

Answer 2

I don't know the python API but I do use the Java API.

I am still looking for a way to associate requests with responses, without having to do the inefficient request....wait for response.... request etc.

As you discovered, you can't send a query for multiple securities for IntradayBarRequests and the feed does not contain security ids (eg tickers) to easily map back to your queries.

The easiest solution is to use a correlation ID. When you submit a request to a session, you can provide your own correlation ID. The example below is in Java but I would think that the python API is similar:

Session session = ...; //Bloomberg Session
CorrelationId cId = new CorrelationId(); //Unique ID
session.sendRequest(bbRequest, cId); //provide your own ID

Then, if you use an asynchronous session, you receive asynchronous messages which have a link to the original CorrelationId:

public void processEvent(Event event, Session session) {
    for (Message msg : event) {
        CorrelationID cId = msg.correlationID();
        //here you can link the result back to your original query
    }
}

Answer 3

I don't believe you can subscript to tick or bar data. Here's one reason why... For pricing data, you get a response with each change to the value. Isn't tick data merely a history of those prices? For bar data, isn't that a history of prices with some detail removed by grouping by a period (ie. 10 minutes, 1 hour). So if you could subscribe to either, when should you receive each new response from Bloomberg? Essentially, you could create your own listener that does the grouping, and then sends out a response from the listener with a frequency of your own choosing.

BTW: I use CLR to talk to the .net blp object from python. My solution predates any support Bloomberg for Python, essentially doing what 'can't be done'.