Extracting multiple contracts from reqHistoricalData not working

Question

I am trying to store the results of reqHistoricalData into a dictionary of dataframes, where each key is the ticker name and the value is the corresponding time series data frame.

def historical_data_handler(msg):
    global hist, df
    if "finished" in msg.date:
        df = pd.DataFrame(index=np.arange(0, len(hist)), columns=('date', 'open', 'high', 'low', 'close', 'volume'))
        for index, msg in enumerate(hist):
            df.loc[index,'date':'volume'] = datetime.strptime(msg.date, '%Y%m%d %H:%M:%S'), msg.open, msg.high, msg.low, msg.close, msg.volume
     else:   
         hist.append(msg)


def error_handler(msg):
    print(msg)

con = ibConnection(port=7496,clientId=75)
con.register(historical_data_handler, message.historicalData)
con.register(error_handler, message.Error)
con.connect()  
print("Connection to IB is ", con.isConnected(), " and starting data pull")

contracts = ["SPY", "AAPL"]
result = {}

for ct in contracts:
    hist = []    
    spec = Contract()  
    spec.m_symbol = ct
    spec.m_secType = 'STK'  
    spec.m_exchange = 'SMART'  
    spec.m_currency = 'USD'
    spec.m_expiry = '' # For futures
    con.reqHistoricalData(0, spec, '', '2 D', '5 mins', 'TRADES', 0, 1)
    print(spec.m_symbol, hist, df)
    result[ct] = df

print("Connection is terminated ", con.disconnect(), " after finishing pulling data")

The behavior of the code is not what I would expect. When I looking at my "result" dictionary. The values are the same across "SPY" and "APPL". I think there is something wrong with how I am defining the global variables as they dont seem to be updating properly in the for loop.

Any help would be greatly appreciated, thanks!

When I look at the head of the two dataframes stored in "result" (they are the same):

[326 rows x 6 columns]
 SPY                      date    open    high     low   close volume
 0    2018-02-09 04:00:00  261.08  261.16  260.92  260.99     68
 1    2018-02-09 04:05:00  260.99     261  260.86  260.99     59

[326 rows x 6 columns]
 AAPL                      date    open    high     low   close volume
 0    2018-02-09 04:00:00  261.08  261.16  260.92  260.99     68
 1    2018-02-09 04:05:00  260.99     261  260.86  260.99     59

Answer 1

Just looking at the data, it seems you have about twice as many rows as you should for 2 days. The hist = [] is run twice in succession before the replies from IB are processed. So you must be getting all the data for SPY and AAPL in one list.

Try clearing hist in the finished branch of the data handler. I think that's where you should ask for the next contract as well. You can then put a sleep(10) to avoid pacing violations if you ever ask for more data.