Unable to get Oanda historical prices

Question

I'm trying to get oanda historical prices for EURUSD. I'm trying to get the last 5000. But I'm only receiving 180.

This is my code:

def get_data_oanda(num_periods, **keyword_parameters):
    """
    Environment                 Description 
    fxTrade (Live)              The live (real money) environment 
    fxTrade Practice (Demo)     The demo (simulated money) environment 
    """
    domainDict = { 'live' : 'api-fxtrade.oanda.com','demo' : 'api-fxpractice.oanda.com' }
    environment = 'demo'
    domain = domainDict[environment]
    access_token = 'xxxx'
    account_id = 'xxxx'
    instruments = 'EUR_USD'
    count = num_periods
    granularity = "M1"
    try:
        s = requests.Session()
        url = "https://" + domain + "/v1/candles"
        headers = {'Authorization' : 'Bearer ' + access_token,
                   # 'X-Accept-Datetime-Format' : 'unix'
                  }
        params = {'instrument' : instruments, 'accountId' : account_id, 'count' : count, 'granularity' : granularity}
        req = requests.Request('GET', url, headers = headers, params = params)
        pre = req.prepare()
        resp = s.send(pre, stream = True, verify = True)
        return resp
    except Exception as e:
        s.close()
        print()
        print("Caught exception when connecting to stream\n" + str(e))


num_periods = 5000
my_date = datetime.datetime.now(pytz.timezone('America/Sao_Paulo')).strftime('%Y-%m-%dT%H:%M:%S')
timezone = 'America/Sao_Paulo' 
response = get_data_oanda(num_periods)
msg = json.loads(response.text)
candles = msg['candles']
for candle in candles:
    df_data = df_data.append({
            'date': datetime.datetime.strptime(candle['time'], '%Y-%m-%dT%H:%M:%S.000000Z').replace(tzinfo=pytz.utc).astimezone(local_tz).strftime('%Y-%m-%d %H:%M:%S'),
            'instrument': msg['instrument'],
            "open": candle['openAsk'],
            "high": candle['highAsk'],
            "low": candle['lowAsk'],
            "close": candle['closeAsk'],
            "volume": candle['volume']
        },ignore_index=True)

But df_data has only 180 rows, not 5000.

[180 rows x 7 columns]

How can I fix this?

Answer 1

How to fix this is to strip it down to the basics. Get a simple requests.get() call working and print the text. Once you have the API returning the results you need, then work on the Pandas side. If the API doesn't return the results you need, contact the service provider.

The proper way to load JSON in Pandas is far simpler, like this:

resp = requests.get(url, headers=headers, params=params, stream=True)
df = pd.read_json(resp.raw)
df_data = pd.io.json.json_normalize(df.candles)
df_data['time'] = pd.to_datetime(df_data.time)

Something like that will replace most of your code, with no slow loops.

Answer 2

for this API-call I know for sure that you should use:

    stream = False

Also consider using one of the API wrappers that are available to access the REST-API. These reduce your code to a few lines. The https://github.com/hootnot/oanda-api-v20 allows you to download more than 5000 records also.

For v1 (end-of life december 2017 ?)

https://github.com/oanda/oandapy

or v2:

https://github.com/oanda/v20-python

https://github.com/hootnot/oanda-api-v20

https://github.com/oanda/v20-python
https://github.com/hootnot/oanda-api-v20