Backtrader problems in resampling to daily
Question
Using backtrader, I would like to retrieve quotes at a five minute scale and resample at a daily scale.
I am able to resample five minutes to sixty minutes, but can't do daily. Here is the code:
from __future__ import absolute_import, division, print_function, unicode_literals
import backtrader as bt
import backtrader.stores.ibstore as ibstore
import datetime
import logging
logging.basicConfig(level=logging.DEBUG)
logger = logging.getLogger(__name__)
_CLIENTID = 100
class St(bt.Strategy):
def __init__(self):
self.sma = bt.indicators.SMA(self.data)
def logdata(self):
txt = []
txt.append('{}'.format(len(self)))
txt.append('{}'.format(self.data.datetime.datetime(0).isoformat()))
txt.append('{:.2f}'.format(self.data.open[0]))
txt.append('{:.2f}'.format(self.data.high[0]))
txt.append('{:.2f}'.format(self.data.low[0]))
txt.append('{:.2f}'.format(self.data.close[0]))
txt.append('{:.2f}'.format(self.data.volume[0]))
logger.debug(','.join(txt))
data_live = False
def notify_data(self, data, status, *args, **kwargs):
print('*' * 5, 'DATA Notification:', data._getstatusname(status), *args)
if status == data.LIVE:
self.data_live = True
def next(self):
self.logdata()
if not self.data_live:
return
_TICKER = "TSLA-STK-SMART-USD"
_FROMDATE = datetime.datetime(2021,1,4)
_TODATE = datetime.datetime(2021,1,29)
_HAS_STATS = False
def run(args=None):
cerebro = bt.Cerebro(stdstats=_HAS_STATS)
store = ibstore.IBStore(host="127.0.0.1", port=7497, clientId= _CLIENTID )
cerebro.broker = store.getbroker()
stockkwargs = dict(
timeframe=bt.TimeFrame.Minutes,
compression=10,
rtbar=False,
historical=True,
qcheck=0.5,
fromdate=_FROMDATE,
todate=_TODATE,
latethrough=False,
tradename=None
)
data0 = store.getdata(dataname=_TICKER, **stockkwargs)
# cerebro.resampledata(data0, timeframe=bt.TimeFrame.Minutes, compression=60)
cerebro.resampledata(data0, timeframe=bt.TimeFrame.Days, compression=1)
cerebro.run()
if __name__ == "__main__":
run()
I get a connection, but no days. Here is the output (with no bars printed):
Server Version: 76
TWS Time at connection:20210303 09:39:17 EST
***** DATA Notification: DELAYED
***** DATA Notification: DISCONNECTED
However, if I go on a 60-minute resample, all is well. The code
cerebro.resampledata(data0, timeframe=bt.TimeFrame.Minutes, compression=60)
# cerebro.resampledata(data0, timeframe=bt.TimeFrame.Days, compression=1)
results in
Server Version: 76
TWS Time at connection:20210303 09:36:27 EST
***** DATA Notification: DELAYED
DEBUG:__main__:30,2021-01-05T17:00:00,749.65,754.40,735.11,753.21,6011.00
DEBUG:__main__:31,2021-01-05T18:00:00,753.39,754.18,751.60,753.34,1126.00
DEBUG:__main__:32,2021-01-05T19:00:00,753.32,753.32,750.49,752.90,1179.00
DEBUG:__main__:33,2021-01-06T04:00:00,748.00,752.55,746.66,751.88,331.00
DEBUG:__main__:34,2021-01-06T05:00:00,751.60,753.00,749.26,750.50,137.00
(omitted)
DEBUG:__main__:286,2021-01-28T17:00:00,833.00,833.25,831.36,831.61,116.00
DEBUG:__main__:287,2021-01-28T18:00:00,831.60,833.96,830.11,831.00,175.00
DEBUG:__main__:288,2021-01-28T19:00:00,830.51,832.00,829.45,829.45,358.00
***** DATA Notification: DISCONNECTED
I am using these versions:
Python 3.7.4
ib 0.8.0
IbPy2 0.8.0
numpy 1.19.2
pandas 1.1.3
TL;DR I did several other experiments.
| getData timeframe | getData compression | resample timeframe | resample compression | outcome |
|---|---|---|---|---|
| Minutes | 5 | Minutes | 60 | can see hourly, as expected |
| Minutes | 5 | Days | 1 | (empty) |
| (blank) | (blank) | Days | 1 | (empty) |
| Days | 1 | Days | 1 | (empty) |
| Days | 1 | (commented out) | (commented out) | (empty) |
1 answers
solution1
1 ACCPTED 2021-03-26 23:18:04
You are right there was a known issue a few years a back with this. There is an easy work-around though. Instead of using the backtrader resampling facility, use Interactive Brokers.
You can bypass resample and call the data directly from IB. Just remember that you must add the shortest timeframe first to Backtrader.
for tf_com in [(bt.TimeFrame.Minutes, 1), (bt.TimeFrame.Days, 1)]:
stockkwargs = dict(
timeframe=tf_com[0],
compression=tf_com[1],
rtbar=False,
historical=True,
qcheck=0.5,
fromdate=_FROMDATE,
todate=_TODATE,
latethrough=False,
tradename=None
)
data0 = store.getdata(dataname=_TICKER, **stockkwargs)
cerebro.adddata(data0)
You can adjust your timeframe and compression in the tuples in the first line. I will adjust it to 5 and 15 minutes so you can see the output.
for tf_com in [(bt.TimeFrame.Minutes, 5), (bt.TimeFrame.Minutes, 15)]:
************ OUTPUT ************
2021-03-25 12:00:00 636.77 632.71
2021-03-25 12:05:00 634.69 632.71
2021-03-25 12:10:00 632.71 632.71
2021-03-25 12:15:00 640.39 643.00
2021-03-25 12:20:00 640.68 643.00
2021-03-25 12:25:00 643.00 643.00
2021-03-25 12:30:00 641.33 640.00
2021-03-25 12:35:00 637.84 640.00
2021-03-25 12:40:00 640.00 640.00
2021-03-25 12:45:00 640.34 633.95
2021-03-25 12:50:00 636.15 633.95
2021-03-25 12:55:00 633.95 633.95
2021-03-25 13:00:00 633.50 637.43
2021-03-25 13:05:00 634.95 637.43
2021-03-25 13:10:00 637.43 637.43
EDIT: IN RESPONSE TO OP COMMENT
I beg to differ. The days are sampled by Interactive Brokers so you will have daily information. You must be sure when using 5 minute data at the same time not to go back too far with your start date. IB is not a historical data provider.
Here is the entire code for your reference.
import backtrader as bt
import backtrader.stores.ibstore as ibstore
import datetime
import os
from dotenv import load_dotenv
load_dotenv()
class St(bt.Strategy):
def __init__(self):
self.data_live = False
self.timeframes = {4: "minute", 5: "day"}
def next(self):
time = self.data.datetime.time()
start_day = datetime.time(4, 20, 0)
end_day = datetime.time(19, 40, 0)
if time < start_day or time > end_day:
print_date = True
else:
print_date = False
if not self.data_live and print_date:
print(
f"{self.data.datetime.datetime()} "
f"data0: tf: {self.timeframes[self.datas[0]._timeframe]} "
f"comp: {self.datas[0]._compression}, "
f"{self.datas[0].close[0]:5.2f} "
f"data1: tf: {self.timeframes[self.datas[1]._timeframe]} "
f"comp: {self.datas[1]._compression} "
f"{self.datas[1].close[0]:5.2f} "
)
return
_TICKER = "TSLA-STK-SMART-USD"
_FROMDATE = datetime.datetime(2021, 3, 10)
_TODATE = datetime.datetime(2021, 3, 24)
_HAS_STATS = False
_CLIENTID = os.getenv("CLIENTID")
_PORT = os.getenv("SOCKET_PORT")
def run():
cerebro = bt.Cerebro(stdstats=_HAS_STATS)
cerebro.addstrategy(St)
store = ibstore.IBStore(host="127.0.0.1", port=int(_PORT), clientId=_CLIENTID)
cerebro.broker = store.getbroker()
for tf_com in [(bt.TimeFrame.Minutes, 5), (bt.TimeFrame.Days, 1)]:
stockkwargs = dict(
timeframe=tf_com[0],
compression=tf_com[1],
rtbar=False,
historical=True,
qcheck=0.5,
fromdate=_FROMDATE,
todate=_TODATE,
latethrough=False,
tradename=None,
)
data = store.getdata(dataname=_TICKER, **stockkwargs)
cerebro.adddata(data)
cerebro.run()
if __name__ == "__main__":
run()
Here is the output with the middle of the days removed.
Please note that backtrader will set the next day value at the end of the last bar of the previous day.
2021-03-10 19:55:00 data0: tf: minute comp: 5, 664.56 data1: tf: day comp: 1 664.56
2021-03-11 04:00:00 data0: tf: minute comp: 5, 699.10 data1: tf: day comp: 1 664.56
2021-03-11 04:05:00 data0: tf: minute comp: 5, 701.50 data1: tf: day comp: 1 664.56
2021-03-11 04:10:00 data0: tf: minute comp: 5, 699.00 data1: tf: day comp: 1 664.56
2021-03-11 04:15:00 data0: tf: minute comp: 5, 701.00 data1: tf: day comp: 1 664.56
2021-03-11 19:45:00 data0: tf: minute comp: 5, 698.05 data1: tf: day comp: 1 664.56
2021-03-11 19:50:00 data0: tf: minute comp: 5, 698.09 data1: tf: day comp: 1 664.56
2021-03-11 19:55:00 data0: tf: minute comp: 5, 698.50 data1: tf: day comp: 1 664.56
2021-03-11 19:55:00 data0: tf: minute comp: 5, 698.50 data1: tf: day comp: 1 698.50
2021-03-12 04:00:00 data0: tf: minute comp: 5, 674.25 data1: tf: day comp: 1 698.50
2021-03-12 04:05:00 data0: tf: minute comp: 5, 676.00 data1: tf: day comp: 1 698.50
2021-03-12 04:10:00 data0: tf: minute comp: 5, 669.00 data1: tf: day comp: 1 698.50
2021-03-12 04:15:00 data0: tf: minute comp: 5, 669.46 data1: tf: day comp: 1 698.50
2021-03-12 19:45:00 data0: tf: minute comp: 5, 693.30 data1: tf: day comp: 1 698.50
2021-03-12 19:50:00 data0: tf: minute comp: 5, 692.80 data1: tf: day comp: 1 698.50
2021-03-12 19:55:00 data0: tf: minute comp: 5, 692.99 data1: tf: day comp: 1 698.50
2021-03-12 19:55:00 data0: tf: minute comp: 5, 692.99 data1: tf: day comp: 1 692.99
2021-03-15 04:00:00 data0: tf: minute comp: 5, 689.00 data1: tf: day comp: 1 692.99
2021-03-15 04:05:00 data0: tf: minute comp: 5, 692.13 data1: tf: day comp: 1 692.99
2021-03-15 04:10:00 data0: tf: minute comp: 5, 692.12 data1: tf: day comp: 1 692.99
2021-03-15 04:15:00 data0: tf: minute comp: 5, 692.31 data1: tf: day comp: 1 692.99
2021-03-15 19:45:00 data0: tf: minute comp: 5, 702.00 data1: tf: day comp: 1 692.99
2021-03-15 19:50:00 data0: tf: minute comp: 5, 702.00 data1: tf: day comp: 1 692.99
2021-03-15 19:55:00 data0: tf: minute comp: 5, 702.00 data1: tf: day comp: 1 692.99
2021-03-15 19:55:00 data0: tf: minute comp: 5, 702.00 data1: tf: day comp: 1 702.00
2021-03-16 04:00:00 data0: tf: minute comp: 5, 704.01 data1: tf: day comp: 1 702.00
2021-03-16 04:05:00 data0: tf: minute comp: 5, 704.98 data1: tf: day comp: 1 702.00
2021-03-16 04:10:00 data0: tf: minute comp: 5, 704.99 data1: tf: day comp: 1 702.00
2021-03-16 04:15:00 data0: tf: minute comp: 5, 706.20 data1: tf: day comp: 1 702.00
2021-03-16 19:45:00 data0: tf: minute comp: 5, 673.65 data1: tf: day comp: 1 702.00
2021-03-16 19:50:00 data0: tf: minute comp: 5, 674.00 data1: tf: day comp: 1 702.00
2021-03-16 19:55:00 data0: tf: minute comp: 5, 674.10 data1: tf: day comp: 1 702.00
2021-03-16 19:55:00 data0: tf: minute comp: 5, 674.10 data1: tf: day comp: 1 674.10
2021-03-17 04:00:00 data0: tf: minute comp: 5, 672.00 data1: tf: day comp: 1 674.10
2021-03-17 04:05:00 data0: tf: minute comp: 5, 675.80 data1: tf: day comp: 1 674.10
2021-03-17 04:10:00 data0: tf: minute comp: 5, 677.19 data1: tf: day comp: 1 674.10
2021-03-17 04:15:00 data0: tf: minute comp: 5, 676.03 data1: tf: day comp: 1 674.10
2021-03-17 19:45:00 data0: tf: minute comp: 5, 699.66 data1: tf: day comp: 1 674.10
2021-03-17 19:50:00 data0: tf: minute comp: 5, 699.90 data1: tf: day comp: 1 674.10
2021-03-17 19:55:00 data0: tf: minute comp: 5, 699.74 data1: tf: day comp: 1 674.10
2021-03-17 19:55:00 data0: tf: minute comp: 5, 699.74 data1: tf: day comp: 1 699.74
2021-03-18 04:00:00 data0: tf: minute comp: 5, 685.00 data1: tf: day comp: 1 699.74
2021-03-18 04:05:00 data0: tf: minute comp: 5, 686.35 data1: tf: day comp: 1 699.74
2021-03-18 04:10:00 data0: tf: minute comp: 5, 688.32 data1: tf: day comp: 1 699.74
2021-03-18 04:15:00 data0: tf: minute comp: 5, 692.50 data1: tf: day comp: 1 699.74
2021-03-18 19:45:00 data0: tf: minute comp: 5, 652.10 data1: tf: day comp: 1 699.74
2021-03-18 19:50:00 data0: tf: minute comp: 5, 651.00 data1: tf: day comp: 1 699.74
2021-03-18 19:55:00 data0: tf: minute comp: 5, 650.56 data1: tf: day comp: 1 699.74
2021-03-18 19:55:00 data0: tf: minute comp: 5, 650.56 data1: tf: day comp: 1 650.56
2021-03-19 04:00:00 data0: tf: minute comp: 5, 661.00 data1: tf: day comp: 1 650.56
2021-03-19 04:05:00 data0: tf: minute comp: 5, 663.00 data1: tf: day comp: 1 650.56
2021-03-19 04:10:00 data0: tf: minute comp: 5, 663.60 data1: tf: day comp: 1 650.56
2021-03-19 04:15:00 data0: tf: minute comp: 5, 666.48 data1: tf: day comp: 1 650.56
2021-03-19 19:45:00 data0: tf: minute comp: 5, 652.50 data1: tf: day comp: 1 650.56
2021-03-19 19:50:00 data0: tf: minute comp: 5, 652.02 data1: tf: day comp: 1 650.56
2021-03-19 19:55:00 data0: tf: minute comp: 5, 652.20 data1: tf: day comp: 1 650.56
2021-03-19 19:55:00 data0: tf: minute comp: 5, 652.20 data1: tf: day comp: 1 652.20
2021-03-22 04:00:00 data0: tf: minute comp: 5, 665.97 data1: tf: day comp: 1 652.20
2021-03-22 04:05:00 data0: tf: minute comp: 5, 664.00 data1: tf: day comp: 1 652.20
2021-03-22 04:10:00 data0: tf: minute comp: 5, 665.00 data1: tf: day comp: 1 652.20
2021-03-22 04:15:00 data0: tf: minute comp: 5, 663.94 data1: tf: day comp: 1 652.20
2021-03-22 19:45:00 data0: tf: minute comp: 5, 668.39 data1: tf: day comp: 1 652.20
2021-03-22 19:50:00 data0: tf: minute comp: 5, 668.56 data1: tf: day comp: 1 652.20
2021-03-22 19:55:00 data0: tf: minute comp: 5, 669.35 data1: tf: day comp: 1 652.20
2021-03-22 19:55:00 data0: tf: minute comp: 5, 669.35 data1: tf: day comp: 1 669.35
2021-03-23 04:00:00 data0: tf: minute comp: 5, 670.25 data1: tf: day comp: 1 669.35
2021-03-23 04:05:00 data0: tf: minute comp: 5, 665.33 data1: tf: day comp: 1 669.35
2021-03-23 04:10:00 data0: tf: minute comp: 5, 664.11 data1: tf: day comp: 1 669.35
2021-03-23 04:15:00 data0: tf: minute comp: 5, 662.93 data1: tf: day comp: 1 669.35
2021-03-23 19:45:00 data0: tf: minute comp: 5, 661.00 data1: tf: day comp: 1 669.35
2021-03-23 19:50:00 data0: tf: minute comp: 5, 661.08 data1: tf: day comp: 1 669.35
2021-03-23 19:55:00 data0: tf: minute comp: 5, 663.00 data1: tf: day comp: 1 669.35
2021-03-23 19:55:00 data0: tf: minute comp: 5, 663.00 data1: tf: day comp: 1 663.00
=== OP Verification ===
Here is what the data0 and data1 plot look like, adding a cerebro.plot() after the cerebro.run() .
That's good looking daily data.
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.
Related Question
Resampling daily to monthly data with Pandas
Resampling Dataframe from Monthly to Daily
Problems resampling Pandas DataFrame
Resampling daily ohlcv data to weekly and monthly
Pandas Resampling Hourly OHLC to Daily OHLC
Resampling with 'how=count' causing problems
Pandas : How to avoid fillna while resampling from hourly to daily data
Pandas resampling from daily to weekly adds an extra week? Is this normal?
Resampling grouped data to obtain daily average data using Pandas
pandas DataFrame interpolating/resampling daily data on a per-group basis
Related Tags