How to created pandasData for backtrader with ccxt

Question

I am trying to back test a strategy using backtrader. Most of the examples I have seen only are using a csv file. I was wondering if it is possible to just get data from an exchange and turn it into pandas dataframe and then use backtrader? When I run it I get an error AttributeError: 'numpy.int64' object has no attribute 'lower' which is refers to pandafeed.py

import ccxt,os
from dotenv import load_dotenv
import backtrader

class Trader:
    def __init__(self) -> None:
        load_dotenv()
        self.connect()

    """ Creates Binance client """
    def connect(self):
        self.exchange = ccxt.binance({
            'apiKey': os.getenv('BINANCE_API_KEY'),
            'secret': os.getenv('BINANCE_API_SECRET')
        })

klines = Trader().exchange.fetch_ohlcv(symbol=trading_pair,timeframe=interval)
dataFrame = pd.DataFrame(klines)
dataFrame[0] = [datetime.fromtimestamp(t/1000) for t in dataFrame[0]]
data = backtrader.feeds.PandasData(dataname=dataFrame)

cerebro = backtrader.Cerebro()
cerebro.broker.set_cash(10000)
cerebro.adddata(data)
cerebro.run()

If I use a column name and change my code to the below

colums = ['datetime', 'open','high', 'low', 'close', 'volume']
dataFrame = pd.DataFrame(klines, columns=colums)
dataFrame["datetime"] = [datetime.fromtimestamp(t/1000) for t in dataFrame["datetime"]]
data = backtrader.feeds.PandasData(dataname=dataFrame)

I get this error

AttributeError: 'int' object has no attribute 'to_pydatetime'

pandafeed.py

My question is: how do I turn a list into something which I can use to run backtrader? thank you.

PS an example data structure return by klines will be like

[ [1621152000000, 49375.28, 49795.89, 48656.0, 49014.99, 10956.006583], [1621166400000, 49014.99, 49249.06, 47566.01, 47727.26, 14166.961995], [1621180800000, 47727.26, 48097.59, 44444.44, 45549.26, 36819.653456], [1621195200000, 45553.24, 46480.0, 43825.39, 46431.5, 28724.055984], [1621209600000, 46426.83, 46686.0, 42777.0, 42915.46, 28171.858447], [1621224000000, 42915.46, 45400.0, 42196.97, 45149.18, 40557.45817], [1621238400000, 45143.28, 45800.0, 44291.84, 45731.39, 23851.50751], [1621252800000, 45733.55, 45791.04, 43156.0, 43362.75, 23137.989315], [1621267200000, 43357.0, 44400.0, 42001.0, 44197.73, 30883.162039], [1621281600000, 44197.73, 44939.2, 42500.0, 43538.04, 20055.197255], [1621296000000, 43538.02, 45281.34, 43150.79, 44779.83, 19252.919453], [1621310400000, 44774.78, 45799.29, 44738.26, 45172.7, 17218.430549], [1621324800000, 45172.69, 45420.0, 44607.08, 45225.71, 8427.020047] ]

Answer 1

I think backtrader is trying to read headers where there are none. Try telling PandasData there are no headers. See the docs

data = backtrader.feeds.PandasData(dataname=dataFrame, header=None)

Answer 2

I got the same issue, and after debug it i found it append because backtrader expect your dataframe to be indexed by the datetime (to make it simple). Then to solve it, just add to your code:

...
dataFrame = dataFrame.set_index('datetime')
...

In my case I also had also to change the type of the 'datetime' field and I did by:

...
dataFrame["datetime"] = dataFrame["datetime"].values.astype(dtype='datetime64[ms]')
dataFrame = dataFrame.set_index('datetime')
...

I hope that will help you, even if you asked that question 2 months ago.