KeyError pandas dataframe

Question

The purpose of this script is to read a csv file then create a data frame out of it.

The file contains forex historical data.

The file has 7 columns Date, Time, Open, High, Low, Close and Volume, and around 600k rows.

Here is a data sample:

                        Open     High      Low    Close  Volume
Release Date                                                   
2020-02-05 01:50:00  109.450  109.452  109.449  109.451      79
2020-02-05 01:51:00  109.451  109.451  109.449  109.450      26
2020-02-05 01:52:00  109.451  109.453  109.449  109.449      29
2020-02-05 01:53:00  109.449  109.449  109.440  109.442      35
2020-02-05 01:54:00  109.443  109.443  109.432  109.432      49
2020-02-05 01:55:00  109.432  109.439  109.432  109.438      19
2020-02-05 01:56:00  109.439  109.450  109.439  109.449      56
2020-02-05 01:57:00  109.449  109.450  109.446  109.446      20
2020-02-05 01:58:00  109.446  109.451  109.446  109.448      33
2020-02-05 01:59:00  109.449  109.454  109.443  109.443      75

After scraping the date and time the script must make some date time calculation like month and day.

Then, some technical analysis using TA-LIB library.

With every new step the code produces a dataframe.

All the new dataframes will be stored in a list.

The last step is to concatenate all these dataframes into one final dataframe.

Here is the code:

import pandas as pd
import talib


class Data:
    def __init__(self):
        self.dfs = []
        self.names = ['Date', 'Time', 'Open', 'High', 'Low', 'Close', 'Volume']
        self.df = pd.DataFrame()
        self.Close = self.df['Close'].astype(float)

    def file(self, file):
        self.df = pd.read_csv(file, names=self.names,
                              parse_dates={'Release Date': ['Date', 'Time']})
        return self.dfs.append(self.df)

    def year(self):
        self.df['year'] = self.df['Release Date'].dt.year
        return self.dfs.append(self.df['year'])

    def month(self):
        self.df['month'] = self.df['Release Date'].dt.month
        return self.dfs.append(self.df['month'])

    def week(self):
        self.df['week'] = self.df['Release Date'].dt.week
        return self.dfs.append(self.df['week'])

    def day(self):
        self.df['day'] = self.df['Release Date'].dt.day
        return self.dfs.append(self.df['day'])

    def hour(self):
        self.df['hour'] = self.df['Release Date'].dt.hour
        return self.dfs.append(self.df['hour'])

    def minute(self):
        self.df['minute'] = self.df['Release Date'].dt.minute
        return self.dfs.append(self.df['minute'])

    def dema(self):
        self.df['DEMA'] = talib.DEMA(self.Close, timeperiod=30)
        return self.dfs.append(self.df['dema'])

    def ema(self):
        self.df['EMA'] = talib.EMA(self.Close, timeperiod=30)
        return self.dfs.append(self.df['ema'])

    def KAMA(self):
        self.df['KAMA'] = talib.KAMA(self.Close, timeperiod=30)
        return self.dfs.append(self.df['KAMA'])

    def ma(self):
        self.df['MA'] = talib.MA(self.Close, timeperiod=30, matype=0)
        return self.dfs.append(self.df['ma'])

    def action(self):
        self.year()
        self.month()
        self.week()
        self.day()
        self.hour()
        self.minute()
        self.dema()
        self.ema()
        self.KAMA()
        self.ma()

    def print(self):
        self.action()
        print(len(self.dfs))


x = Data()
x.file(r"D:\Projects\Project Forex\EURUSD.csv")
x.print()

Here is the error:

Traceback (most recent call last):

  File "C:\Users\Sayed\miniconda3\lib\site-packages\pandas\core\indexes\base.py", line 2646, in get_loc
    return self._engine.get_loc(key)

  File "pandas\_libs\index.pyx", line 111, in pandas._libs.index.IndexEngine.get_loc

  File "pandas\_libs\index.pyx", line 138, in pandas._libs.index.IndexEngine.get_loc

  File "pandas\_libs\hashtable_class_helper.pxi", line 1619, in pandas._libs.hashtable.PyObjectHashTable.get_item

  File "pandas\_libs\hashtable_class_helper.pxi", line 1627, in pandas._libs.hashtable.PyObjectHashTable.get_item

KeyError: 'Close'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

  File "C:/Users/Sayed/PycharmProjects/project/Technical Analysis.py", line 74, in <module>
    x = Data()

  File "C:/Users/Sayed/PycharmProjects/project/Technical Analysis.py", line 10, in __init__
    self.Close = self.df['Close'].astype(float)

  File "C:\Users\Sayed\miniconda3\lib\site-packages\pandas\core\frame.py", line 2800, in __getitem__
    indexer = self.columns.get_loc(key)

  File "C:\Users\Sayed\miniconda3\lib\site-packages\pandas\core\indexes\base.py", line 2648, in get_loc
    return self._engine.get_loc(self._maybe_cast_indexer(key))

  File "pandas\_libs\index.pyx", line 111, in pandas._libs.index.IndexEngine.get_loc

  File "pandas\_libs\index.pyx", line 138, in pandas._libs.index.IndexEngine.get_loc

  File "pandas\_libs\hashtable_class_helper.pxi", line 1619, in pandas._libs.hashtable.PyObjectHashTable.get_item

  File "pandas\_libs\hashtable_class_helper.pxi", line 1627, in pandas._libs.hashtable.PyObjectHashTable.get_item

KeyError: 'Close'

Answer 1

You have several problems that have been quickly mentioned by Serge in his comments.

You try to define self.Close as self.Close = self.df['Close'].astype(float)

but self.df is initialized as an empty dataframe so there is no column named 'Close' .

You then proceed to define:

    def dema(self):
        self.df['DEMA'] = talib.DEMA(self.Close, timeperiod=30)
        return self.dfs.append(self.df['dema'])

    def ema(self):
        self.df['EMA'] = talib.EMA(self.Close, timeperiod=30)
        return self.dfs.append(self.df['ema'])

    def ma(self):
        self.df['MA'] = talib.MA(self.Close, timeperiod=30, matype=0)
        return self.dfs.append(self.df['ma'])

In these three cases you define a column using uppercase letters (eg 'MA' )

But then try to append a column with a lowercase name (eg self.dfs.append(self.df['ma']) )

Answer 2

What do you want to accomplish with self.Close? Do you want a DataFrame with only the column close?

If yes, try self_close instead of self.close.