During handling of the above exception, another exception occurred: 'Date'

Question

Background -

I have a DataFrame, that has two columns (Date and Precipitation) in the following format -

precipitation_df = pd.DataFrame('2016-09-05', NaN), ('2016-09-06', NaN) etc

Objective -
I would like to remove the rows containing NaN values for the date range 2016-08-24 - 2017-08-24 , however firstly wanted to analysis what dates were impacted in this range. I therefore decided to create a new DataFrame, using this code, with all the NaN values in the range I am interested in -

start_date = '2016-08-23'
end_date = '2017-08-23'

nan_values_df = (precipitation_df['Date'] > start_date) & (df['Date'] <= end_date)

Issues -
When I run this code, I get a huge error, which references 'Date':

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
~/anaconda3/lib/python3.7/site-packages/pandas/core/indexes/base.py in get_loc(self, key, method, tolerance)
   2645             try:
-> 2646                 return self._engine.get_loc(key)
   2647             except KeyError:

pandas/_libs/index.pyx in pandas._libs.index.IndexEngine.get_loc()

pandas/_libs/index.pyx in pandas._libs.index.IndexEngine.get_loc()

pandas/_libs/hashtable_class_helper.pxi in pandas._libs.hashtable.PyObjectHashTable.get_item()

pandas/_libs/hashtable_class_helper.pxi in pandas._libs.hashtable.PyObjectHashTable.get_item()

KeyError: 'Date'

During handling of the above exception, another exception occurred:

KeyError                                  Traceback (most recent call last)
<ipython-input-131-146522d4b445> in <module>
      2 end_date = '2017-08-23'
      3 
----> 4 nan_values_df = (precipitation_df['Date'] > start_date) & (df['Date'] <= end_date)

~/anaconda3/lib/python3.7/site-packages/pandas/core/frame.py in __getitem__(self, key)
   2798             if self.columns.nlevels > 1:
   2799                 return self._getitem_multilevel(key)
-> 2800             indexer = self.columns.get_loc(key)
   2801             if is_integer(indexer):
   2802                 indexer = [indexer]

~/anaconda3/lib/python3.7/site-packages/pandas/core/indexes/base.py in get_loc(self, key, method, tolerance)
   2646                 return self._engine.get_loc(key)
   2647             except KeyError:
-> 2648                 return self._engine.get_loc(self._maybe_cast_indexer(key))
   2649         indexer = self.get_indexer([key], method=method, tolerance=tolerance)
   2650         if indexer.ndim > 1 or indexer.size > 1:

pandas/_libs/index.pyx in pandas._libs.index.IndexEngine.get_loc()

pandas/_libs/index.pyx in pandas._libs.index.IndexEngine.get_loc()

pandas/_libs/hashtable_class_helper.pxi in pandas._libs.hashtable.PyObjectHashTable.get_item()

pandas/_libs/hashtable_class_helper.pxi in pandas._libs.hashtable.PyObjectHashTable.get_item()

KeyError: 'Date'

I have a feeling it might be something to do with the date format, but I am stumped and wondered if anyone could point me in the right direction?

Answer 1

Your instruction refers to df instead of precipitation_df .

But to make your code shorter, more readable and less prone to such errors, change this instruction to:

nan_values_df = precipitation_df.query('@start_date < Date <= @end_date')

Answer 2

Think you may have a typo:

You wrote:

nan_values_df = (precipitation_df['Date'] > start_date) & (df['Date'] <= end_date)

But you should be referencing the same DataFrame:

nan_values_df = (precipitation_df['Date'] > start_date) & (precipitation_df['Date'] <= end_date)