Airflow error with pandas: AttributeError: 'Pendulum' object has no attribute 'nanosecond'

Question

I have a pandas.DataFrame df with df.index which yeilds something like this:

DatetimeIndex(['2014-10-06 00:55:11.357899904',
               '2014-10-06 00:56:39.046799898',
               '2014-10-06 00:56:39.057499886',
               '2014-10-06 00:56:40.684299946',
               '2014-10-06 00:56:41.115299940',
               '2014-10-06 01:03:52.764300108',
               '2014-10-06 01:21:18.448499918',
               '2014-10-06 01:21:18.457200050',
               '2014-10-06 01:21:18.584199905',
               '2014-10-06 01:21:18.594700098',
               ...
               '2014-11-05 00:25:47.996000051',
               '2014-11-05 00:56:45.081799984',
               '2014-11-05 00:56:45.096899986',
               '2014-11-05 05:50:57.639699936',
               '2014-11-05 06:08:56.365000010',
               '2014-11-05 06:11:20.519099950',
               '2014-11-05 06:15:03.470400095',
               '2014-11-05 06:15:03.981600046',
               '2014-11-05 06:25:31.514300108',
               '2014-11-05 06:25:59.310400009'],
              dtype='datetime64[ns]', name='time', length=1000, freq=None)

I am running a DAG on airflow, which stops at the following line df.loc[start_date:end_date] , saying that:

AttributeError: 'Pendulum' object has no attribute 'nanosecond'

I cannot reproduce the error without running the code in Airflow. The same code runs just fine without Airflow.

The start_date is the Airflow macro execution_date and end_date is the next_execution_date .

I guess the issues is to do with the date-time dtype of the df not being compatable with the ones from the start_date & end_date , but I have no idea how to address it.

I tried removing time zones, changing the dtype but nothing worked.

Answer 1

After some searching, I found the source of the problem and a solution.

the problem

The issue is caused by the two macros passed down from Airflow:

• start_date , which is the execution_date macro

• end_date , which is the next_execution_date macro

The types of them are pendulum.datetime , and not datetime.datetime , as the Airflow documentation says. This causes the clash with pandas.DataFrame .

pandas and pendulum currently don't work well together and the problem is well described in this StackOverflow asnwer.

the solution

The solution seesm to convery the start_date and end_date from pendulum.datetime to datetime.datetime .

For this I created this simple function, which converts from to string beofore converting to datetime.datetime . I am sure they are better ways to do it, but this was quite simple and safe, hence why I used it.

Here is the function itself:

def pendulum_to_datetime(pendulum_date):
    """
    Convert pendulum to datetime format.

    The conversion is done from pendulum -> string -> dateime.

    Args:
        pendulum_date (pendulum): The date you wish to convert.

    Returns:
        (datetime) The converted date.
    """
    fmt = '%Y-%m-%dT%H:%M:%S%z'
    string_date = pendulum_date.strftime(fmt)
    return datetime.strptime(string_date, fmt)