将日期时间的 numpy 数组与时间戳进行比较

Question

I am not understanding why this code throws an error. Isn't '>' vectorized? I can see there is no problem with x_month_begin[0,0] > st_d comparison. Would appreciate insights and fix suggestions.

    import pandas as pd
    import numpy as np
    import datetime




    end_d = pd.to_datetime('23/02/2018', format="%d/%m/%Y")


    x_month_begin = pd.date_range(datetime.datetime(year=end_d.year-1, month=1, day=1), 
                                      datetime.datetime(year=end_d.year+1, month=12, day=1), freq='MS')

    # stacking with each row for each year        
    x_month_begin = np.vstack(np.split(x_month_begin, 3))
    # transposing for each column to be a year
    x_month_begin = np.transpose(x_month_begin)

    st_d = pd.to_datetime('01/2016', format="%m/%Y")

    x_month_begin > st_d

Answer 1

If you check the type of objects, the problem becomes clear.

1. x_month_begin : is a 3 dimensional numpy array ( x_month_begin.shape )
2. st_d : is a pandas timestamp variable

Both cannot be compared directly. For comparison, you can do something like this:

[y > st_d for x in x_month_begin for y in x ]

Answer 2

So you start off with a pandas structure:

In [133]: x_month_begin                                                         
Out[133]: 
DatetimeIndex(['2020-01-01', '2020-02-01', '2020-03-01', '2020-04-01',
               '2020-05-01', '2020-06-01', '2020-07-01', '2020-08-01',
               '2020-09-01', '2020-10-01', '2020-11-01', '2020-12-01'],
              dtype='datetime64[ns]', freq='MS')

the same thing as a numpy array:

In [134]: x_month_begin.values                                                  
Out[134]: 
array(['2020-01-01T00:00:00.000000000', '2020-02-01T00:00:00.000000000',
       '2020-03-01T00:00:00.000000000', '2020-04-01T00:00:00.000000000',
       '2020-05-01T00:00:00.000000000', '2020-06-01T00:00:00.000000000',
       '2020-07-01T00:00:00.000000000', '2020-08-01T00:00:00.000000000',
       '2020-09-01T00:00:00.000000000', '2020-10-01T00:00:00.000000000',
       '2020-11-01T00:00:00.000000000', '2020-12-01T00:00:00.000000000'],
      dtype='datetime64[ns]')

You manipulate that into a (n,3) array (I suspect this can be done more directly with a reshape and possible transpose):

In [135]: x_month_begin = np.vstack(np.split(x_month_begin, 3))                 
In [138]: x_month_begin = np.transpose(x_month_begin)                           
In [139]: x_month_begin                                                         
Out[139]: 
array([['2020-01-01T00:00:00.000000000', '2020-05-01T00:00:00.000000000',
        '2020-09-01T00:00:00.000000000'],
       ['2020-02-01T00:00:00.000000000', '2020-06-01T00:00:00.000000000',
        '2020-10-01T00:00:00.000000000'],
       ['2020-03-01T00:00:00.000000000', '2020-07-01T00:00:00.000000000',
        '2020-11-01T00:00:00.000000000'],
       ['2020-04-01T00:00:00.000000000', '2020-08-01T00:00:00.000000000',
        '2020-12-01T00:00:00.000000000']], dtype='datetime64[ns]')
In [140]: _.shape                                                               
Out[140]: (4, 3)

Any ways, now your comparison:

In [141]: st_d = pd.to_datetime('01/2016', format="%m/%Y")                      
In [142]: st_d                                                                  
Out[142]: Timestamp('2016-01-01 00:00:00')
In [143]: x_month_begin >st_d                                                   
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-143-30567662e59d> in <module>
----> 1 x_month_begin >st_d

pandas/_libs/tslibs/c_timestamp.pyx in pandas._libs.tslibs.c_timestamp._Timestamp.__richcmp__()

TypeError: '<' not supported between instances of 'Timestamp' and 'int'

numpy arrays can do < comparisons, but they have certain rules about what dtypes are compatible. (eg comparing strings and numbers doesn't work). In addition pandas plays its own games with dates and times, some formats are internal, some are compatible with the numpy datatime64 .

For example if we convert your timestamp to a numpy equivalent:

In [144]: st_d.to_numpy()                                                       
Out[144]: numpy.datetime64('2016-01-01T00:00:00.000000000')

the comparison works:

In [145]: x_month_begin>st_d.to_numpy()                                         
Out[145]: 
array([[ True,  True,  True],
       [ True,  True,  True],
       [ True,  True,  True],
       [ True,  True,  True]])

pandas is built on numpy , or at least uses numpy arrays to store its data. But none of the numpy code is pandas aware. If given a non-numpy object it will try, naively, to convert it, eg

In [146]: np.asarray(st_d)                                                      
Out[146]: array(Timestamp('2016-01-01 00:00:00'), dtype=object)

is different from Out[144] . [146] is the conversion that produces your error.

The original `DatetimeIndex` can be tested against the timestamp.  That's a 'pure' pandas operation.

In [152]: _133>st_d                                                             
Out[152]: 
array([ True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True])

_133.to_numpy().reshape(-1,4).T gives the x_month_begin array directly.