OverflowError: Python int too large to convert to C long in python though date are defined as datetime64[ns] type object

Question

Getting overflow error while setting the frequency

df_train.index.freq = 'W'.

Error:

 OverflowError Traceback (most recent call last) <ipython-input-37-bbb3114abe9d> in <module> ----> 1 df_train.index.freq = 'W' /opt/omniai/software/Miniconda/lib/python3.6/site-packages/pandas/core/indexes/extension.py in fset(self, value) 64 65 def fset(self, value): ---> 66 setattr(self._data, name, value) 67 68 fget.__name__ = name /opt/omniai/software/Miniconda/lib/python3.6/site-packages/pandas/core/arrays/datetimelike.py in freq(self, value) 978 if value is not None: 979 value = frequencies.to_offset(value) --> 980 self._validate_frequency(self, value) 981 982 self._freq = value /opt/omniai/software/Miniconda/lib/python3.6/site-packages/pandas/core/arrays/datetimelike.py in _validate_frequency(cls, index, freq, **kwargs) 1039 try: 1040 on_freq = cls._generate_range( -> 1041 start=index[0], end=None, periods=len(index), freq=freq, **kwargs 1042 ) 1043 if not np.array_equal(index.asi8, on_freq.asi8): /opt/omniai/software/Miniconda/lib/python3.6/site-packages/pandas/core/arrays/datetimes.py in _generate_range(cls, start, end, periods, freq, tz, normalize, ambiguous, nonexistent, closed) 406 end = end.tz_localize(None) 407 # TODO: consider re-implementing _cached_range; GH#17914 --> 408 values, _tz = generate_regular_range(start, end, periods, freq) 409 index = cls._simple_new(values, freq=freq, dtype=tz_to_dtype(_tz)) 410 /opt/omniai/software/Miniconda/lib/python3.6/site-packages/pandas/core/arrays/_ranges.py in generate_regular_range(start, end, periods, freq) 79 xdr = generate_range(start=start, end=end, periods=periods, offset=freq) 80 ---> 81 values = np.array([x.value for x in xdr], dtype=np.int64) 82 83 return values, tz /opt/omniai/software/Miniconda/lib/python3.6/site-packages/pandas/core/arrays/_ranges.py in <listcomp>(.0) 79 xdr = generate_range(start=start, end=end, periods=periods, offset=freq) 80 ---> 81 values = np.array([x.value for x in xdr], dtype=np.int64) 82 83 return values, tz /opt/omniai/software/Miniconda/lib/python3.6/site-packages/pandas/tseries/offsets.py in generate_range(start, end, periods, offset) 2765 2766 if end is None: -> 2767 end = start + (periods - 1) * offset 2768 2769 if start is None: pandas/_libs/tslibs/offsets.pyx in >pandas._libs.tslibs.offsets.BaseOffset.__radd__() pandas/_libs/tslibs/offsets.pyx in >pandas._libs.tslibs.offsets._BaseOffset.__add__() /opt/omniai/software/Miniconda/lib/python3.6/site-packages/pandas/tseries/offsets.py in wrapper(self, other) 109 other = other.tz_localize(None) 110 --> 111 result = func(self, other) 112 113 if self._adjust_dst: /opt/omniai/software/Miniconda/lib/python3.6/site-packages/pandas/tseries/offsets.py in apply(self, other) 1577 k -= 1 1578 -> 1579 return other + timedelta(weeks=k) 1580 1581 @apply_index_wraps pandas/_libs/tslibs/c_timestamp.pyx in >pandas._libs.tslibs.c_timestamp._Timestamp.__add__() OverflowError: Python int too large to convert to C long

Answer 1

You can try reindexing your dataframe with the appropriate number of periods

df = df.set_index(pd.date_range('2010-02-05', periods=100, freq='W'))