False ValueError with scipy.signal.savgol_filter

Question

I am confused. I have 21 files that have been generated by the same process and I am filtering them all using savitzky-golay filter with the same parameters. It works normally for some files, but at some point, I receive the ValueError: array must not contain infs or NaNs . The problem is, I checked the file and there aren't any infs or NaNs!

print "nan", df.isnull().sum()
print "inf", np.isinf(df).sum()

gives

nan x        0
T          0
std_T      0
sterr_T    0
dtype: int64
inf x        0
T          0
std_T      0
sterr_T    0
dtype: int64

So the problem may be in implementation of the filter? Could this result from for example the choice of window length or polyorder relative to the length or step of the data?

Complete traceback:

Traceback (most recent call last):

  File "<ipython-input-7-40b33049ef41>", line 1, in <module>
    runfile('D:/data/scripts/DailyProfiles_Gradients.py', wdir='D:/data/DFDP2/DFDP2B/DTS/DTS_scripts')

  File "C:\Users\me\AppData\Local\Continuum\Anaconda2\lib\site-packages\spyderlib\widgets\externalshell\sitecustomize.py", line 714, in runfile
    execfile(filename, namespace)

  File "C:\Users\me\AppData\Local\Continuum\Anaconda2\lib\site-packages\spyderlib\widgets\externalshell\sitecustomize.py", line 74, in execfile
    exec(compile(scripttext, filename, 'exec'), glob, loc)

  File "D:/data/scripts/DailyProfiles_Gradients.py", line 142, in <module>
    grad = gradient(y, x, scale,PO)

  File "D:/data/scripts/DailyProfiles_Gradients.py", line 76, in Tgradient
    smoothed=savgol_filter(list(x), scale, PO, deriv=1, delta=dy[0])

  File "C:\Users\me\AppData\Local\Continuum\Anaconda2\lib\site-packages\scipy\signal\_savitzky_golay.py", line 337, in savgol_filter
    coeffs = savgol_coeffs(window_length, polyorder, deriv=deriv, delta=delta)

  File "C:\Users\me\AppData\Local\Continuum\Anaconda2\lib\site-packages\scipy\signal\_savitzky_golay.py", line 140, in savgol_coeffs
    coeffs, _, _, _ = lstsq(A, y)

  File "C:\Users\me\AppData\Local\Continuum\Anaconda2\lib\site-packages\scipy\linalg\basic.py", line 822, in lstsq
    b1 = _asarray_validated(b, check_finite=check_finite)

  File "C:\Users\me\AppData\Local\Continuum\Anaconda2\lib\site-packages\scipy\_lib\_util.py", line 187, in _asarray_validated
    a = toarray(a)

  File "C:\Users\me\AppData\Local\Continuum\Anaconda2\lib\site-packages\numpy\lib\function_base.py", line 1033, in asarray_chkfinite
    "array must not contain infs or NaNs")

ValueError: array must not contain infs or NaNs

This problem is rather specific to the data and method and I have not been able to produce a minimum reproducible working example. I am not asking for fixing my code, I am just asking for some brainstorming: What aspects have I not checked yet that might be causing the error? What should the function parameters look like, other than that the window length must be an odd number greater than the polyorder?

I am grateful for the discussion that has arisen, it helped, eventually.

Answer 1

I can reproduce the error ValueError: array must not contain infs or NaNs if delta is extremely small (eg delta=1e-310 ). Check your code and data to ensure that the values that you pass for delta are reasonable.