Scipy curve_fit silently fails only for very specific x-values

Question

I have a larger piece of code that fits functions to data at its core. The data and the functions to be fitted are dynamic. Recently I appended an additional data point to the whole system and now curve_fit always returns the initial guess (or something too close to it), no matter how I chose it. This happens for very different y-values and x-values (ten sets of the former, two sets of the latter).

I know that choosing the starting values is important, but I never had trouble with using the default before (my functions are generally simple) and can revert to the state where it just works by uncommenting the new code that adds the additional data point. Now one would think that obviously the new code is the problem, but there are quite some steps between the new addition and actually feeding the data to curve_fit . I have already checked that the type of the input to curve_fit is the same: np.ndarray , just one element longer in the problematic case.

However, while creating a MWE, I noticed that it is only the exact x-array that leads to the problem. It goes away entirely when I copy the printed x-vector of the main program in my MWE instead of the internal representation. Hence I can only show the problem with an external file: local_z.npy [150kB]

The MWE:

import numpy as np
from scipy.optimize import curve_fit

values = np.array([[1.37712972, 1.58475346, 1.78578759, 1.9843099,  1.73393093],
                   [-0.0155715,  -0.01534987, -0.00910744, -0.00189728, -1.73393093],
                   [1.23613934, 0.76894505, 0.18876817, 0.06376843, 1.1637315 ],
                   [0.8535248,  0.53093829, 0.13033993, 0.04403058, 0.80352895],
                   [0.51505805, 0.32039379, 0.0786534,  0.02657018, 0.48488813]])
heights = np.array([ 22.110203,  65.49054,  110.321526, 156.54034,  166.59094])
local_z = np.load('local_z.npy')
print('difference in heights', local_z - heights)

def func(z, a0, a1):
    return a0 + a1*z

for v in values:
    popt_non_working = curve_fit(func, local_z, v)[0]
    print('not working: ', popt_non_working)
    popt_working = curve_fit(func, heights, v)[0]
    print('working: ', popt_working)

My output with Python 2.7.6, numpy 1.14.1, and scipy 1.0.0:

$ python auxiliary/SUBLIME_fit_test.py  
('difference in heights', array([-2.10693358e-07, -4.49218746e-07, -4.26269537e-07,  4.23828126e-06, 2.38281251e-06]))
/home/csag5117/venv/local/lib/python2.7/site-packages/scipy/optimize/minpack.py:785: OptimizeWarning: Covariance of the parameters could not be estimated category=OptimizeWarning)
('not working: ', array([1., 1.]))
('working: ', array([1.35420488, 0.00325281]))
('not working: ', array([1., 1.]))
('working: ', array([ 0.38896878, -0.00714073]))
('not working: ', array([1., 1.]))
('working: ', array([ 1.06301278, -0.00363439]))
('not working: ', array([1., 1.]))
('working: ', array([ 0.73398503, -0.00250946]))
('not working: ', array([1., 1.]))
('working: ', array([ 0.442922  , -0.00151433]))

As you can see, the version where I use heights as x-values works as expected (returns the fitting parameters), whereas the version where I use the stored `local_z' does not, even though the difference between the two arrays is very small. I only show multiple y-values to show that this is not some one in a million failure that could be fixed with proper starting values. This is also only one example, I also have one with more data points (24 instead of 5) with the same behavior.

For completeness sake, the code block I added (when I turn this off everything works). Interestingly enough, leaving out the last value in local_z (which is the one being added by the code block) by using local_z[:-1] in the MWE does not fix the problem.

zi_minus_dd -= 1
zf_long = np.append(out.zf, np.squeeze(data.zf[t])[z_mask_full[-1] + 1])
u_zi = np.interp(zi_minus_dd, zf_long,
    np.append(out.u, np.squeeze(data.u[t])[z_mask_full[-1] + 1]))
v_zi = np.interp(zi_minus_dd, zf_long,
    np.append(out.v, np.squeeze(data.v[t])[z_mask_full[-1] + 1]))
th_zi = np.interp(zi_minus_dd, zf_long,
    np.append(out.th, np.squeeze(data.th[t])[z_mask_full[-1] + 1]))

zh_long = np.append(out.zh, np.squeeze(data.zh[t])[z_mask_full[-1] + 1])
uw_zi = np.interp(zi_minus_dd, zf_long,
    np.append(out.uw_raw, np.squeeze(data.uw[t])[z_mask_full[-1] + 1]))
vw_zi = np.interp(zi_minus_dd, zf_long,
    np.append(out.vw_raw, np.squeeze(data.vw[t])[z_mask_full[-1] + 1]))
tke_zi = np.interp(zi_minus_dd, zf_long,
    np.append(out.tke, np.squeeze(data.TKE[t])[z_mask_full[-1] + 1]))

out.zf = np.append(out.zf, zi_minus_dd)
out.u = np.append(out.u, u_zi)
out.v = np.append(out.v, u_zi)
out.th = np.append(out.th, u_zi)

out.zh = np.append(out.zh, zi_minus_dd)
out.uw_raw = np.append(out.uw_raw, u_zi)
out.vw_raw = np.append(out.vw_raw, u_zi)
out.tke = np.append(out.tke, u_zi)

out.zf and out.zh are the vectors that are later made into local_z . The whole code is rather large and also depends on netCDF files (the data in the snippet above). I already asked about it here , but that was for the working code.

I'm pretty much stumped and have no idea how I could fix this or even continue to debug. Is it possible that there is some problem with copy vs deepcopy or something like that? Although I wonder how that would transfer to the MWE via the stored array...

Answer 1

Tracking this down was kind of fun. :-)

It's not the values, but their types . It's a precision issue: heights , which works, is float64, local_z, which doesn't work, is only float32.

We have

In [70]: heights
Out[70]: array([ 22.110203,  65.49054 , 110.321526, 156.54034 , 166.59094 ])

In [71]: heights.dtype
Out[71]: dtype('float64')

In [72]: curve_fit(func, heights, v)[0]
Out[72]: array([1.35420488, 0.00325281])

and

In [73]: local_z
Out[73]: 
array([ 22.110205,  65.49054 , 110.321526, 156.54034 , 166.59094 ],
      dtype=float32)

In [74]: curve_fit(func, local_z, v)[0]
C:\Python\lib\site-packages\scipy\optimize\minpack.py:794: OptimizeWarning: Covariance of the parameters could not be estimated
  category=OptimizeWarning)
Out[74]: array([1., 1.])

But if we want to, we can make local_z work:

In [75]: curve_fit(func, local_z.astype(np.float64), v)[0]
Out[75]: array([1.35420488, 0.00325281])

or heights fail:

In [76]: curve_fit(func, heights.astype(np.float32), v)[0]
C:\Python\lib\site-packages\scipy\optimize\minpack.py:794: OptimizeWarning: Covariance of the parameters could not be estimated
  category=OptimizeWarning)
Out[76]: array([1., 1.])