卡方用于多项式拟合的最佳顺序

Question

I have the following code, in which DGauss is a function that generates the expected values. The two arrays, on the other hand, allow me to generate a distribution, that I take as observed values. The code, based on the observed values, extracts a polynomial (for the moment of the seventh degree) that describes its trend.

import matplotlib.pyplot as plt
import numpy as np
from scipy.optimize import curve_fit

def DGauss(x,I1,I2,sigma1,sigma2):
    return I1*np.exp(-x*x/(2*sigma1*sigma1)) + I2*np.exp(-x*x/(2*sigma2*sigma2))

Pos = np.array([3.28, 3.13, 3.08, 3.03, 2.98, 2.93, 2.88, 2.83, 2.78, 2.73, 2.68,
       2.63, 2.58, 2.53, 2.48, 2.43, 2.38, 2.33, 2.28, 2.23, 2.18, 2.13,
       2.08, 2.03, 1.98, 1.93, 1.88, 1.83, 1.78, 1.73, 1.68, 1.63, 1.58,
       1.53, 1.48, 1.43, 1.38, 1.33, 1.28, 1.23, 1.18, 1.13, 1.08, 1.03,
       0.98, 0.93, 0.88, 0.83, 0.78, 0.73, 0.68, 0.63, 0.58, 0.53, 0.48,
       0.43, 0.38, 0.33, 0.28, 0.23, 0.18, 0.13, 0.08, 0.03])

Val = np.array([0.00986279, 0.01529543, 0.0242624 , 0.0287456 , 0.03238484,
       0.03285927, 0.03945234, 0.04615091, 0.05701618, 0.0637672 ,
       0.07194268, 0.07763934, 0.08565687, 0.09615262, 0.1043281 ,
       0.11350606, 0.1199406 , 0.1260062 , 0.14093328, 0.15079665,
       0.16651464, 0.18065023, 0.1938894 , 0.2047541 , 0.21794024,
       0.22806706, 0.23793043, 0.25164404, 0.2635118 , 0.28075974,
       0.29568682, 0.30871501, 0.3311846 , 0.34648062, 0.36984661,
       0.38540666, 0.40618835, 0.4283945 , 0.45002014, 0.48303911,
       0.50746062, 0.53167057, 0.5548792 , 0.57835128, 0.60256181,
       0.62566436, 0.65704847, 0.68289386, 0.71332794, 0.73258027,
       0.769608  , 0.78769989, 0.81407275, 0.83358852, 0.85210239,
       0.87109068, 0.89456217, 0.91618782, 0.93760247, 0.95680234,
       0.96919757, 0.9783219 , 0.98486193, 0.9931429 ])

f = np.linspace(-9,9,2*len(Pos))    

plt.errorbar(Pos, Val, xerr=0.02, yerr=2.7e-3, fmt='o')
popt, pcov = curve_fit(DGauss, Pos, Val)
plt.plot(xfull, DGauss(f, *popt), '--', label='Double Gauss')

x = Pos
y = Val
#z, w = np.polyfit(x, y, 7, full=False, cov=True)
p = np.poly1d(z)
u = np.array(p)
xp = np.linspace(1, 6, 100)
_ = plt.plot(xp, p(xp), '-', color='darkviolet')
x = symbols('x')
list = u[::-1]
poly = sum(S("{:7.3f}".format(v))*x**i for i, v in enumerate(list))
eq_latex = sympy.printing.latex(poly)
print(eq_latex)

#LOOP SUGGESTED BY @Fourier
dof = [1,2,3,4,5,6,7,8,9,10]
for i in dof:
    z = np.polyfit(x, y, i, full=False, cov=True)
    chi = np.sum((np.polyval(z, x) - y) ** 2)
    chinorm = chi/i
    plt.plot(chinorm)

What I would like to do now is to make a fit by varying the order of the polynomial to figure out which is the minimum order I need to have a good fit and not exceed the number of free parameters. In particular, I would like to make this fit with different orders and plot the chi-squared, which must be normalized with respect to the number of degrees of freedom. Could someone help me kindly?

Thanks!

Answer 1

Based on the posted code this should work for your purpose:

chiSquares = []

dofs = 10
for i in np.arange(1,dofs+1):
    z = np.polyfit(x, y, i, full=False, cov=False)
    chi = np.sum((np.polyval(z, x) - y) ** 2) / np.std(y) #ideally you should divide this using an error for Val array
    chinorm = chi/i
    chiSquares.append(chinorm)

plt.plot(np.arange(1,dofs+1),chiSquares)

If not evident from the plot, you can further use the F-test to check how much dof is really needed:

n = len(y)
for d, (rss1,rss2) in enumerate(zip(chiSquares,chiSquares[1:])):
    p1 = d + 1
    p2 = d + 2        
    F = (rss1-rss2/(p2-p1)) / (rss2/(n-p2))
    p = 1.0 - scipy.stats.f.cdf(F,p1,p2)
    print 'F-stats: {:.3f},  p-value: {:.5f}'.format(F,p)