How to calculate sums of squares in Python?

Question

First, is the formula TSS = ESS + RSS always correct? Even for an exponential model? If it is, I just do not understand where am I wrong.

I have 2 arrays of x and y values, where y depends on x.

x = np.array([1.5, 2.1, 2.4, 2.7, 3.2, 3.4, 3.6, 3.7, 4.0, 4.5, 5.1, 5.6])
y = np.array([0.6, 1.2, 1.3, 1.4, 1.45, 1.5, 1.6, 1.8, 1.9, 1.95, 2.1, 2.2])

I have a function that determines coefficients a and b and returns an equation of linear regression (or just a and b if needed)

def Linear(x, y, getAB = False):
    AVG_X = np.average(x)
    AVG_Y = np.average(y)
    DISP_X = np.var(x)
    DISP_Y = np.var(y)
    STD_X = np.std(x)
    STD_Y = np.std(y)

    AVG_prod = np.average(x*y)
    cov = AVG_prod - (AVG_X*AVG_Y)

    b = cov/DISP_X
    a = AVG_Y - b*AVG_X

    if getAB:
        return a, b

    return lambda X: a + b*X

I have a function that determines coefficients a and b and returns an equation of EXPONENTIAL regression

def Exponential(x, y, getAB = False):
    LOG_Y_array = [math.log(value) for value in y]

    A, B = Linear(x, LOG_Y_array, getAB = True)

    a = math.exp(A)
    b = math.exp(B)

    if getAB:
        return a, b

    return lambda X: a * (b**X)

I created the array of calculated y values based of exponential model

Exponential_Prediction = Exponential(x, y)
Exponential_Prediction_y = [Exponential_Prediction(value) for value in x]

And finally, that is how I calculate TSS, ESS and RSS

TSS = np.sum((y - np.average(y))**2)
ESS_Exp = np.sum((Exponential_Prediction_y - np.average(y))**2)
RSS_Exp = np.sum((y-Exponential_Prediction_y)**2)

That is all pretty clear, except the output of this

print(str(TSS) + " = " + str(ESS_Exp) + " + " + str(RSS_Exp))

is 2.18166666667 = 2.75523753042 + 0.432362713806

I do not understand how ESS could be more than TSS

Answer 1

You're missing a term that is zero when you're using linear regression, since you're not, you have to add it. In the link that Vince commented, you can see that TSS = ESS + RSS + 2*sum((y - yhat)*(yhat - ybar)).

You need to include that extra term in order for it to add up:

extra_term = 2 * np.sum((y - Exponential_Prediction_y) * (Exponential_Prediction_y - y.mean())) 
print(str(TSS) + " = " + str(ESS_Exp) + " + " + str(RSS_Exp) + " + " + str(extra_term))