查找numpy.corr（）的非线性函数的相关性

Question

I wrote a program that reads a CSV file and computes the correlation between the two columns. The problem is the standard method of finding correlation does not work on curves and other non-linear functions. Is there another function or an easy way to modify the data to determine correlation? Below is my code so far, the csv input, and the current output.

def findCorrelation(csvFileName):
    data = pd.read_csv(csvFileName)
    data = data.values
    df = pd.DataFrame(data=data)
    npList = np.asarray(df)
    np2 = npList.astype(float)
    df2 = pd.DataFrame(data=np2)
    corr = df2.corr()
    corr = corr.values
    return corr[0][1]

def correlationMeaning(corr):
    if corr == 1:
        return ['perfect', 'positive', str(corr)]
    elif corr > 0.9:
        return ['high', 'positive', str(corr)]
    elif corr > 0.5:
        return ['medium', 'positive', str(corr)]
    elif corr > 0.1:
        return ['low', 'positive', str(corr)]
    elif corr > -0.1:
        return ['no', str(corr)]
    elif corr > -0.5:
        return ['low', 'negative', str(corr)]
    elif corr > -0.9:
        return ['medium', 'negative', str(corr)]
    elif corr > -1:
        return ['high', 'negative', str(corr)]
    elif corr == -1:
        return ['perfect', 'negative', str(corr)]
    else:
        return ['error']

print correlationMeaning(findCorrelation('CurveData.csv'))

CSV input:

Temp,Sales
30,50
34,52
38,54
42,56
46,58
50,60
54,62
58,62
62,60
66,58
70,56
74,54
78,52
82,50

Output:

['no', '0.0']

Answer 1

GENERAL ANSWER

The correlation coefficient is, by definition, a linear fit. What I think you want is some manner of curve-fitting. The problem is that you have to experiment, either by hand or programmatically, to find a good fit.

Also, these do not provide a direct cognate to the correlation coefficient, although the least-squares error can be easily harnessed to this purpose.

SPECIFIC APPLICATION

The given case is a simple "vee" shape; you need a non-linear transformation on your independent variable (Temp) to get a nice fit: X <= abs(X-56) . Now you have a perfect correlation.

If you want a program to experiment with various fits and derive the best one for each arbitrary data set, you'll have to program the exterior shell yourself, I'm afraid. However, there are a number of packages (such as SciKit) which provide function to optimize a set of equations with a given error function. If you want to tackle the larger project, you might want to research those facilities.

In the meantime, perhaps a simple plotting function would help you narrow the field for your specific needs?

Answer 2

Try the correlation method using element-wise; go over all the elements of the curves and find the correlation values for each pair. Then, you can average the values of the correlation to one value which will indicates if you are in High, Medium, Low or No correlation.