我不确定我的代码有什么问题..（线性/多项式回归）

Question

I have a data set (csv file) with three seperate columns. Column 0 is the signal time, Column 1 is the frequency, and Column 2 is the intensity. The is alot of noise in the data that can be sorted though by finding the variance of each signal frequency. If it is <2332 then it is the right frequency. Hence, this would be the data I would want to calculate Linear/Poly regression on. ps I have to calc linear manually:(. The nested for loop decision structure I have isn't currently working. Any solutions would be helpful! thanks

data = csv.reader(file1)
sort = sorted(data, key=(operator.itemgetter(1))) #sorted by the frequencies
for row in sort:
x.append(float(row[0]))
y.append(float(row[2]))
frequencies.append(float(row[1]))

for i in range(499) : 
freq_dict.update({ frequencies[i] : [x[i], y[i]] })

for key in freq_dict.items(): 
   for row in sort : 
       if key == float(row[1]):
           a.append(float(row[1]))
           b.append(float(row[2]))
           c.append(float(row[0]))
       else :
           num = np.var(a)
           if num < 2332.0: 
               linearRegression(c, b, linear)
               print('yo')
               polyRegression(c, b, d, linear, py)
               mplot.plot(linear, py)
           else: 
               a = [] 
               b = [] 
               c = []

variances of 2332 or less are the frequencies I need

variances of 2332 or less are the frequencies I need

I used range of 499 because that is the length of my data set. Also, I tried to clear the lists (a,b,c) if the frequency wasn't correct.

Answer 1

There are several issues I see going on. I am unsure why you sort your data, if you all ready know the exact values you are looking for. I am unsure why you split up the data into separate variables as well. The double "for" loops means that you are repeating everything in "sort" for every single key in freq_dict. Not sure if that was your intention to repeat all those values multiple times. Also, freq_dict.items() produces tuples (key,value pairs), so your "key" is a tuple, hence "key" will never equal a float. Anyway, here is an attempt to re-write some code.

import csv, numpy
import matplotlib.pyplot as plt
from scipy import stats

data   = csv.reader(file1)                    #Read file. 
f_data = filter(lambda (x,f,y):f<2332.0,data) #Filter data to condition. 
x,_,y  = list(zip(*f_data))                   #Split data down column. 

#Standard linear stats function. 
slope,intercept,r_value,p_value,std_err = stats.linregress(x,y)

#Plot the data and the fit line. 
plt.scatter(x,y)
plt.plot(x,numpy.array(x)*slope+intercept)
plt.show()

Answer 2

A more similar solution was using the corrcoef of the list. But in similar style it was as follows:

for key, value in freq_dict.items(): #1487
for row in sort:   #when row -> goes to a new freq it calculates corrcoef of an empty list.
    if key == float(row[1]): #1487
        a.append(float(row[2]))
        b.append(float(row[0])) 
    elif key != float(row[1]): 
        if a: 
            num = np.corrcoef(b, a)[0,1]
            if (num < somenumber).any(): 
                do stuff
        a = [] #clear the lists and reset number
        b = [] 
        num = 0