计算熊猫中一列中单个细胞的线性回归的问题

Question

I'm trying to implement a linear regression calculation for individual empty cells based on previous data in a column. Since I do not understand how to use the python libraries, I wrote the whole calculation in steps.

This is my dataframe:

index   value    delta
-52       0      42517
-51       0      42524
-50      216     42531
-49      345     42538
-48      237     42545
...
 -2      367     42862
 -1      310     42869
  0      226     42876
  1      NaN     42883
  2      NaN     42890
...
 49      NaN     43213
 50      NaN     43220
 51      NaN     43227
 52      NaN     43234

Values where index = 0 and lower are always 52 . Above, there may be a different number, but it is known to me beforehand, in this example their 52 . Unknown values always start where index = 1 .

For a single value, I'm counting so (here i count for value in column delta dd = 42883 ):

x = dftest['delta']
y = dftest['value']
x_mean= np.mean(x)
y_mean = np.mean(y)
x_std = np.std(x)
y_std = np.std(y)
corr = np.corrcoef(y, x)[1,0]
slope = corr * y_std / x_std
intercept = y_mean - slope * x_mean
n_vl = intercept + slope * dd

So he calculates, but I do not understand how to write a loop, so that he does this for all empty cells (beginning with index = 1 ), while taking into account the previously calculated value.

I tried to use the code that is in the first response here and change it, but it does not work.

Below part of the code:

vl = dftest['value'].values
delta =  dftest['delta'].values
for index in range(0, vl.shape[0]):
    if np.isnan(vl[index]):
        x = delta.take(range(index-52,index+1),mode='wrap')
        y = vl.take(range(index-52,index+1),mode='wrap')
        y1 = np.nanmean(vl.take(range(index-52,index+1),mode='wrap'))
        y2 = np.nanstd(vl.take(range(index-52,index+1),mode='wrap'))
        x1 = np.nanmean(delta.take(range(index-52,index+1),mode='wrap'))
        x2 = np.nanstd(delta.take(range(index-52,index+1),mode='wrap'))
        corr = np.corrcoef(y, x)[1,0] 
        slope = corr * y2 / x2
        intercept = y1 - slope * x1
        n_vl = intercept + slope * dd
print (y)
print (x)        
print (y1)
print (y2)
print (x1)
print (x2)
print (corr)
print (slope)
print (intercept)
print (n_vl)

But it takes a value below the index = 0 , not above. I do not know how to change this and how to write it so that it counts for every empty cell.

That's what I get at the output for one value (from my code with a loop).

[ 226.   nan   nan   nan   nan   nan   nan   nan   nan   nan   nan   nan
nan   nan   nan   nan   nan   nan   nan   nan   nan   nan   nan   nan
nan   nan   nan   nan   nan   nan   nan   nan   nan   nan   nan   nan
nan   nan   nan   nan   nan   nan   nan   nan   nan   nan   nan   nan
nan   nan   nan   nan   nan]
[42876 42883 42890 42897 42904 42911 42918 42925 42932 42939 42946 42953
 42960 42967 42974 42981 42988 42995 43002 43009 43016 43023 43030 43037
 43044 43051 43058 43065 43072 43079 43086 43093 43100 43107 43108 43115
 43122 43129 43136 43143 43150 43157 43164 43171 43178 43185 43192 43199
 43206 43213 43220 43227 43234]
226.0
0.0
43055.8490566
104.701263481
nan
nan
nan
nan

I have been stuck on this for a long time and can not move forward, I really need help.

Answer 1

Just do

dftest['value'].fillna(52)

which will fill all the NaN 's in the value column with number 52 . If you need to be extra sure to fill NaN 's only if index <= 0 (in other words you expect NaN 's in value column for index > 0 ) then do:

dftest.loc[index <= 0, 'value'].fillna(52)

Remember, that if you feel you need to use loops in Pandas, you most likely doing it wrong.

Answer 2

So, I decided to fill the empty cells in the column using linear regression based on known data.

import statsmodels.formula.api as smf
#Here I choose the known data and fit the model
smresults = smf.ols('value ~ delta', df.iloc[:53]).fit()
smresults.summary()
#Here I fill empty cells using the model
df.value[53:] = smresults.predict(df.iloc[53:])

This is the best solution I managed to get.