趋势的最佳契合线

Question

I have the following data

df = pd.DataFrame({ 
'region'  : ['a', 'a', 'a','a',' a','a','a', 's', 's','s','l','a','c','a', 'e','a','g', 'd','c','d','a','f','a','a','a'],
'month_number' : [5, 12, 3, 12, 3, 6,7,8,9,10,11,12,4,5,2,6,7,8,3, 4, 7, 6,7,8,8],
'score' : [2.5, 5, 3.5, 2.5, 5.5, 3.5,2,3.5,4,2,1.5,1,1.5,4,5.5,2,3,1,2,3.5,4,2,3.5,3,4]})

I want to calculate the mean of the score in a region and create its trend over the year, as last I want to have a line of best fit to see if the trend is rising or falling over time. (Not for predicted values, simply on the mean)

I filtered a region 'a':

filtered = df[(df['region'] == 'a')]

And created a trend:

filtered.groupby(['month_number','region']).mean()['score'].unstack().plot(figsize=(10,6))

This give the following:

Now I am stuck at the part how to fit the best line over the trend. My goal after all is to create a column with values of plus's or minus's indicating rising or falling trend in that region. If there is any other approach to this, I would like to hear it.

Answer 1

You can do it using seaborn 's regression plot regplot as following. The shaded region is the confidence interval.

import seaborn as sns
import pandas as pd

df = pd.DataFrame({ 
'region'  : ['a', 'a', 'a','a',' a','a','a', 's', 's','s','l','a','c','a', 'e','a','g', 'd','c','d','a','f','a','a','a'],
'month_number' : [5, 12, 3, 12, 3, 6,7,8,9,10,11,12,4,5,2,6,7,8,3, 4, 7, 6,7,8,8],
'score' : [2.5, 5, 3.5, 2.5, 5.5, 3.5,2,3.5,4,2,1.5,1,1.5,4,5.5,2,3,1,2,3.5,4,2,3.5,3,4]})

filtered = df[(df['region'] == 'a')]
df1 = filtered.groupby(['month_number','region']).mean()['score'].unstack()
sns.regplot(x=df1.index.tolist(), y=df1['a'], data=df1)

If you don't want the shaded confidence interval, you can use ci=0 as

sns.regplot(x=df1.index.tolist(), y=df1['a'], data=df1, ci=0)

Answer 2

If you want to just plot the straight line fit, use Seaborn.

However, if, you want to calculate the straight line fit for the data, use numpy.polyfit .

import numpy as np
f1 = filtered.groupby('month_number').mean().reset_index()
x = f1.month_number.values
y = f1.score.values
m, c = np.polyfit(x, y, 1)

You have calculated the slope and the y-intercept for your point.

You can calculate points above and below your positions as follows:

yHat = m*x + c
yError = y - yHat

For your new column, just use the error values:

f1['HiLo'] = [ ('+' if m else '-')  for m in yError>0]

You will get your plusses and minuses ..

month_number     score HiLo
           3  3.500000    +
           5  3.250000    -
           6  2.750000    -
           7  3.166667    +
           8  3.500000    +
          12  2.833333    -