Matlab curve fitting toolbox - wrong data fit?

Question

In matlab I have 2 data sets that consists of 192 points each and I am using the matlab curve fitting toolbox to find the correlation between the data. It does a polynomial fit for the data I get an r-square value. Till now there was no problem, but I was doing the same procedure for more data sets and I got the following fit which just looks wrong to me. The original data is a bit noisy, but not so noisy that the correlation would be so small.

I did not do any standardising or normalising to the data set before doing this fit. I just open cftool and then plot one data set on the x-axis, the other on the y-axis and then it gives me a fit.

My x-data for this particular plot is (it is a 1x192 vector):

   [0.00567884422104210 0.00569408830031418 0.00572138721599404 0.00569975708436488 0.00570545448583814 0.00569369374744166 0.00569563036622742 0.00566596177157741 0.00564522789669859 0.00564777221738393 0.00569035412216911 0.00567315592749138 0.00569878611862423 0.00569518382892348 0.00570283089572832 0.00567842496456661 0.00568067523788957 0.00566608538366334 0.00570454736884095 0.00566818166357483 0.00570961691287243 0.00572044020550668 0.00574088624152054 0.00566479613576671 0.00566608538366334 0.00566640778733622 0.00571469547537104 0.00566963384315990 0.00579483752389121 0.00553722841745620 0.00546670792516501 0.00573609142438767 0.00566173652214560 0.00566705270476430 0.00568400872896068 0.00568925305209833 0.00570380259930853 0.00569585871152685 0.00573213608623289 0.00571606342327985 0.00570972175158768 0.00571091687663606 0.00569708052924811 0.00568509554179675 0.00567202993133767 0.00566417423795219 0.00569597250745323 0.00567864418006962 0.00568891146251751 0.00569579324896427 0.00567684749344899 0.00569558715400373 0.00567844008671576 0.00570959789841987 0.00573349711019414 0.00570907582131189 0.00571094284081769 0.00571231953756342 0.00571123926053416 0.00568473432845418 0.00564484259733881 0.00566433279754403 0.00566439804473876 0.00568613333023755 0.00567631361064464 0.00567379185182118 0.00570342823629353 0.00568363772351148 0.00568505478199934 0.00568845947423812 0.00566499368455101 0.00568390381379168 0.00570081854849188 0.00571914776237673 0.00570766987394644 0.00571470680144348 0.00566915578039862 0.00565964601579385 0.00566302033316934 0.00566430371497644 0.00566961965465689 0.00567931476686888 0.00569099378053500 0.00568676388316038 0.00568351321709490 0.00569420010875536 0.00568090747310061 0.00567817959036125 0.00567496918909057 0.00568601151761792 0.00570068432296731 0.00568152083831520 0.00569698510782558 0.00567267156373908 0.00567738630382919 0.00567644084952437 0.00565230197496677 0.00568102525133482 0.00567619348815465 0.00567783704226933 0.00568029144278347 0.00570161890971965 0.00568436017206210 0.00569265664596718 0.00567127030453059 0.00567499736589668 0.00569610702265938 0.00569839910466992 0.00568783186020722 0.00567342243258319 0.00568817765560068 0.00569436802133702 0.00568154577259346 0.00569015913920067 0.00569374830185126 0.00567963282984794 0.00567171804390853 0.00567366726894102 0.00571257082933444 0.00568497453785619 0.00568885763847554 0.00568008265357695 0.00569387121280633 0.00566680261731652 0.00569677457803111 0.00570214269919954 0.00565244753071010 0.00564394435584691 0.00566157836051888 0.00566381845950213 0.00566847923265304 0.00569407028359330 0.00571005428023826 0.00568591442803343 0.00569223802921068 0.00568280396557755 0.00569379429701353 0.00568185797845967 0.00567578846141909 0.00567894666171056 0.00567320653743196 0.00564915245464934 0.00564795336885811 0.00563876138924631 0.00570327030061123 0.00572680520535131 0.00571341874853335 0.00570107369687699 0.00569366065757143 0.00568594220398016 0.00567630269020819 0.00567557721022348 0.00570737562187293 0.00568463677306191 0.00568722490736285 0.00569534732529629 0.00570479990114402 0.00571411996406732 0.00571919739151671 0.00570542626235309 0.00566178844454233 0.00566838603319001 0.00568828948707550 0.00567046602172531 0.00570652500348306 0.00574602789466369 0.00572372616038287 0.00569588990905875 0.00570224072822409 0.00566492842363350 0.00569041869408171 0.00568476030982822 0.00569910124619053 0.00574897246393032 0.00574568231591548 0.00572309009522084 0.00570247024465303 0.00570511017839644 0.00572662555187936 0.00571990918102634 0.00571698038953088 0.00571766341142988 0.00571654118780795 0.00568621131344125 0.00567004964850449 0.00569846565094612 0.00569116456760390 0.00569510598289396 0.00568087262388984 0.00570926927674283 0.00572360544301340 0.00570546782794645]

my y-data is:

 [0.0722052523936974    -0.116284726028758  -0.0344955010810505 0.276009056945174   0.539214054114762   0.697655655788995   0.876703275086714   1.35496012084228    2.01348390764194    2.37342873676440    2.07957359248374    1.26036319473217    0.929334696235048   0.590598025351673   0.235834445927379   -0.634855444766222  -0.196035504475008  0.0607926210102202  0.143712062082165   0.261387916727359   1.15208072958088    2.26956029539525    2.14140323725389    1.32527937956009    0.429278098437174   0.187344008350156   0.0921821895387395  -0.596412065951111  -0.639186355715831  -0.0337201126995248 0.315383126436722   -0.303408538825290  0.402049748955380   -0.207361277768321  -0.761283125558432  -0.778208393542492  -0.227622649685446  -0.0288009492235658 0.262994409393257   0.708909565660383   0.862183004974083   0.662850335501189   0.465101063799991   0.253832450093375   -0.0321163889325910 0.0196147382615682  0.0315117409166952  -0.0783649110194694 0.233676109491918   -0.165704737948315  -0.276695047616687  -0.0503088475859882 0.159017398241689   0.774822568696547   1.63033542173908    1.96838543280211    1.78792906416767    1.13295614618529    0.807083189320715   0.542620751782781   0.487896272058663   -0.00418587643191634    -0.144295513271167  0.183873423191098   0.113533725207394   -0.00430357220947427    -0.196791317313870  -0.643489823492006  -0.951364421237257  -0.383361081478214  0.426211243875390   1.13334866991336    1.27654664494661    0.742748215803317   0.273748243034661   -0.0202656354243274 0.106200919981608   -0.0916810713096844 -0.184773494785167  0.345836159551591   0.378069148554025   -0.106302503639941  -0.678181507182131  -0.378431726956584  -0.0465689524969581 0.0700301240621313  0.413028957779804   1.13116222671878    1.21226247864782    0.963424288062250   0.351751964306848   0.0788086243226716  0.296354131830850   0.159612903347104   -0.0612765052608151 -0.135787574724828  0.000814395527494810    0.0168806776690738  -0.311891165068794  -0.278675668483064  0.00753515744635787 0.135343330694322   0.0280662030524211  0.0595791272469946  -0.252306230696664  -0.588670120243735  -0.634102873468700  -0.232647761703011  0.0517640016372520  0.192936288397654   0.199136689455075   -0.514612856119197  -0.177998975401088  0.144590675561822   -0.0229188957843282 -0.128466737771051  -0.0474331907200261 0.156260656563529   0.558399012388735   0.862981103714190   1.17242711442847    0.744839900638128   0.233199325540570   -0.434334973997602  -0.490085597200587  -0.402577374203381  0.0198723789840384  0.216008528886140   -0.153188926184130  0.0673573786883735  0.487663020673144   0.823850286313647   1.16294695947181    0.948514383294815   0.380337141536444   -0.210720418596695  -0.394244050556542  -0.334141497118032  -0.0795506855064582 0.152861140581226   0.403287746810583   0.126481956417593   -0.108866581219803  -0.206337064585274  0.473057912851193   -0.224741145734541  -0.622433405012376  -0.893040090399582  -0.719194071466464  0.0587807370031474  0.675532690894235   0.981569882283263   0.501648786405245   -0.340750486834334  -1.00302428896732   -1.31187571878483   -0.742195541090809  -0.377781672865874  -0.0280225451392606 0.274264649464704   0.283584937354784   -0.0731474524135590 -0.490698906647292  -0.302573312222626  0.169071932483100   0.295244780718635   -0.00362078349316385    -0.379120810663201  -0.501961543947379  -0.246842022011707  -0.272864519628719  -0.234391733447854  0.0633256095408698  0.353791189510313   0.0911936917798627  -0.271051028413234  0.0225884492332092  0.0108052266881251  0.0382593201666643  0.123282127801682   0.545365640522478   0.756198929543784   1.05092258341412    1.34567714114993    1.55525356844634    1.30396631038941    0.985535529263796   0.878095221851050   1.02480699518592    0.466575891400046   -0.183830651227942  -0.213031465771310]

Does anyone have a better suggestion for this than what I am currently doing?

thanks.

Answer 1

As am304, with such a data set I would strongly suggest to fit you data initially in the YX referential, then only calculate the equivalent in the XY referential if you really need the polynomial coefficients this way.

One very useful function (I use it extensively ) in the curvefit toolbox is the function smooth . In older version of Matlab, it used to be accessible directly from the cftool , but some some reasons I ignore, it is not so obvious anymore but it is still part of the toolbox.

If you only provide one argument (a simple vector), it does nothing more than a moving average (still useful but could be achieved by several other ways). However, it becomes very powerful at smoothing scattered data like you have, specially when you have data scattered in one dimension ( y in your case), and too dense in the other dimension ( x ).

It will usually produce a better fit, and also visually it allows the operator to judge the quality of the fit much easier. Enough talk, graphical example:

First I do the calculations in the YX domain (just replace x by y and vice versa everywhere):

%% // calculations in the Y-X referential
%// this will do the calculations to obtain x = f(y) = pinv(1).y+pinv(2)
x2   = smooth( y , x  , 0.2 ) ;
pinv = polyfit( y , x2 ,1 ) ;
yp   = [min(y) max(y)] ;
xp   = polyval( pinv , yp ) ;

Note that I did the fit on the smoothed data ( x2 ). Also note that I send both variable to the function smooth (the last parameter 0.2 is the span of the data set used for the smoothing). These smoothed data are the red points in the figure below. It is already much easier visually to have an idea of the trend of your data. Plotting all that:

%% // plot in the Y-X referential
figure ; hold on
plot( y ,x  ,'.')
plot( y ,x2 ,'or')
plot( yp,xp ,'k','LineWidth',2 )

Gives the following result:

You can then replot the lot in the XY domain:

%% // plot in the X-Y referential
figure ; hold on
plot( x , y  ,'.')
plot( x2 ,y ,'or')
plot( xp,yp ,'k','LineWidth',2 )

To obtain:

---

Now this gave you a polynomial coefficient which express x=f(y)=pinv(1).y+pinv(2) . If you are happy with this relationship then you are done. If you really need the coefficients to express y=f(x)= p(1).x+p(2) , it is easy enough to reverse the coefficients:

%% // recalculate the polynomial coefficients in X-Y referential
%// to have y = f(x) = p(1).x+p(2)
p = [1 -pinv(2)]./pinv(1) ;

Now the polynomial p contains the right coefficients, and if you build a fit curve with that:

yp2 = polyval( p , xp ) ;
plot( xp, yp2,'m')

You will see that this line in magenta exactly match the black line in the last figure.

Answer 2

To expand slightly on what I said in the comments, my suggestion would be remove the outliers and fit x to y rather y to x . I only have Octave, not MATLAB, but the following is equivalent and should work just as well in MATLAB as in Octave:

p = polyfit(y(x>=0.0056 & x<=0.00575),x(x>=0.0056 & x<=0.00575),1)
yi = linspace(min(y),max(y),50);
xi = polyval(p,yi);
plot(x,y,'bo',xi,yi,'r-')

which gives the following result: