Stack Smashing when returning pointer to array

Question

my function neldermead looks like this:

double * neldermead (double data[], double (*function)(int, double, double, double, double, double),
double ia1, double ia2, double ia3, double ia4, double ia5, double rad, int k) {

...

printf("8\n");
double * xout = malloc(5 * sizeof(double));

xout[0] = x[0][0];  
xout[1] = x[1][0]; 
xout[2] = x[2][0];  
xout[3] = x[3][0];  
xout[4] = x[4][0];

printf("10\n");  
return xout;
}

The function is called as such:

...
double * newX;
printf("11\n");  
newX = neldermead (data,  &function1, F1A1, F1A2, F1A3, F1A4, F1A5, 10, 10000);
printf("12\n");  
...

The output is as follows:

11
8
10
*** stack smashing detected ***: ./a.out terminated
Aborted (core dumped)

With -fno-stack-protector the code works fine.

Is there something wrong with how I returned the array or what?

////////////////////////////////////

I modified the code to do return by reference.

my function neldermead looks like this:

double * neldermead (double * xout, double data[], double (*function)(int, double, double, double, double, double),
double ia1, double ia2, double ia3, double ia4, double ia5, double rad, int k) {

...

printf("8\n");


xout[0] = x[0][0];  
xout[1] = x[1][0]; 
xout[2] = x[2][0];  
xout[3] = x[3][0];  
xout[4] = x[4][0];

printf("9\n");  

}

The function is called as such:

...
double newX[5];
printf("10\n");  
neldermead (newX, data,  &function1, F1A1, F1A2, F1A3, F1A4, F1A5, 10, 10000);
printf("11\n");  
...

The output is as follows:

10
8
9
*** stack smashing detected ***: ./a.out terminated
Aborted (core dumped)

Damn, didn't help.

Answer 1

The function

 double (*function)(int, double, double, double, double, double)

Takes parameters. When function returns, it cleans up the stack. It expects parameters to be have been pushed on the stack.

You call it with:

newX = neldermead (data,  &function1, F1A1, F1A2, F1A3, F1A4, F1A5, 10, 10000);

You are not passing parameters to the function. When it returns, it returns to wrong address.