The levenberg-marquardt method for solving non-linear equations

Question

I tried implement the levenberg-marquardt method for solving non-linear equations on Julia based on Numerical Optimization using the Levenberg-Marquardt Algorithm presentation. This my code:

function get_J(ArrOfFunc,X,delta)
  N = length(ArrOfFunc)
  J = zeros(Float64,N,N)
  for i = 1:N
    for j=1:N
      Temp = copy(X);
      Temp[j]=Temp[j]+delta;
      J[i,j] = (ArrOfFunc[i](Temp)-ArrOfFunc[i](X))/delta;
    end
  end
  return J
end

function get_resudial(ArrOfFunc,Arg)
  return  map((x)->x(Arg),ArrOfFunc)
end

function lm_solve(Funcs,Init)
  X = copy(Init)
  delta = 0.01;
  Lambda = 0.01;
  Factor = 2;
  J = get_J(Funcs,X,delta)
  R = get_resudial(Funcs,X)
  N = 5
  for t = 1:N

    G = J'*J+Lambda.*eye(length(X))
    dC = J'*R
    C = sum(R.*R)/2;
    Xnew = X-(inv(G)\dC);
    Rnew = get_resudial(Funcs,Xnew)
    Cnew =  sum(Rnew.*Rnew)/2;
    if ( Cnew < C)
      X = Xnew;
      R = Rnew;
      Lambda = Lambda/Factor;
      J = get_J(Funcs,X,delta)
    else
      Lambda = Lambda*Factor;
    end
    if(maximum(abs(Rnew)) < 0.001)
      return X
    end
  end
  return X
end

function test()
  ArrOfFunc = [
  (X)->X[1]+X[2]-2;
  (X)->X[1]-X[2]
  ];

  X = lm_solve(ArrOfFunc,Float64[3;3])
  println(X)
  return X
end

But from any starting point the step not accepted. What's I doing wrong? Any help would be appreciated.

Answer 1

I have at the moment no way to test this, but one line does not make sense mathematically:

In the computation of Xnew it should be either inv(G)*dC or G\\dC , but not a mix of both. Preferably the second, since the solution of a linear system does not require the computation of the inverse matrix.

With this one wrong calculation at the center of the iteration, the trajectory of the computation is almost surely going astray.