Neural Network Cost Function in MATLAB

Question

How would I implement this neural network cost function in matlab:

Here are what the symbols represent:

% m is the number of training examples.   [a scalar number]
% K is the number of output nodes.   [a scalar number]
% Y is the matrix of training outputs.   [an m by k matrix]
% y^{(i)}_{k} is the ith training output (target) for the kth output node.   [a scalar number]
% x^{(i)} is the ith training input.   [a column vector for all the input nodes]
% h_{\theta}(x^{(i)})_{k} is the value of the hypothesis at output k, with weights theta, and training input i.   [a scalar number]

%note: h_{\theta}(x^{(i)}) will be a column vector with K rows.

I'm having problems with the nested sums, the bias nodes, and the general complexity of this equation. I'm also struggling because there are 2 matrices of weights, one connecting the inputs to the hidden layer, and one connecting the hidden layer to the outputs. Here's my attempt so far.

Define variables

m = 100            %number of training examples
K = 2              %number of output nodes
E = 2              %number of input nodes
A = 2              %number of nodes in each hidden layer
L = 1              %number of hidden layers

Y = [2.2,   3.5    %targets for y1 and y2 (see picture at bottom of page)
     1.7,   2.1
     1.9,   3.6
      .     .      %this is filled out in the actual code but to save space I have used ellipsis. there will be m rows.
      .     .
      .     .
     2.8,   1.6]

X = [1.1,   1.8    %training inputs. there will be m rows
     8.5,   1.0
     9.5,   1.8
      .     .
      .     .
      .     . 
     1.4,   0.8]

W1 = [1.3,  .    .  0.4    %this is just an E by A matrix of random numbers. this is the matrix of initial weights.
       .    .    .  - 2
       .    .    .  3.1
       .    .    .  - 1
      2.1, -8, 1.2, 2.1]

W2 = [1.3,  .    .  0.4    %this is an A by K matrix of random numbers. this is the matrix of initial weights.
       .    .    .  - 2
       .    .    .  3.1
       .    .    .  - 1
      2.1, -8, 1.2, 2.1]

Hypothesis using these weights equals...

Htheta = sigmf( dot(W2 , sigmf(dot(W1 , X))) )   %This will be a column vector with K rows.

Cost Function using these weights equals... (This is where I am struggling)

  sum1 = 0
  for i = 1:K
  sum1 = sum1 + Y(k,i) *log(Htheta(k)) + (1 - Y(k,i))*log(1-Htheta(k))

I just keep writing things like this and then realising it's all wrong. I can not for the life of me work out how to do the nested sums, or include the input matrix, or do any of it. It's all very complicated.

How would I create this equation in matlab?

Thank you very much!

A 2 layer neural network with 2 inputs, 2 outputs, 2 hidden nodes, and 2 bias units http://imagizer.imageshack.us/v2/320x240q90/40/92bn.jpg

Note: The code has strange colours as stackoverflow doesn't know I am programing in MATLAB. I have also wrote the code straight into stackoverflow, so it may have syntax errors. I am more interested in the general idea of how I should go about doing this rather than just having a code to copy and paste. This is the reason I haven't bothered with semi colons and such.

Answer 1

I've implemented neural networks using the same error function as the one you've mentioned above. Unfortunately, I haven't worked with Matlab for quite some time, but I'm fairly proficient in Octave,which hopefully you can still find useful, since many of the functions in Octave are similar to those of Matlab.

@sashkello provided a good snippet of code for computing the cost function. However, this code is written with a loop structure, and I would like to offer a vectorized implementation.

In order to evaluate the current theta values, we need to perform a feed forward/ forward propagation throughout the network. I'm assuming you know how to write the feed forward code, since you're only concerned with the J(theta) errors. Let the vector representing the results of your forward propagation be F

Once you've performed feedforward, you'll need to carry out the equation. Note, I'm implementing this in a vectorized manner.

J = (-1/m) * sum(sum(Y .* log(F) + (1-Y) .* log(1-F),2));

This will compute the part of the summation concerning:

Now we must add the regularization term, which is:

Typically, we would have arbitrary number of theta matrices, but in this case we have 2, so we can just perform several sums to get:

J =J + (lambda/(2*m)) * (sum(sum(theta_1(:,2:end).^2,2)) + sum(sum(theta_2(:,2:end).^2,2)));

Notice how in each sum I'm only working from the second column through the rest. This is because the first column will correspond to the theta values we trained for the `bias units.

So there's a vectorized implementation of the computation of J.

I hope this helps!

Answer 2

I think Htheta is a K*2 array. Note that you need to add bias ( x0 and a0 ) in the forward cost function calculation. I showed you the array dimensions in each step under the assumption that you have two nodes at input , hidden, and output layers as comments in the code.

m = size(X, 1);  
X = [ones(m,1) X]; % m*3 in your case
% W1 2*3, W2 3*2
a2 = sigmf(W1 * X');  % 2*m
a2 = [ones(m,1) a2'];  % m*3    
Htheta = sigmf(a2 * W2);  % m*2    

J = (1/m) * sum ( sum (  (-Y) .* log(Htheta)  -  (1-Y) .* log(1-Htheta) ));

t1 = W1(:,2:size(W1,2));
W2 = W2';
t2 = W2(:,2:size(W2,2));

% regularization formula
Reg = lambda  * (sum( sum ( t1.^ 2 )) + sum( sum ( t2.^ 2 ))) / (2*m);

Answer 3

Well, as I understand your question has nothing to do with neural networks, but basically asking how to make a nested sum in matlab. I don't really want to type in the whole equation above, but, ie, the first part of the first sum will look like:

Jtheta = 0
for i=1:m,
    for j=1:K,
        Jtheta = Jtheta + Y(i, j) * log(Htheta(X(i))(j)) 
    end
end

where Jtheta is your result.

Answer 4

This works on any number of hidden layers:

% Allow arbitrary network architectures. Create cell array of all Theta parameters
Theta={Theta1; Theta2};

% Compute unregularised cost (J)
J = 1/m * sum(sum((-y .* log(hX) - (1 - y) .* log(1 - hX))));

% Add regularisation
for i = 1:length(Theta)
  J += lambda / 2 / m * sum(sum(Theta{i}(:,2:end) .^ 2));
end