Backpropagation for neural network

Question

I'm trying to implement a simple neural network. I know there is a lot of library already available out there, that is not the point.

My network has only 3 layers: one input layer one hidden layer one output layer

The output layer has 8 neuron representing each a different class.

I've understand how to implement the feedfoward algorythm but i'm really struggling for the backpropagation one.

here is what i've come up with so far :

private void backPropagation(List<List<Input>> trainingData)
{
    List<Input> trainingSet = new ArrayList<Input>();
    for (int row = 0; row < trainingData.size(); row++) {           
        trainingSet = trainingData.get(row);
        //we start by getting the output of the network
        List<Double> outputs = feedFoward(trainingSet); 

        //Im using the Iris dataset, so here the desiredOutput is
        //the species where 
        // 1 : setosa
        // 2 : versicolor
        // 3 : virginica
        double desiredOutput = getDesiredOutputFromTrainingSet(trainingSet);    
        //We are getting the output neuron that fired the highest result
        //like if we have
        //Ouput layer :
        //Neuron 1 --> 0.001221513
        //Neuron 2 --> 0.990516510
        //Neuron 3 --> 0.452221000
        //so the network predicted that the trainingData correspond to (2) versicolor
        double highestOutput = Collections.max(outputs);
        //What our neuron should aim for
        double target = 0;

        List<Double> deltaOutputLayer = new ArrayList<Double>();
        List<List<Double>> newWeightsOutputLayer = new ArrayList<List<Double>>();
        for (int j = 0; j < outputs.size(); j++) {  
            double out = outputs.get(j);
            //Important to do j + 1 because the species classes start at 1 (1 : setosa, 2: versicolor, 3:virginica)
            if(out == highestOutput && (j + 1) == desiredOutput)
                target = 0.99; //1
            else
                target = 0.01; //0

            //chain rule
            double delta = (out - target) * LogisticFonction.sigmoidPrime(out);
            deltaOutputLayer.add(delta);


            //get the new weigth value from delta and neta
            List<Double> newWeights = new ArrayList<Double>();
            for (int weightIndex = 0; weightIndex < _outputLayer.get(j).get_weigths().size(); weightIndex++) {
                double gradient = delta * _outputsAfterActivationHiddenLayer.get(weightIndex);
                double newWeight = _outputLayer.get(j).get_weigths().get(weightIndex) - (_learningRate * gradient);
                newWeights.add(newWeight);
            }
            newWeightsOutputLayer.add(newWeights);  
        }

        //hidden layer
        double totalError = 0;
        for (int i = 0; i < _neuronsHiddenLayer.size(); i++) {
            for (int j = 0; j < deltaOutputLayer.size(); j++) {
                double wi = _outputLayer.get(j).get_weigths().get(i);
                double delta = deltaOutputLayer.get(j);
                double partialError = wi * delta;
                totalError += partialError;
            }

            double z = _outputsAfterActivationHiddenLayer.get(i);
            double errorNeuron = LogisticFonction.sigmoidPrime(z);

            List<Double> newWeightsHiddenLayer = new ArrayList<Double>();

            for (int k = 0; k < _neuronsHiddenLayer.get(i).get_weigths().size(); k++) {
                double in = _neuronsHiddenLayer.get(i).get_inputs().get(k);
                double gradient =  totalError * errorNeuron * in;
                double oldWeigth = _neuronsHiddenLayer.get(i).get_weigths().get(k);
                double newWeigth = oldWeigth - (_learningRate * gradient);
                _neuronsHiddenLayer.get(i).get_weigths().set(k, newWeigth);
                newWeightsHiddenLayer.add(newWeigth);
            }
        }


        //then update the weigth of the output layer with the new values.
        for (int i = 0; i < newWeightsOutputLayer.size(); i++) {
            List<Double> newWeigths = newWeightsOutputLayer.get(i);
            _outputLayer.get(i).set_weigths(newWeigths);
        }
    }   
}

I've try testing with the Iris dataset : https://en.wikipedia.org/wiki/Iris_flower_data_set

but my result are very inconsistant leading me to belive there is a bug in my backpropagation algorythm.

If anyone can see some major flaw tell me please!

thanks a lot.

Answer 1

In this part of the code:

if(out == highestOutput && (j + 1) == desiredOutput)
     target = 0.99; //1
else
     target = 0.01; //0

The target output of the neuron is 0.99 when the condition (out == highestOutput && (j + 1) == desiredOutput) . It means that you would only expect the output of the neuron to be 0.99 when the feedforward outputs the same neuron as the training example. This is incorrect.

The condition on that part of the code should've been only (j + 1) == desiredOutput . Remove out == highestOutput condition. The target output should be 0.99 for the desiredOutput neuron no matter whether the feedforward results in that neuron or not. So this is the corrected code:

if((j + 1) == desiredOutput)
     target = 0.99; //1
else
     target = 0.01; //0