error decrease too slowly on Neural Network BackPropagation Training

Question

I tried to implement Neural Network backpropagation using JAVA, I already code it, but the result is unsatifying. the error is decreasing too slow. Below are the example of train result:

epoch:1  current error:0.5051166876846451

epoch:2  current error:0.4982484527652138

epoch:3  current error:0.4965995467118879

epoch:4  current error:0.49585659139683363

epoch:5  current error:0.4953426236386938

epoch:6  current error:0.4948766985413233

epoch:7  current error:0.49441754405152294

epoch:8  current error:0.4939551661406868

epoch:9  current error:0.49348601614718984

epoch:10 current error:0.4930078119902486

epoch:11 current error:0.49251846766886453

Based on this I started to doubt my code and its algorithm. The activation function used are sigmoid. Below are The sample code of the training.

public void learning(int epoch,double learningRateTemp,double desiredErrorTemp,DataSet ds,double momentum){
    int processEpoch=0;
    double sumSquaredError=0;
    DataSetRow dsr;
    Connector conTemp;
    double sumError=0;
    double errorInformation=0;
    double activationValue;
    double partialDerivative;

    do{
        processEpoch++;
        sumSquaredError=0;
        System.out.println("epoch:"+processEpoch);
        //data training set
        for(int a=0;a<ds.countRows();a++){
            dsr=ds.getSpecificRow(a);
            sumError=0;
            double[]input=dsr.getInput();
            double[]output=dsr.getdesiredOutput();
            double sumDeltaInput=0;
            double weightTempValue=0;
            //forward calculation
            this.forwardCalculation(input);               
            //backpropagateofError
            //for output unit

            for(int k=0;k<NeuralLayers[totalLayer-1].getTotalNode();k++){
                activationValue=NeuralLayers[totalLayer-1].getNeuron(k).getValue();
                partialDerivative=(activationValue)*(1-activationValue);
                Neuron Temp=NeuralLayers[totalLayer-1].getNeuron(k);
                errorInformation=(output[k]-Temp.getValue())*partialDerivative;
                Temp.SetErrorInformationTerm(errorInformation);
                sumError+=Math.pow((output[k]-Temp.getValue()),2);
                NeuralLayers[totalLayer-1].setNeuron(k, Temp);
            }
            //end of output unit
             //for hidden Unit
            for(int l=totalLayer-2;l>0;l--){
                for(int j=1;j<NeuralLayers[l].getTotalNode();j++){
                     sumDeltaInput=0;
                     for(int k=0;k<NeuralLayers[l+1].getTotalNode();k++){                                
                       conTemp=NeuralLayers[l+1].getConnector(k, j);
                       if(conTemp.getStatusFrom()==false){
                              weightTempValue=conTemp.getWeight().getValue();
                              sumDeltaInput+=(NeuralLayers[l+1].getNeuron(k).GetErrorInformationTerm()*weightTempValue);
                            }
                        }
                     activationValue=NeuralLayers[l].getNeuron(j).getValue();
                     partialDerivative=(activationValue)*(1-activationValue);
                     errorInformation= sumDeltaInput*partialDerivative;                
                     Neuron neuTemp=NeuralLayers[l].getNeuron(j);
                     neuTemp.SetErrorInformationTerm(errorInformation);
                     NeuralLayers[l].setNeuron(j, neuTemp);
                     }
                }
            updateWeight(learningRateTemp,momentum);
            sumSquaredError+=sumError;            
            }       
    sumSquaredError/=(double)(ds.countRows()*NeuralLayers[totalLayer-1].getTotalNode());
    sumSquaredError=Math.sqrt(sumSquaredError);
    System.out.println("current error:"+sumSquaredError);
    } while(processEpoch<epoch && sumSquaredError>desiredErrorTemp);
}

}

for the forward calculation

private void forwardCalculation(double[] inputValue){
        Connector Contemp;
        double SumNodeWeight=0;
        int start=1;
        int count=0;
        setNodeValue(inputValue,0);
        do{
            count++;
            if("output".equals(NeuralLayers[count].statusLayer))
                     start=0;
                else start=1;
           //get sum of all input  
            for(int j=start;j<NeuralLayers[count].getTotalNode();j++){
                for(int i=0;i<NeuralLayers[count].sizeConnector(j);i++){
                    Contemp=NeuralLayers[count].getConnector(j, i);
                    SumNodeWeight+=Contemp.getCombinedweightInput();
                }
                SumNodeWeight=(1/(1+Math.exp(-SumNodeWeight)));
                NeuralLayers[count].setNeuronValue(j, SumNodeWeight); 
                SumNodeWeight=0;
            }
    }while(!"output".equals(NeuralLayers[count].statusLayer));
 }

and to update the weights

private void updateWeight(double learningRateTemp,double momentum){
    double newWeight;
    double errorInformation;
    Connector conTemp;
    for(int LayerPosition=totalLayer-1;LayerPosition>0;LayerPosition--){
        for(int node=1;node<NeuralLayers[LayerPosition].getTotalNode();node++){
            errorInformation=NeuralLayers[LayerPosition].getNeuron(node).GetErrorInformationTerm();
                //for bias weight
                newWeight=learningRateTemp*errorInformation;
                conTemp=NeuralLayers[LayerPosition].getConnector(node, 0);
                conTemp.updateWeight(newWeight,false,0);
                NeuralLayers[LayerPosition].updateConnector(conTemp, node, 0);
                /////////////////////
                //for other node weight
                for(int From=1;From<NeuralLayers[LayerPosition].sizeConnector(node);From++){
                    conTemp=NeuralLayers[LayerPosition].getConnector(node, From);
                    double weightCorrection=learningRateTemp*errorInformation*NeuralLayers[LayerPosition-1].getNeuron(From).getValue();
                    conTemp.updateWeight(weightCorrection,true,momentum);
                    NeuralLayers[LayerPosition].updateConnector(conTemp,node,From);
                }
        }
    }
}

am I on the right Track? I already searched for the bugs in few days, and it still nothing. Does my formula to calculate the error is correct? thank you very much!

Answer 1

Well I'm not any kind of expert on this, nor Java programming, but It might be affecting, you put you variable sumError declared as 0 at the beggining, then you add the error from the outputs, and then in the for cycle of the hidden layers it appears again added to sumSquaredError variable, but if you are going to calc the error of training, why is it inside the "hidden layer cucle"?

for(int l=totalLayer-2;l>0;l--){ 
    for(int j=1;j<NeuralLayers[l].getTotalNode();j++){

    }
    updateWeight(learningRateTemp,momentum);
    sumSquaredError+=sumError;
}

Shouldn't it be outside?

I´ll make reference to the pseudocode of someone who answered me before.

link

Hope this helps!