Convolutional Neural Network training returns illogical values

Question

I am using https://github.com/cbovar/ConvNetSharp to use convolutional neural networks in my application.

Unfortunately the libary is not well documented at all. But anyway I have a question about the training process:

I declared a dictionary of types string and List of Bitmaps.

I then added some letters ( exempli gratia a, b, c et cetera) with its corresponding names to the dictionary.

After this, I create the neural network as follows:

            Network.AddLayer(new InputLayer(Width, Height, 1));

        Network.AddLayer(new ConvLayer(5, 5, NumberFilters) { Stride = 1, Pad = 2 });
        Network.AddLayer(new ReluLayer());

        Network.AddLayer(new PoolLayer(2, 2) { Stride = 2 });

        Network.AddLayer(new ConvLayer(5, 5, NumberFilters * 2) { Stride = 1, Pad = 2 });
        Network.AddLayer(new ReluLayer());

        Network.AddLayer(new PoolLayer(3, 3) { Stride = 3 });

        Network.AddLayer(new FullyConnLayer(TrainingSet.Count));
        Network.AddLayer(new SoftmaxLayer(TrainingSet.Count));

If I now start the training process by this method:

        public void StartTraining()
    {
        _initializeLayers();
        _locked = true;
        int _increment = 0;

        Console.WriteLine("Training...");

        foreach (var data in TrainingSet)
        {     
            for (int i = 0; i < data.Value.Count; i++)
            {
                var map = MapBmpToDouble(data.Value[i]);
                var input = new Volume(map, new Shape(map.Length));
                var output = Network.Forward(input, true);
                for (int j = 0; j < Iterations; j++)
                {
                    var trainer = new SgdTrainer(Network) { LearningRate = Epsilon, L2Decay = Decay, Momentum = 0.9, BatchSize = TrainingSet.Count };
                    trainer.Train(input, new Volume(GetTrainingValues(_increment), new Shape(TrainingSet.Count)));
                }
            }
            _increment++;
        }



        Console.WriteLine("Done.");
    }

and test the result for example the letter a, I got values that are very illogical.

The probabilty of the last item/class of the network is ALWAYS the highest one. For the training part I just use the method

    public double[] GetTrainingValues(int index)
    {
        double[] values = new double[TrainingSet.Count];
        values[index] = 1.0;
        return values;
    }

to define which class-position has to be the highest for image X with corresponding label Y.

Also if I use the same image for every label I get the same result! Why?

Do You know how to train best, or can You recommend me a better library?

Thank you very much !

Answer 1

You should instantiate the trainer only once, outside the training loop.

var trainer = new SgdTrainer(Network) { 
                            LearningRate = Epsilon, 
                            L2Decay = Decay, 
                            Momentum = 0.9, 
                            BatchSize = TrainingSet.Count };

for (int j = 0; j < Iterations; j++)
{
    trainer.Train(input, 
                  new Volume(GetTrainingValues(_increment), 
                  new Shape(TrainingSet.Count)));
}