具有随机权重的神经网络无法学习

Question

I wanted to compare my classifier that used a VGG16 model pretrained on imagenet to how it would have performed if I did not use the imagenet weights, so I loaded the model by using

model = applications.VGG16(weights=None, include_top=False, input_shape=(img_width, img_height, 3))

According to the Keras documentation using "weights=None" results in randomly initialized weights.

My problem now though is that the neural network always gives the same output, even after training for multiple epochs and trying different learning rates it always predicts all images as the same class.

I do not think the input data (images of 2 different classes) or my code is the problem, because when initializing with the imagenet weights and training on that my classifier learned very well and reached 90% accuracy on the testset.

What could the problem be? Maybe the weight initializtaion? But I don't know how to use a different Initializer when loading the model like that

Answer 1

You are probably facing a vanishing gradient problem.

If you use relu activation function, look at "kaiming initialisation" for your weight. The objective is to keep a mean of 0 and a standard deviation of 1 for your output after each layer during the forward pass.

For relu activation fuction you have to initialize with random normal distribution multiplied by the square root of 2/(number of input for the given layer).

weight_initialisation = random_normal * sqrt(2/(number of input for the layer))

For CNN, I think the number of input will be number of filter * number of cell in the kernel (or 5 * 5 for a [5, 5] kernel)