稀疏连接的网络 Keras

Question

I am working with custom-made networks in Keras, which tend to be sparsely more often than densely connected, but there is a problem. Instead of usual 10 seconds to train and evaluate the results of a network on MNIST dataset, these sparsely connected networks take about 15 mins each which is a really big difference, so my concern here is - is this the normal behavior of sparsely connected networks that are badly connected or is it something else? Also, the optimizer that I use is SGD, but I don't think that it is the cause of this problem.

Answer 1

You might have to post some of the output from both your dense and sparse network, but if you haven't changed any other parameters besides the network, then I would say the time increase is normal. HOWEVER , I would need to know the exact architecture of both networks to be 100% certain.

You also mentioned that you are using the optimizer SGD, which I believe is also perfectly fine. However, if you want to increase the speed of training, I would suggest you look into the Adam optimizer. This is because, while SGD generalizes better for finding the minimum, Adam is faster. Here is a blog post about the differences between SGD and Adam and which would work in your specific case.