feature number in tensorflow tf.nn.conv2d

Question

In the Tensorflow example "Deep MNIST for Experts" https://www.tensorflow.org/get_started/mnist/pros

I am not clear how to determine the feature number specified in weight of activation function.

For example:

We can now implement our first layer. It will consist of convolution, followed by max pooling. The convolution will compute 32 features for each 5x5 patch.

W_conv1 = weight_variable([5, 5, 1, 32])

Why 32 is picked here?

In order to build a deep network, we stack several layers of this type. The second layer will have 64 features for each 5x5 patch.

W_conv2 = weight_variable([5, 5, 32, 64])

Again, why 64 is picked？

Now that the image size has been reduced to 7x7, we add a fully-connected layer with 1024 neurons to allow processing on the entire image.

W_fc1 = weight_variable([7 * 7 * 64, 1024])

Why 1024 here?

Thanks

Answer 1

Each of these filters will actually do something, like check for edges, check for colour change, or right-shift, left-shit the image, sharpen, blur etc. Each of these filters are actually working on finding out the meaning of the image by sharpening, enhancing, smoothening, intensifying etc.

For eg check this link which explains the meaning of these filters http://setosa.io/ev/image-kernels/

So all these filters are actually neurons where the output will be max-pooled and eventually fed into a FC layer after some activation.

If you are looking for just understanding the filters, that is another approach. However if you are looking to learn how conv. architectures work but since these are tried and tested filters over the dataset, you hsould just go with it for now.

The filters also learn through Backprop.

Answer 2

32 and 64 are number of filters in the respective layers. 1024 is the number of output neurons in the fully connected layer. Your question basically is about the reason behind the choice of these hyperparameters.

There is no mathematical or programming reason behind these specific choices. These have been picked up after experiments as they delivered a good accuracy over MNIST dataset. You can change these numbers and that is one way by which you can modify a model. Unfortunately you cannot yet explore the reason for the choice behind these parameters within TensorFlow or any other literature source.