TensorFlow CNN教程：如何编辑顶层以进行本地连接？

Question

I have some background in machine learning and python, but I am just learning TensorFlow. I am going through the tutorial on deep convolutional neural nets to teach myself how to use it for image classification. Along the way there is an exercise, which I am having trouble completing.

EXERCISE: The model architecture in inference() differs slightly from the CIFAR-10 model specified in cuda-convnet. In particular, the top layers of Alex's original model are locally connected and not fully connected. Try editing the architecture to exactly reproduce the locally connected architecture in the top layer.

The exercise refers to the inference() function in the cifar10.py model . The 2nd to last layer (called local4) has a shape=[384, 192], and the top layer has a shape=[192, NUM_CLASSES], where NUM_CLASSES=10 of course. I think the code that we are asked to edit is somewhere in the code defining the top layer:

with tf.variable_scope('softmax_linear') as scope:
    weights = _variable_with_weight_decay('weights', [192, NUM_CLASSES],
                                      stddev=1/192.0, wd=0.0)
    biases = _variable_on_cpu('biases', [NUM_CLASSES],
                          tf.constant_initializer(0.0))
    softmax_linear = tf.add(tf.matmul(local4, weights), biases,name=scope.name
    _activation_summary(softmax_linear)

But I don't see any code that determines the probability of connecting between layers, so I don't know how we can change the model from fully connected to locally connected. Does somebody know how to do this?

Answer 1

I'm also working on this exercise. I'll try and explain my approach properly, rather than just give the solution. It's worth looking back at the mathematics of a fully connected layer ( https://www.tensorflow.org/get_started/mnist/beginners ).

So the linear algebra for a fully connected layer is:

y = W * x + b

where x is the n dimensional input vector, b is an n dimensional vector of biases, and W is an n -by- n matrix of weights. The i th element of y is the sum of the i th row of W multiplied element-wise with x .

So....if you only want y[i] connected to x[i-1] , x[i] , and x[i+1] , you simply set all values in the i th row of W to zero, apart from the (i-1) th, i th and (i+1) th column of that row. Therefore to create a locally connected layer, you simply enforce W to be a banded matrix ( https://en.wikipedia.org/wiki/Band_matrix ), where the size of the band is equal to the size of the locally connected neighbourhoods you want. Tensorflow has a function for setting a matrix to be banded ( tf.batch_matrix_band_part(input, num_lower, num_upper, name=None) ).

This seems to me to be the simplest mathematical solution to the exercise.

Answer 2

I'll try to answer your question although I'm not 100% I got it right as well.

Looking at the cuda-convnet architecture we can see that the TensorFlow and cuda-convnet implementations start to differ after the second pooling layer.

TensorFlow implementation implements two fully connected layers and softmax classifier.

cuda-convnet implements two locally connected layers, one fully connected layer and softmax classifier.

The code snippet you included refers only to the softmax classifier and is in fact shared between the two implementations. To reproduce the cuda-convnet implementation using TensorFlow we have to replace the existing fully connected layers with two locally connected layers and a fully connected one.

Since Tensor doesn't have locally connected layers as part of the SDK we have to figure out a way to implement it using the existing tools. Here is my attempt to implement the first locally connected layers:

  with tf.variable_scope('local3') as scope:
    shape = pool2.get_shape()
    h = shape[1].value
    w = shape[2].value

    sz_local = 3 # kernel size
    sz_patch = (sz_local**2)*shape[3].value
    n_channels = 64

    # Extract 3x3 tensor patches
    patches = tf.extract_image_patches(pool2, [1,sz_local,sz_local,1], [1,1,1,1], [1,1,1,1], 'SAME')
    weights = _variable_with_weight_decay('weights', shape=[1,h,w,sz_patch, n_channels], stddev=5e-2, wd=0.0)
    biases = _variable_on_cpu('biases', [h,w,n_channels], tf.constant_initializer(0.1))

    # "Filter" each patch with its own kernel 
    mul = tf.multiply(tf.expand_dims(patches, axis=-1), weights)
    ssum = tf.reduce_sum(mul, axis=3)
    pre_activation = tf.add(ssum, biases)
    local3 = tf.nn.relu(pre_activation, name=scope.name)