TensorFlow中的自定义卷积层

Question

Let's assume i want to make the following layer in a neural network: Instead of having a square convolutional filter that moves over some image, I want the shape of the filter to be some other shape, say a rectangle, circle, triangle, etc (this is of course a silly example; the real case I have in mind is something different). How would I implement such a layer in TensorFlow?

I found that one can define custom layers in Keras by extending tf.keras.layers.Layer , but the documentation is quite limited without many examples. A python implementation of a convolutional layer by for example extending the tf.keras.layer.Layer would probably help as well, but it seems that the convolutional layers are implemented in C. Does this mean that I have to implement my custom layer in C to get any reasonable speed or would Python TensorFlow operations be enough?

Edit: Perhaps it is enough if I can just define a tensor of weights, but where I can customize entries in the tensor that are identically zero and some weights showing up in multiple places in this tensor, then I should be able to by hand build a convolutional layer and other layers. How would I do this, and also include these variables in training?

Edit2: Let me add some more clarifications. We can take the example of building a 5x5 convolutional layer with one output channel from scratch. If the input is say 10x10 (plus padding so output is also 10x10)), I would imagine doing this by creating a matrix of size 100x100. Then I would fill in the 25 weights in the correct locations in this matrix (so some entries are zero, and some entries are equal, ie all 25 weights will show up in many locations in this matrix). I then multiply the input with this matrix to get an output. So my question would be twofold: 1. How do I do this in TensorFlow ? 2. Would this be very inefficient and is some other approach recommended (assuming that I want to later customize what this filter looks like and thus the standard conv2d is not good enough).

Edit3: It seems doable by using sparse tensors and assigning values via a previously defined tf.Variable . However I don't know if this approach will suffer from performance issues.

Answer 1

Just use regular conv. layers with square filters, and zero out some values after each weight update:

   g = tf.get_default_graph()
   sess.run(train_step, feed_dict={x: batch_xs, y_: batch_ys})
   conv1_filter = g.get_tensor_by_name('conv1:0')
   sess.run(tf.assign(conv1_filter, tf.multiply(conv1_filter, my_mask)))

where my_mask is a binary tensor (of the same shape and type as your filters) that matches the desired pattern.

EDIT: if you're not familiar with tensorflow, you might get confused about using the code above. I recommend looking at this example , and specifically at the way the model is constructed (if you do it like this you can access first layer filters as 'conv1/weights'). Also, I recommend switching to PyTorch :)