How to make a customized tf.nn.conv2d() of TensorFlow?

Question

Almost same as existing conv2d(), but I would like to add a special function to it.

Suppose that we have a shape of 5x5 filter and a shape of 100x100 input.

In my understanding, conv2d() internally does:

1. choosing a 5x5 input area from the entire input.
2. calculating convolution between the 5x5 input area and the filter.
3. moving onto the next input area by a given strides.

In contrast, customized_conv2d() internally wants to do:

1. choosing a 5x5 input area from the entire input. (same)
2. subtracting a scalar value of f(5x5 input area) from each value in the 5x5 input area. (added)
3. calculating convolution between the subtracted values of the 5x5 input area and the filter. (same)
4. moving onto the next input area by a given strides. (same)

where f(x) is (max(x) + min(x)) / 2

• Is it easy to make it?
• Is the customized code able to run on a GPU?

Browsing relevant source codes of nn_ops.py, gen_nn_ops.py, and nn_ops.cc, I come to ask for your help.

Thanks in advance!

APPENDED:

What I have learned so far:

1) The easiest way for running on CPU might be to make customized TensorFlow. Tweak, as less as possible, tensorflow/core/kernels/conv_ops.cc and tensorflow/core/kernels/deep_conv2d.cc . With this special TensorFlow, conv2d() behaves in a desired way, instead of original one. Adding a new op, such as customized_conv2d(), by duplicating relevant lines and renaming all functions/methods from python wrapper to c++ code could be excessive efforts in this case.

2) No hope for running on GPU through CUDA. It seems that conv2d() of TensorFlow eventually calls cudnnConvolutionForward() in the NVIDIA CUDA Deep Neural Network library (cuDNN). Primary part of conv2d() is done in this library. Software License Agreement (SLA) for NVIDIA cuDNN does not allow reverse engineering or modification. No source code of the library is provided.

3) There might be another hope for running on GPU through OpenCL, though.

Answer 1

Since convolution itself is linear, inserting any linear operation can be done by doing convolution via tf.nn.conv2d first and then that operation.

The (max(x) + min(x)) / 2 on each 5x5 patch can be realized by (tf.nn.max_pool(x) - tf.nn.max_pool(-x)) * 0.5 . To subtract this after conv, you also need to multiple the results by corresponding sums of convolution kernels. After that, you may apply the nonlinear activation function.

However, in general I don't know how to efficiently add nonlinear operations, eg get z-score for each 5x5 patch before convolutional multiplication. Perhaps other answers could provide some insights.