在keras自定义层中包括高级计算（类似于scikit）

Question

Normally I would preprocess the data before I feed it into my model for classification.

This is however not possible and thus am stuck either to enhance the performance of the model further (somehow) or include useful preprocessing steps directly inside the model.

How can I do that? The best solution I found thus far, included re-implementing the functionality I want using Keras backend. This is far from a good solution and thus I am hoping someone has an idea, how to salavage the situation.

Below are links I found useful + my current code.

Useful links:

Keras Custom Layer with advanced calculations

How to Switch from Keras Tensortype to numpy array for a custom layer?

How to create a Keras Custom Layer using functions not included in the Backend, to perform tensor sampling?

My code thus far:

def freezeBaseModelLayers(baseModel):
    for layer in baseModel.layers:
        layer.trainable = False


def preprocess_input(x):
    # TODO: Not working, but intention should be clear
    numpy_array = tf.unstack(tf.unstack(tf.unstack(x, 224, 0), 224, 0), 1, 0)
    from skimage.feature import hog
    from skimage import data, exposure
    img_adapteq = exposure.equalize_adapthist(numpy_array, orientations=8, pixels_per_cell=(3, 3),
                                              cells_per_block=(1, 1), visualize=True, multichannel=False)
    [x1, x2, x3] = tf.constant(img_adapteq), tf.constant(img_adapteq), tf.constant(img_adapteq)
    img_conc = Concatenate([x1, x2, x3])
    return img_conc


def create(x):
    is_training = tf.get_variable('is_training', (), dtype=tf.bool, trainable=False)
    with tf.name_scope('pretrained'):
        # Add preprocess step here...
        input_layer = Lambda(preprocess_input(x), input_shape=(224, 224, 1), output_shape=(224, 224, 3))

        baseModel = vgg16.VGG16(weights='imagenet', include_top=False, input_shape=(224, 224, 3))    
        freezeBaseModelLayers(baseModel)
        layer = baseModel(input_layer)
        layer = GlobalMaxPooling2D()(layer)
        layer = Dense(1024, activation='relu')(layer)
        layer = Dense(2, activation=None)(layer)
        model = Model(input=input_layer.input, output=layer)
        output = model(x)
        return output




I would like to include prepocessing steps inside my model

The models I am working with are receiving noisy data. In order to enhance the performance of the models, I would like to do some preprocessing steps e.g. equalize_adapthist.

Answer 1

A better way to do this is via a custom keras layer. Here is an example:

import tensorflow as tf
from keras.layers import Layer, Input, Conv2D
from keras.models import Model
from keras import backend as K
from skimage.feature import hog
from skimage import data, exposure

def equalize(img):
  img_adapteq = exposure.equalize_adapthist(img)
  return img_adapteq

def preprocess_input(img):
  return tf.py_func(equalize, 
                     [img],
                     'float32',
                     stateful=False,
                     name='custom_image_op')


class CustomLayer(Layer):
  def __init__(self, output_dim, **kwargs):
    self.output_dim = output_dim
    self.trainable = False
    super(CustomLayer, self).__init__(**kwargs)

  def call(self, x):
    res = tf.map_fn(preprocess_input, x)
    res.set_shape([x.shape[0],
                   self.output_dim[1], 
                   self.output_dim[0],
                   x.shape[-1]])
    return res

output_dim = (224,224)
inputs = Input(shape=(224,224,3))
x = CustomLayer(output_dim)(inputs)
x = Conv2D(32, (3,3))(x)
x = Flatten()(x)
x = Dense(1)(x)

model = Model(inputs, x)
model.summary()

# test
sample = np.random.rand(4, 224,224,3).astype(np.float32)
y = np.random.randint(2, size=(4,))

model.compile("sgd", "mse")
model.fit(sample, y)

Answer 2

To do this with a Lambda layer, you would need to write histogram equalization in pure tensorflow . Indeed, when building the graph, the function ( preprocess_input ) will be called on tensorflow placeholders which, in the case of this skimage function which expects numpy arrays, will not work.

This question shows how to write it in pure tensorflow . Copy-pasting here for the sake of redundancy/ease-of-reading (I have not tested it myself but a test is available in the question):

def tf_equalize_histogram(image):
    values_range = tf.constant([0., 255.], dtype = tf.float32)
    histogram = tf.histogram_fixed_width(tf.to_float(image), values_range, 256)
    cdf = tf.cumsum(histogram)
    cdf_min = cdf[tf.reduce_min(tf.where(tf.greater(cdf, 0)))]

    img_shape = tf.shape(image)
    pix_cnt = img_shape[-3] * img_shape[-2]
    px_map = tf.round(tf.to_float(cdf - cdf_min) * 255. / tf.to_float(pix_cnt - 1))
    px_map = tf.cast(px_map, tf.uint8)

    eq_hist = tf.expand_dims(tf.gather_nd(px_map, tf.cast(image, tf.int32)), 2)
    return eq_hist

Btw you should write the pre-processing step as (once you made the preprocessing step pure tensorflow ):

input_layer = Lambda(preprocess_input, input_shape=(224, 224, 1), output_shape=(224, 224, 3))(x)

Another way to do this is to write a custom layer as pointed out by mlRocks.