how do I implement Salt& Pepper layer in Keras?

Question

I need to implement salt & pepper layer in keras like Gaussian noise, I tried to use the following code but it produces several errors. could you please tell me what is the problem? do you have any other suggestion for implementing S&P layer? Thank you.

from keras.engine.topology import Layer

class SaltAndPepper(Layer):

    def __init__(self, ratio, **kwargs):
        super(SaltAndPepper, self).__init__(**kwargs)
        self.supports_masking = True
        self.ratio = ratio

    def call(self, inputs, training=None):
        def noised():
            r = self.ratio*10
            s = inputs.shape[1]
            n = int( s * r/10 )
            perm = np.random.permutation(r)[:n]
            inputs[perm] = (np.random.rand(n) > 0.5)
            return inputs

        return K.in_train_phase(noised(), inputs, training=training)

    def get_config(self):
        config = {'ratio': self.ratio}
        base_config = super(SaltAndPepper, self).get_config()
        return dict(list(base_config.items()) + list(config.items()))

Traceback (most recent call last):

File "", line 125, in decoded_noise=SaltAndPepper(0.5)(decoded)

File "D:\\software\\Anaconda3\\envs\\py36\\lib\\site-packages\\keras\\engine\\base_layer.py", line 457, in call output = self.call(inputs, **kwargs)

File "", line 57, in call return K.in_train_phase(noised(), inputs, training=training)

File "", line 52, in noised n = int( s * r/10 )

TypeError: unsupported operand type(s) for /: 'Dimension' and 'int'

Update:

I used @today's solution and wrote the following code:

decoded_noise=call(0.05,bncv11)#16

which bncv11 is the output of batch normalization layer before it.

but it produces this error, why does it happen?

Traceback (most recent call last):

File "", line 59, in decoded_noise=call(0.05,bncv11)#16

File "", line 34, in call return K.in_train_phase(noised(), inputs, training=training)

File "", line 29, in noised mask_select = K.random_binomial(shape=shp, p=self.ratio)

AttributeError: 'float' object has no attribute 'ratio'

after saving the model and used it produces this error:

Traceback (most recent call last):

File "", line 1, in b=load_model('Desktop/los4x4_con_tile_convolw_FBN_SigAct_SandPAttack05.h5',custom_objects={'tf':tf})

File "D:\\software\\Anaconda3\\envs\\py36\\lib\\site-packages\\keras\\engine\\saving.py", line 419, in load_model model = _deserialize_model(f, custom_objects, compile)

File "D:\\software\\Anaconda3\\envs\\py36\\lib\\site-packages\\keras\\engine\\saving.py", line 225, in _deserialize_model model = model_from_config(model_config, custom_objects=custom_objects)

File "D:\\software\\Anaconda3\\envs\\py36\\lib\\site-packages\\keras\\engine\\saving.py", line 458, in model_from_config return deserialize(config, custom_objects=custom_objects)

File "D:\\software\\Anaconda3\\envs\\py36\\lib\\site-packages\\keras\\layers__init__.py", line 55, in deserialize printable_module_name='layer')

File "D:\\software\\Anaconda3\\envs\\py36\\lib\\site-packages\\keras\\utils\\generic_utils.py", line 145, in deserialize_keras_object list(custom_objects.items())))

File "D:\\software\\Anaconda3\\envs\\py36\\lib\\site-packages\\keras\\engine\\network.py", line 1022, in from_config process_layer(layer_data)

File "D:\\software\\Anaconda3\\envs\\py36\\lib\\site-packages\\keras\\engine\\network.py", line 1008, in process_layer custom_objects=custom_objects)

File "D:\\software\\Anaconda3\\envs\\py36\\lib\\site-packages\\keras\\layers__init__.py", line 55, in deserialize printable_module_name='layer')

File "D:\\software\\Anaconda3\\envs\\py36\\lib\\site-packages\\keras\\utils\\generic_utils.py", line 138, in deserialize_keras_object ': ' + class_name)

ValueError: Unknown layer: SaltAndPepper

I put this code in my program where I define my network structure:

from keras.engine.topology import Layer

class SaltAndPepper(Layer):

    def __init__(self, ratio, **kwargs):
        super(SaltAndPepper, self).__init__(**kwargs)
        self.supports_masking = True
        self.ratio = ratio

    # the definition of the call method of custom layer
    def call(self, inputs, training=True):
        def noised():
            shp = K.shape(inputs)[1:]
            mask_select = K.random_binomial(shape=shp, p=self.ratio)
            mask_noise = K.random_binomial(shape=shp, p=0.5) # salt and pepper have the same chance
            out = inputs * (1-mask_select) + mask_noise * mask_select
            return out

        return K.in_train_phase(noised(), inputs, training=training)

    def get_config(self):
        config = {'ratio': self.ratio}
        base_config = super(SaltAndPepper, self).get_config()
        return dict(list(base_config.items()) + list(config.items()))

Answer 1

In image processing, salt and pepper noise basically changes the value of a ratio of pixels, which are selected randomly, to either salt (ie white, which is usually 1 or 255 depending on the range of image values) or pepper (ie black which is usually 0). Although, we can use the same idea in other domains besides image processing. So you must specify three things first:

1. How many of the pixels should be changed? (noise ratio)
2. Which pixels should be selected and changed?
3. Which of those selected pixels should be salted (and the other peppered)?

Since there is a function in Keras backend to generate random values from a binomial distribution (ie 0 or 1) with a given probability , we can easily do all the above steps by generating two masks: one for selecting the pixels with the given ratio, and the other one for applying either of salt or pepper to those selected pixels. Here is how to do it:

from keras import backend as K

# NOTE: this is the definition of the call method of custom layer class (i.e. SaltAndPepper)
def call(self, inputs, training=None):
    def noised():
        shp = K.shape(inputs)[1:]
        mask_select = K.random_binomial(shape=shp, p=self.ratio)
        mask_noise = K.random_binomial(shape=shp, p=0.5) # salt and pepper have the same chance
        out = inputs * (1-mask_select) + mask_noise * mask_select
        return out

    return K.in_train_phase(noised(), inputs, training=training)

Note in the code above I have assumed a few things:

• The value of the given noise ratio is in the range [0,1].
• If you are using this layer to directly apply it on images, then you must note that it assumes that the images are grayscale (ie single color channel). To use it for RGB images (ie three color channels) you may need to modify it a little such that a pixel with all of its channels are selected for adding the noise (although, this depends on how you define and use salt and pepper noise).

• It assumes the value of salt is 1 and the value of pepper is 0. Though, you can easily change the value of salt to x and pepper to y by changing the definition of mask_noise as follow:

   mask_noise = K.random_binomial(shape=shp, p=0.5) * (xy) + y 

• The same noise pattern is applied on the all the samples in a batch (however, it would be different from batch to batch).

Answer 2

I don't think it's possible to play with the layers in a model, but in deep learning there's always a process called image preprocessing which includes removing noise, image resize...etc so you can preprocess the images before feeding them into the neural.

open cv library is the best for doing such a problem like this

check : https://docs.opencv.org/3.0-beta/doc/py_tutorials/py_imgproc/py_filtering/py_filtering.html