Implementing Custom Min_MAX_Pooling Layer in Tensorflow

Question

Hi i am trying to implement coustom min max plooing layer in tensorflow using lambda layers to reduce noise in time series data. Here is the function that dose the min max pooling

def min_max_pooling(sequence, window=5):

    output = tf.constant([],dtype='float64')
    max_ = tf.Variable(0,dtype = 'float64')
    min_ = tf.Variable(0,dtype = 'float64')

    # loop over sequence in chunks, get the min max values and concat all of them into single 
    tensor and return as output.

    for i in range(window, len(sequence) + window, window):
        chunk = sequence[i - window:i]
        print(i)
        
        # get the max and min values from chunk

        max_.assign(chunk[tf.argmax(chunk)])
        min_.assign(chunk[tf.argmin(chunk)])

        # get the index of max and min values from chunk

        max_index = tf.argmax(chunk)
        min_index = tf.argmin(chunk)

        
        # append values to output tensor according to the original sequence
        # if min was first in sequence than max i,e.  tf.greater(max_index , min_index) == True,  
        # append min first and then max else vice versa

        if tf.greater(max_index , min_index):
            output = tf.concat([output, [min_]],-1)
            output = tf.concat([output, [max_]],-1) 

        else:
            output = tf.concat([output, [max_]],-1)
            output = tf.concat([output, [min_]],-1)

    return tf.convert_to_tensor(output)

# print(tf.autograph.to_code(min_max_pooling))

# min_max_pooling = tf.autograph.to_graph(min_max_pooling)

The function accepts two arguments a 1d tensor of time series data(scaled between 0 to 1) and window size. It than calculates output sequence w.r.t to windows size and returns a tensor. Basicly its a function that works just like maxpooling1d which helps to reduce noise(downsample's data) but also accounts for min values which is the reason why i want to implement it. Here is test output of this function.

tf.Tensor( [0.99941323 0.98313041 0.97799619 0.98533079 0.98635764 0.99457239 0.99413232 0.99105178 0.99193193 0.98753117 0.98489071 0.98371718 0.98459733 0.98445064 0.98386387 0.98547748 0.99163855 0.99061171 0.99735954 1. ], shape=(20,), dtype=float64)

[![enter image description here][1]][1]

after minmaxpooling

[![enter image description here][2]][2]

Now the problem arises when i use it as a lambda layer in tensorflow models i get all kinds of error tried to solve all of them but still cant figure out the problem. I cant get it working with tensorflow.here is the code for that.\

input_layer = tf.keras.layers.Input(shape=(1000,), name="input_layer")

output_layer = tf.keras.layers.Lambda(min_max_pooling, name="lambda_layer")(input_layer)

model = tf.keras.models.Model(input_layer, output_layer, name="model")

model.compile(optimizer=tf.keras.optimizers.Adam(learning_rate=0.0005), loss="categorical_crossentropy", run_eagerly=True)

All i want to build is a layer that accepts a sequence of time series data and removes the noise from it just like maxpooling1d except also taking into account the significance of min values in sequence.

The end result should be like passing a tensor to the layer

tf.Tensor(
[0.99941323 0.98313041 0.97799619 0.98533079 0.98635764 0.99457239
 0.99413232 0.99105178 0.99193193 0.98753117 0.98489071 0.98371718
 0.98459733 0.98445064 0.98386387 0.98547748 0.99163855 0.99061171
 0.99735954 1.        ], shape=(20,), dtype=float64)

and getting the downsampled output as

tf.Tensor(
[0.99941323 0.97799619 0.99457239 0.98753117 0.98489071 0.98371718
 0.98547748 1.        ], shape=(8,), dtype=float64)

Now i know that i dont know a lot about tensorflow. But please can anyone help me implement the full working code for the problem.? And also dont know how to pass the window paramater to lambda layer? want help with that to. Any kind of help is appreciated.

ValueError: Exception encountered when calling layer "lambda_layer" (type Lambda).

New error

The following Variables were created within a Lambda layer (lambda_layer)
but are not tracked by said layer:
  <tf.Variable 'lambda_layer/map/while/Variable:0' shape=() dtype=float32>
  <tf.Variable 'lambda_layer/map/while/Variable:0' shape=() dtype=float32>
The layer cannot safely ensure proper Variable reuse across multiple
calls, and consequently this behavior is disallowed for safety. Lambda
layers are not well suited to stateful computation; instead, writing a
subclassed Layer is the recommend way to define layers with
Variables.

2nd Error

data = pd.read_csv('/content/Stock_data.csv', parse_dates=False,
                   index_col=1)

tensor = data.close.head(10).to_numpy(dtype='float64')
tensor = tensor / max(tensor)
print(tensor)
# tensor = tf.convert_to_tensor(tensor)

print(tensor)
p = model.predict(tensor)
print(p)

error:

[1.         0.98370762 0.97857038 0.98590929 0.98693674 0.99515632
 0.99471598 0.99163364 0.99251431 0.98811096]
[1.         0.98370762 0.97857038 0.98590929 0.98693674 0.99515632
 0.99471598 0.99163364 0.99251431 0.98811096]
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-59-5028b0f38a02> in <module>()
      8 
      9 print(tensor)
---> 10 p = model.predict(tensor)
     11 print(p)

1 frames
<ipython-input-54-a3aac9d10348> in multiple_min_max_pooling(sequences)
      1 def multiple_min_max_pooling(sequences):
----> 2     return tf.map_fn(min_max_pooling, sequences)

ValueError: Exception encountered when calling layer "lambda_layer" (type Lambda).

in user code:

    File "<ipython-input-53-495c7ee6064b>", line 10, in min_max_pooling  *
        for i in range(window, len(sequence) + window, window):

    ValueError: len requires a non-scalar tensor, got one of shape []


Call arguments received:
  • inputs=tf.Tensor(shape=(10,), dtype=float32)
  • mask=None
  • training=False

Answer 1

Apart from the fact that usually TF uses float32 in my experience, as float 64 is double the memory, and usually the additional precision/big numbers are not useful, your problem is that you are not considering that TF uses batches of data

In other words, your layer will receive a batch of sequences, not a single one. Your code works fine for a single one, thus can be easily (but not efficiently) fixed with a tf.map_fn :

def multiple_min_max_pooling(sequences):
    return tf.map_fn(min_max_pooling, sequences)

...
tf.keras.layers.Lambda(multiple_min_max_pooling, name="lambda_layer")(input_layer)

About the time window, you have 2 choices, either defined a custom layer (more elegant and readable), or define a function that return a function (this "design pattern" is called higher-order functions if you want to read more about it online to understand how it works):

def multiple_min_max_pooling(window=5):
    def fn(sequences):
        return tf.map_fn(min_max_pooling(window), sequences)
    return fn
def min_max_pooling(window=5):
    def fn(sequence):
        output = tf.constant([],dtype='float32')
        max_ = tf.Variable(0,dtype = 'float32')
        min_ = tf.Variable(0,dtype = 'float32')

        # loop over sequence in chunks, get the min max values and concat all of them into single

        for i in range(window, len(sequence) + window, window):
            chunk = sequence[i - window:i]
            print(i)

            # get the max and min values from chunk

            max_.assign(chunk[tf.argmax(chunk)])
            min_.assign(chunk[tf.argmin(chunk)])

            # get the index of max and min values from chunk

            max_index = tf.argmax(chunk)
            min_index = tf.argmin(chunk)


            # append values to output tensor according to the original sequence
            # if min was first in sequence than max i,e.  tf.greater(max_index , min_index) == True,
            # append min first and then max else vice versa

            if tf.greater(max_index , min_index):
                output = tf.concat([output, [min_]],-1)
                output = tf.concat([output, [max_]],-1)

            else:
                output = tf.concat([output, [max_]],-1)
                output = tf.concat([output, [min_]],-1)

        return tf.convert_to_tensor(output)
    return fn

# and now you can do this:
tf.keras.layers.Lambda(multiple_min_max_pooling(window=2), name="lambda_layer")(input_layer)