tensorflow 2 keras shuffle 每行梯度问题

Question

I need a NN which will be giving same output for any permutation of same input. Was trying to search for solution ('permutation invariance'), found some layers, but failed to make them work.

I chose different approach: I want to create a layer, to add as I first in the model, which will randomly shuffle input (each row independently) - please let's follow this approach, I know it can be done outside the model, but I want it as a part of the model. I tried:

class ShuffleLayer(tf.keras.layers.Layer):
    def __init__(self, **kwargs):
        super(ShuffleLayer, self).__init__(**kwargs)

    def call(self, inputs):
        batchSize = tf.shape(inputs)[0]
        cols = tf.shape(inputs)[-1]
        order0 = tf.tile(tf.expand_dims(tf.range(0, batchSize), -1), [1, cols])
        order1 = tf.argsort(tf.random.uniform(shape=(batchSize, cols)))
        indices = tf.stack([tf.reshape(order0, [-1]), tf.reshape(order1, [-1])], axis=-1)
        outputs = tf.reshape(tf.gather_nd(inputs, indices), [batchSize, cols])
        return outputs

I am getting following error:

ValueError: Variable has None for gradient. Please make sure that all of your ops have a gradient defined (ie are differentiable). Common ops without gradient: K.argmax, K.round, K.eval.

How to avoid it ?? I tried to use tf.stop_gradient , but unsuccessfully.

Answer 1

Use Lambda layers:

First of all, if your layer doesn't have trainable weights, you should use a Lambda layer, not a custom layer. It's way simpler and easier.

def shuffleColumns(inputs):
    batchSize = tf.shape(inputs)[0]
    cols = tf.shape(inputs)[-1]
    order0 = tf.tile(tf.expand_dims(tf.range(0, batchSize), -1), [1, cols])
    order1 = tf.argsort(tf.random.uniform(shape=(batchSize, cols)))
    indices = tf.stack([tf.reshape(order0, [-1]), tf.reshape(order1, [-1])], axis=-1)
    outputs = tf.reshape(tf.gather_nd(inputs, indices), [batchSize, cols])
    return outputs

In the model, use a Lambda(shuffleColumns) layer.

About the error

If this is the first layer, this error is probably not caused by this layer. (Unless newer versions of Tensorflow are demanding that custom layers have weights and def build(self, input_shape): defined, which doesn't seem very logical).

It seems you are doing something else in another place. The error is: you are using some operation that blocks backpropagation because it's impossible to have the derivative of that operation.

Since the derivatives are taken with respect to the model's "weights", this means that the operation is necessarily after the first weight tensor in the model (ie: after the first layer that contains trainable weights).

You need to search for anything in your model that doesn't have derivatives, like the error suggests: round, argmax, conditionals that return constants, losses that return sorted y_true but don't return operations on y_pred , etc.

Of course that K.stop_gradients is also an operation that blocks backpropagation and will certainly cause this error if you just use it like that. (This may even be the "cause" of your problem, not the solution)

Below there are easier suggestions for your operation, but none of them will fix this error because this error is somewhere else.

Suggested operation 1

Now, it would be way easier to use tf.random.shuffle for this:

def shuffleColumns(x):
    x = tf.transpose(x)
    x = tf.random.shuffle(x)
    return tf.transpose(x)

Use a Lambda(shuffleColumns) layer in your model. It's true that this will shuffle all columns equally, but every batch will have a different permutation. And since you're going to have many epochs, and you will be shuffling (I presume) samples between each epoch (this is automatic in fit ), you will hardly ever have repeated batches. So:

• each batch will have a different permutation
• it will be almost impossible to have the same batch two times

This approach will probably be way faster than yours.

Suggested operation 2

If you want them permutation invariant, why not use tf.sort instead of permutations? Sort the columns and, instead of having infinite permutations to train, you simply eliminate any possibility of permutation. The model should learn faster, and yet the order of the columns in your input will not be taken into account.

Use the layer Lambda(lambda x: tf.sort(x, axis=-1))

This suggestion must be used both in training and inference.