tf.keras.layers.Dense output 对同一输入行的更改

Question

I'm using a dense layer in TF2 using graph mode. Input to the dense layer has shape batch_size * sen_len * embedding_size and the dense layer is defined as tf.keras.layers.Dense(units=feature_dim, activation='relu')

Output is batch_size * sen_len * units

My expectation was, each row of output only depends on the same row in the input. However, I see that the values of the first row changes when the input adds a new row (however the first row in not changing). In one of the cases: input:

[[[0.0571852736 0.0287841056 0.101935618 0.0874886662 1.05053329 0.98418349 0.969990492 0.945339322]
[0.847961783 0.140915036 0.0435606614 0.0663332716 0.540852189 1.11434972 0.913121104 0.955932319]
[0.981501162 0.268874854 0.0995861515 0.0536317527 -0.327278584 0.866326749 0.987028778 0.913541615]
[0.213323712 0.365725756 0.109582983 0.0546317473 -0.901124239 0.841596663 0.986778796 0.913539171]]]

output:

[[[0.578112364 0 0.658412695 0 0 0 0.261683643 0]
[0.310602546 0 0.123107374 0 0 0 0.483636916 0]
[0.275210589 0.00601896644 0 0 0 0.212952077 0.767679 0]
[0.45270589 0.806572795 0 0 0 0 0.787857771 0]]]

and after we add another row to the input, output changes: input:

[[[0.0571852736 0.0287841056 0.101935618 0.0874886662 1.05053329 0.98418349 0.969990492 0.945339322]
[0.847961783 0.140915036 0.0435606614 0.0663332716 0.540852189 1.11434972 0.913121104 0.955932319]
[0.981501162 0.268874854 0.0995861515 0.0536317527 -0.327278584 0.866326749 0.987028778 0.913541615]
[0.213323712 0.365725756 0.109582983 0.0546317473 -0.901124239 0.841596663 0.986778796 0.913539171]
[-0.215699628 1.06977916 0.238905698 0.0676310733 -0.868791223 -0.142939389 0.974456 0.91341567]]]

output:

[[[0.578112364 0 0.658412755 0 0 0 0.261683613 0]
[0.310602546 0 0.123107374 0 0 0 0.483636916 0]
[0.275210589 0.00601896644 0 0 0 0.212952077 0.767679 0]
[0.45270589 0.806572795 0 0 0 0 0.787857771 0]
[0.670830309 0.880182147 0 0.296360224 0 0 0.905243635 0]]]

As mentioned, output changed for the first row while the input for that row is the same.

Answer 1

The output of your model is the same in both the cases, if you want the values to match exactly every time then you would have to set precision for the output variables.

You can do this in two ways:

 # 1. Apply keras mixed precision policy on your network
 
     from tf.keras import mixed_precision
     policy = mixed_precision.Policy('float32')
     mixed_precision.set_global_policy(policy)

 # 2. Apply precision checks on your output tensor.
     
     output = model.predict(input)
     output = np.round(output, 2)