如何實現隨機深度，並隨機丟棄整個卷積層？

Question

我正在嘗試實現這個想法： https://arxiv.org/abs/1603.09382 。 基本思想是在基於“保持概率”的訓練期間丟棄 Conv2D 層，例如 Dropout。 我想我可以用這樣的自定義層來做到這一點：

class StochasticConv2D(layers.Layer):
    def __init__(self, **kwargs):
        super(StochasticConv2D, self).__init__()
        self.conv2D = layers.Conv2D(**kwargs)

    def call(self, inputs, training, keep_prob):
        if training and (np.random.uniform() > keep_prob):
            return inputs
        return self.conv2D(inputs)

當我嘗試使用 training = True 時，我收到此錯誤：

ValueError: tf.function-decorated function tried to create variables on non-first call.

如果我讓它工作，我不太確定如何實現非訓練模式。 我是否使用 training = false 再次定義 model 並加載訓練中保存的權重？ 如果我將validation_data 傳遞給model.fit()，那么在運行驗證時如何將“training”設置為false？

Answer 1

要隨機凍結過濾器，您只需制作一個tf.keras.layers.Dropout層，其形狀與卷積過濾器的通道數相同。 在這里，我們有 10 個：

import tensorflow as tf
import numpy as np

model = tf.keras.Sequential([
    tf.keras.layers.Conv2D(10, 3, input_shape=(28, 28, 1)),
    tf.keras.layers.Dropout(.5, noise_shape=(1, 1, 1, 10))])

x = np.random.rand(1, 28, 28, 1)

np.max(model(x, training=True), axis=(1, 2))

array([[-0.        , -0.        ,  0.        ,  0.53856176, -0.        ,
        -0.        ,  0.16301194, -0.        ,  0.76797724,  0.54769045]],
      dtype=float32)

這些都是 10 個卷積濾波器的最大值。 您會看到其中一半只是零。

要退出圖層，您可以執行類似的操作：

import tensorflow as tf
import numpy as np

conv_dropout_layer = tf.keras.Sequential([
    tf.keras.layers.Conv2D(4, 3),
    tf.keras.layers.Dropout(.5, noise_shape=(1, 1, 1, 1))])

x = np.random.rand(1, 28, 28, 1)

model(x, training=True)

一半的時間，所有這些權重都會被凍結。

要返回卷積結果的標識，您可以執行以下操作：

import tensorflow as tf
import numpy as np

class StochasticConv2D(tf.keras.layers.Layer):
    def __init__(self, filters, kernel_size, **kwargs):
        super(StochasticConv2D, self).__init__()
        self.filters = filters
        self.kernel_size = kernel_size
        self.conv2D = tf.keras.layers.Conv2D(filters, kernel_size, padding='SAME', **kwargs)

    def call(self, inputs, **kwargs):
        coin_toss = tf.random.uniform(())
        return tf.cond(tf.greater(.5, coin_toss), lambda: inputs, lambda: self.conv2D(inputs))


x = np.random.rand(1, 7, 7, 10)

s = StochasticConv2D(10, 3)

s(x, training=True).shape

Answer 2

這似乎可以做到（先前解決方案的修改版本）：

class StochasticConv2D(layers.Layer):
    def __init__(self, keep_prob, **kwargs):
        super(StochasticConv2D, self).__init__()
        self.keep_prob = keep_prob
        self.conv2D = layers.Conv2D(**kwargs)

    def call(self, inputs):
        if keras.backend.learning_phase():
            coin_toss = tf.random.uniform(())
            return tf.cond(tf.greater(coin_toss, self.keep_prob), lambda: inputs, lambda: self.conv2D(inputs))
            
        return self.conv2D(inputs)

Answer 3

tensorflow_addons有一個 StochasticDepth 層

import tensorflow_addons as tfa
import numpy as np
import tensorflow as tf

inputs = tf.keras.Input(shape=(28, 28, 1))
residual = tf.keras.layers.Conv2D(32, kernel_size=(3, 3), 
                                  activation="relu",
                                  padding='SAME')(inputs)
x = tfa.layers.StochasticDepth()([inputs, residual])
x = tf.keras.layers.MaxPooling2D(pool_size=(2, 2))(x)
residual = tf.keras.layers.Conv2D(32, kernel_size=(3, 3), 
                                  activation="relu",
                                  padding='SAME')(x)
x = tfa.layers.StochasticDepth()([x, residual])
x = tf.keras.layers.MaxPooling2D(pool_size=(2, 2))(x)
x = tf.keras.layers.Flatten()(x)
x = tf.keras.layers.Dropout(0.5)(x)
outputs = tf.keras.layers.Dense(10, 
                                activation="softmax")(x)

model = tf.keras.Model(inputs=inputs, outputs=outputs)

model.summary()

(x_train, y_train), (x_test, y_test) = tf.keras.datasets.mnist.load_data()

x_train = x_train.reshape(60000, 28, 28, 1).astype("float32") / 255
x_test = x_test.reshape(10000, 28, 28, 1).astype("float32") / 255

model.compile(
    loss=tf.keras.losses.SparseCategoricalCrossentropy(),
    optimizer=tf.keras.optimizers.RMSprop(),
    metrics=["accuracy"],
)

history = model.fit(x_train, y_train, 
                    batch_size=64, epochs=2, 
                    validation_split=0.2)