[英]TensorFlow 2.0 Keras layers with custom tensors as variables
In TF 1.x, it was possible to build layers with custom variables.在 TF 1.x 中,可以使用自定义变量构建层。 Here's an example:这是一个例子:
import numpy as np
import tensorflow as tf
def make_custom_getter(custom_variables):
def custom_getter(getter, name, **kwargs):
if name in custom_variables:
variable = custom_variables[name]
else:
variable = getter(name, **kwargs)
return variable
return custom_getter
# Make a custom getter for the dense layer variables.
# Note: custom variables can result from arbitrary computation;
# for the sake of this example, we make them just constant tensors.
custom_variables = {
"model/dense/kernel": tf.constant(
np.random.rand(784, 64), name="custom_kernel", dtype=tf.float32),
"model/dense/bias": tf.constant(
np.random.rand(64), name="custom_bias", dtype=tf.float32),
}
custom_getter = make_custom_getter(custom_variables)
# Compute hiddens using a dense layer with custom variables.
x = tf.random.normal(shape=(1, 784), name="inputs")
with tf.variable_scope("model", custom_getter=custom_getter):
Layer = tf.layers.Dense(64)
hiddens = Layer(x)
print(Layer.variables)
The printed variables of the constructed dense layer will be custom tensors we specified in the custom_variables
dict:构建的密集层的打印变量将是我们在custom_variables
字典中指定的自定义张量:
[<tf.Tensor 'custom_kernel:0' shape=(784, 64) dtype=float32>, <tf.Tensor 'custom_bias:0' shape=(64,) dtype=float32>]
This allows us to create layers/models that use provided tensors in custom_variables
directly as their weights, so that we could further differentiate the output of the layers/models with respect to any tensors that custom_variables
may depend on (particularly useful for implementing functionality in modulating sub-nets , parameter generation , meta-learning , etc.).这允许我们创建层/模型,直接使用custom_variables
中提供的张量作为它们的权重,以便我们可以进一步区分层/模型的 output 相对于custom_variables
可能依赖的任何张量(对于实现调制中的功能特别有用子网、参数生成、元学习等)。
Variable scopes used to make it easy to nest all off graph-building inside scopes with custom getters and build models on top of the provided tensors as their parameters.变量作用域用于轻松将所有图形构建嵌套在具有自定义 getter 的作用域内,并在提供的张量之上构建模型作为其参数。 Since sessions and variable scopes are no longer advisable in TF 2.0 (and all of that low-level stuff is moved to tf.compat.v1
), what would be the best practice to implement the above using Keras and TF 2.0?由于 TF 2.0 中不再建议使用会话和变量范围(并且所有这些低级内容都已移至tf.compat.v1
),使用 Keras 和 TF 2.0 实现上述内容的最佳实践是什么?
(Related issue on GitHub .) ( GitHub 上的相关问题。)
Given you have:鉴于您有:
kernel = createTheKernelVarBasedOnWhatYouWant() #shape (784, 64)
bias = createTheBiasVarBasedOnWhatYouWant() #shape (64,)
Make a simple function copying the code from Dense
:从Dense
复制代码,制作一个简单的 function :
def custom_dense(x):
inputs, kernel, bias = x
outputs = K.dot(inputs, kernel)
outputs = K.bias_add(outputs, bias, data_format='channels_last')
return outputs
Use the function in a Lambda
layer:在Lambda
层中使用 function:
layer = Lambda(custom_dense)
hiddens = layer([x, kernel, bias])
Warning:
kernel
andbias
must be produced from a Keras layer, or come from ankernel = Input(tensor=the_kernel_var)
andbias = Input(tensor=bias_var)
警告:kernel
和bias
必须从 Keras 层产生,或者来自kernel = Input(tensor=the_kernel_var)
和bias = Input(tensor=bias_var)
If the warning above is bad for you, you can always use kernel
and bias
"from outside", like:如果上面的警告对您不利,您始终可以使用kernel
并“从外部” bias
,例如:
def custom_dense(inputs):
outputs = K.dot(inputs, kernel) #where kernel is not part of the arguments anymore
outputs = K.bias_add(outputs, bias, data_format='channels_last')
return outputs
layer = Lambda(custom_dense)
hiddens = layer(x)
This last option makes it a bit more complicated to save/load models.最后一个选项使保存/加载模型变得更加复杂。
You should probably use a Keras Dense layer and set its weights in a standard way:您可能应该使用 Keras 密集层并以标准方式设置其权重:
layer = tf.keras.layers.Dense(64, name='the_layer')
layer.set_weights([np.random.rand(784, 64), np.random.rand(64)])
If you need that these weights are not trainable, before compiling the keras model you set:如果您需要这些权重不可训练,请在编译 keras model 之前设置:
model.get_layer('the_layer').trainable=False
If you want direct access to the variables as tensors, they are:如果您想直接访问作为张量的变量,它们是:
kernel = layer.kernel
bias = layer.bias
There are plenty of other options, but that depends on your exact intention, which is not clear in your question.还有很多其他选择,但这取决于您的确切意图,这在您的问题中并不清楚。
Below is a general-purpose solution that works with arbitrary Keras models in TF2.下面是一个通用解决方案,适用于 TF2 中的任意 Keras 模型。
First, we need to define an auxiliary function canonical_variable_name
and a context manager custom_make_variable
with the following signatures (see implementation in meta-blocks library ).首先,我们需要定义一个辅助 function canonical_variable_name
和一个具有以下签名的上下文管理器custom_make_variable
(参见元块库中的实现)。
def canonical_variable_name(variable_name: str, outer_scope: str):
"""Returns the canonical variable name: `outer_scope/.../name`."""
# ...
@contextlib.contextmanager
def custom_make_variable(
canonical_custom_variables: Dict[str, tf.Tensor], outer_scope: str
):
"""A context manager that overrides `make_variable` with a custom function.
When building layers, Keras uses `make_variable` function to create weights
(kernels and biases for each layer). This function wraps `make_variable` with
a closure that infers the canonical name of the variable being created (of the
form `outer_scope/.../var_name`) and looks it up in the `custom_variables` dict
that maps canonical names to tensors. The function adheres the following logic:
* If there is a match, it does a few checks (shape, dtype, etc.) and returns
the found tensor instead of creating a new variable.
* If there is a match but checks fail, it throws an exception.
* If there are no matching `custom_variables`, it calls the original
`make_variable` utility function and returns a newly created variable.
"""
# ...
Using these functions, we can create arbitrary Keras models with custom tensors used as variables:使用这些函数,我们可以使用自定义张量作为变量创建任意 Keras 模型:
import numpy as np
import tensorflow as tf
canonical_custom_variables = {
"model/dense/kernel": tf.constant(
np.random.rand(784, 64), name="custom_kernel", dtype=tf.float32),
"model/dense/bias": tf.constant(
np.random.rand(64), name="custom_bias", dtype=tf.float32),
}
# Compute hiddens using a dense layer with custom variables.
x = tf.random.normal(shape=(1, 784), name="inputs")
with custom_make_variable(canonical_custom_variables, outer_scope="model"):
Layer = tf.layers.Dense(64)
hiddens = Layer(x)
print(Layer.variables)
Not entirely sure I understand your question correctly, but it seems to me that it should be possible to do what you want with a combination ofcustom layers and keras functional api .不完全确定我是否正确理解了您的问题,但在我看来,应该可以通过自定义层和keras 功能 api的组合来做您想做的事情。
Custom layers allow you to build any layer you want in a way that is compatible with Keras, eg:自定义层允许您以与 Keras 兼容的方式构建所需的任何层,例如:
class MyDenseLayer(tf.keras.layers.Layer):
def __init__(self, num_outputs):
super(MyDenseLayer, self).__init__()
self.num_outputs = num_outputs
def build(self, input_shape):
self.kernel = self.add_weight("kernel",
shape=[int(input_shape[-1]),
self.num_outputs],
initializer='normal')
self.bias = self.add_weight("bias",
shape=[self.num_outputs,],
initializer='normal')
def call(self, inputs):
return tf.matmul(inputs, self.kernel) + self.bias
and the functional api allows you to access the outputs of said layers and re-use them:和功能 api 允许您访问所述层的输出并重用它们:
inputs = keras.Input(shape=(784,), name='img')
x1 = MyDenseLayer(64, activation='relu')(inputs)
x2 = MyDenseLayer(64, activation='relu')(x1)
outputs = MyDenseLayer(10, activation='softmax')(x2)
model = keras.Model(inputs=inputs, outputs=outputs, name='mnist_model')
Here x1
and x2
can be connected to other subnets.这里x1
和x2
可以连接到其他子网。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.