如何使用 keras 后端将可训练的标量参数添加到张量流图中

Question

I want to approximate the function g(x) = exp(x) with a linear combination of functions, h(x) = sum_i(a_i * f_i(x)) using a neural network with TF2.

Now, the network input is just x and the outputs are the functions f_i.

The custom loss function is simple: the mean squared difference |g(x) - h(x)|^2.

My problem is that I don't understand how to define/use a_i?

First, all a_i are just scalars, In addition they don't depend on x.

Defining a_i as inputs doesn't make sense since I want them to be optimized.

Defining as outputs makes them depend on x which I don't want.

How can I add these variables as scalars to the network and make them optimized by the optimization process ?.

Answer 1

During training, Tensorflow tracks all tf.Variable objects defined in the model. If you define your constant as a tf.Variable , it will be ajusted via backpropagation.

Let's say we have a dataset wi X , and y which is y = X * 2 :

import tensorflow as tf

x = tf.random.uniform((10_000, 1))
y = x * 2

We will create a model which has a constant inside, which will need to replicate the relationship between X and y . We will of course initialize this value as something other than 2. The relationship between X and y is 2 so the training should make the constant converge towards 2. So let's define a model that is nothing but a constant.

class CustomModel(tf.keras.models.Model):
    def __init__(self):
        super(CustomModel, self).__init__()
        self.constant = tf.Variable(initial_value=0.1, dtype=tf.float32, trainable=True)

    def call(self, x, **kwargs):
        x = self.constant * x
        return x


model = CustomModel()

Now just compile and train the model:

model.compile(loss='mae')

history = model.fit(x, y, epochs=25, verbose=0)

Now look at the weights. The constant, which was initialized with value 0.1, is now 2. It was optimized, as it understood the relationship between X and y , which is 2.

model.weights

[<tf.Variable 'Variable:0' shape=() dtype=float32, numpy=2.0>]