Tensorflow: Saving model and restoring individual weights

Question

My question is: I save a model. Now I want to change the model (add or remove some layers), but still import the weights that I previously trained. How can I restore individual parts of the model, while initializing new parts?

To be more specific:

I have a tensorflow model with weights that looks like this:

global_step = tf.get_variable("global_step", shape=[1],
                  trainable=False, initializer=tf.constant_initializer(1))

#TODO FUTURE: Time-video as input, for possible Seq2Seq model
#TODO: Add "regularizer=None"
Weights = {
    "W_Conv1": tf.get_variable("W_Conv1", shape=[3, 3, 1, 64],
            initializer=tf.random_normal_initializer(mean=0.00, stddev=0.01),
    ),
    "W_Conv2": tf.get_variable("W_Conv2", shape=[3, 3, 64, 64],
            initializer=tf.random_normal_initializer(mean=0.00, stddev=0.01),
    ),
    #not sure if we sum over all 64 channels?!
    "W_Local1": tf.get_variable("W_Local1", shape=[1, 16 * 8 * 64],
            initializer=tf.random_normal_initializer(mean=0.00, stddev=0.01),
    ),
    "W_Local2": tf.get_variable("W_Local2", shape=[1, 16 * 8 * 64],
            initializer=tf.random_normal_initializer(mean=0.00, stddev=0.01),
    ),
    "W_Affine1": tf.get_variable("W_Affine1", shape=[16*8*64, 512],
            initializer=tf.random_normal_initializer(mean=0.00, stddev=0.01),
    ),
    "W_Affine2": tf.get_variable("W_Affine2", shape=[512, 128],
            initializer=tf.random_normal_initializer(mean=0.00, stddev=0.01),
    ),
    "W_Affine3": tf.get_variable("W_Affine3", shape=[128, 10],
            initializer=tf.random_normal_initializer(mean=0.00, stddev=0.01),
    )
}

#TODO: Do local layers have a bias term?
Bias = {
    "b_Conv1": tf.get_variable("b_Conv1", shape=[1, 16, 8, 64],
            initializer=tf.random_normal_initializer(mean=0.00, stddev=0.01),
    ),
    "b_Conv2": tf.get_variable("b_Conv2", shape=[1, 16, 8, 64],
            initializer=tf.random_normal_initializer(mean=0.00, stddev=0.01),
    ),
    "b_Local1": tf.get_variable("b_Local1", shape=[1, 8192],
            initializer=tf.random_normal_initializer(mean=0.00, stddev=0.01),
    ),
    "b_Local2": tf.get_variable("b_Local2", shape=[1, 8192],
            initializer=tf.random_normal_initializer(mean=0.00, stddev=0.01),
    ),
    "b_Affine1": tf.get_variable("b_Affine1", shape=[1, 512],
            initializer=tf.random_normal_initializer(mean=0.00, stddev=0.01),
    ),
    "b_Affine2": tf.get_variable("b_Affine2", shape=[1, 128],
            initializer=tf.random_normal_initializer(mean=0.00, stddev=0.01),
    ),
    "b_Affine3": tf.get_variable("b_Affine3", shape=[1, 10],
            initializer=tf.random_normal_initializer(mean=0.00, stddev=0.01),
    )
}

Assume I save this model using the tf.train.Saver() . Now, I want to run this model. However, at some point in future, I might want to change the model, but keep the weights that I trained so far. How can I individually restore certain weights? To be more specific, how can I restore the model again , if I add the line

tmpval = tf.get_variable("new_var", shape=[1],
             initializer=tf.constant_initializer(0.1))

to the model-weights defined above?

I am a beginner with tensorflow, so any advice and ideas are welcome. Thanks in advance! :)

Answer 1

I think that your question is covered in TF tutorial 'Choosing which variables to save and restore'

# Create some variables.
v1 = tf.Variable(..., name="v1")
v2 = tf.Variable(..., name="v2")
...
# Add ops to save and restore only 'v2' using the name "my_v2"
saver = tf.train.Saver({"my_v2": v2})
# Use the saver object normally after that.
...