How to restore multiple models and average them in tensorflow?

Question

I have trained two models separately and I want to load their variables and average them. But it goes wrong with tf.get_default graph()

Here is my code structure (I know it is wrong but how to write correctly?)

sess = tf.session()
saver_one = tf.train.import_meta_graph('./model1.ckpt.meta')
saver_one.restore(sess,'./model1.ckpt')
graph_one = tf.get_default_graph()
wc1 = graph_one.get_tensor_by_name('wc1:0')
……
saver_two = tf.train.import_meta_graph('./model2.ckpt.meta')
saver_two.restore(sess,'./model2.ckpt')
graph_two = tf.get_default_graph()
wc1_two = graph_two.get_tensor_by_name('wc1:0')
……

and the errors come are:

 Traceback (most recent call last): File "/home/dan/Documents/deep-prior-master/src/ESB_ICVL_TEST_ALL.py", line 143, in <module> saver_two.restore(sess,'./cache/cnn_shallow/model2.ckpt') File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/saver.py", line 1548, in restore {self.saver_def.filename_tensor_name: save_path}) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 789, in run run_metadata_ptr) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 997, in _run feed_dict_string, options, run_metadata) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1132, in _do_run target_list, options, run_metadata) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1152, in _do_call raise type(e)(node_def, op, message) tensorflow.python.framework.errors_impl.InvalidArgumentError: Assign requires shapes of both tensors to match. lhs shape= [27] rhs shape= [9] [[Node: save/Assign_6 = Assign[T=DT_FLOAT, _class=["loc:@outb"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/gpu:0"](outb, save/RestoreV2_6/_1)]] 

Thank you very much for give me any advice. =(^.^)=

Answer 1

You are trying to overwrite the graphs and it runs into mismatches (some dimensions don't match). Probably better to have them separated.

graph_one = tf.Graph()
with graph_one.as_default():
  session_one = tf.Session()
  with session_one.as_default():
    saver_one = tf.train.import_meta_graph('./model1.ckpt.meta')
    wc1_one_value = session_one.run([graph_one.get_tensor_by_name('wc1:0')])

# Similar for graph_two
...

print (wc1_one_value + wc1_two_value) / 2  # Or whatever you want

To assign them back into a session, construct the graph, then execute tf.assign operations.

with graph_one.as_default(), session_one.as_default():
   session_one.run([tf.assign(<variable>, (wc1_one_value + wc1_two_value) / 2 )])

To get the variable you can use get_trainable_variables or define it again with reuse=True . Afterwards export the model again.