有什么办法可以将tf.train.Checkpoint与MonitoredTrainingSession一起使用吗？

Question

Now I am trying to use tf.train.Checkpoint instead of tf.train.Saver in various situations.

When I tried to use Checkpoint with MonitoredTrainingSession, I noticed that Checkpoint.save changes tf.Graph. Therefore, I cannot use Checkpoint as follows.

a = tf.Variable([3], name="a")
checkpoint = tf.train.Checkpoint(model=a)

with tf.train.MonitoredTrainingSession() as sess:
    checkpoint.save(file_prefix="parameter/ckpt", session=sess)

I got an error "RuntimeError: Graph is finalized and cannot be modified".

Can I use Checkpoint with MonitoredTrainingSession?

Answer 1

I found a weird technique, which call checkpoint.write with a temporary session to construct a graph preliminarily.

with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    checkpoint.save("./parameter/sample")

with tf.train.MonitoredTrainingSession() as sess:
    checkpoint.save(file_prefix="parameter/ckpt", session=sess)