How to combine sequential operations with side effects in TensorFlow

Question

I'm developing a GAN in TensorFlow. Currently the training schedule is

feed_dict = ...
sess.run(discriminator_train_op, feed_dict);
sess.run(generator_train_op, feed_dict);
sess.run(generator_train_op, feed_dict);

We train the generator twice each step because we find that it results in better stability.

Now I want to combine the operations together so I only need to feed the network once, as feeding is slow in Tensorflow. I tried

with tf.control_dependencies([discriminator_train_op]):
    train_op = tf.group(generator_train_op);
with tf.control_dependencies([train_op]):
    train_op = tf.group(generator_train_op);

Supposedly control_dependencies specify one operation must happen after another. But the profiling timeline shows that certain gradient descent done in generator are parallel to those in discriminator. In other words, the order is not enforced. In addition, I find out by adding debug statements in the network that the combined train_op trains only the generator once, not twice.

Is there any way that I can move the control of this sequence of operations from Python to Tensorflow?

Answer 1

with tf.control_dependencies([discriminator_train_op]):
    train_op_g1 = tf.group(generator_train_op);
with tf.control_dependencies([train_op_g1]):
    train_op_g2 = tf.group(generator_train_op);

sess.run([discriminator_train_op, train_op_g1, train_op_g2], feed_dict)