分布式Tensorflow，Master在培训期间卡住了，工人在使用SyncReplicasOptimizer和MonitoredTrainingSession时没有开始培训？

Question

I am trying to write a synchronous training code in distributed tensorflow using SyncReplicaOptimizer and MonitoredTraining Session.

The problem I am facing is that the master hangs up the training in between after some steps and none of the workers start training. Has any one encountered this before?

This is the code I have written. Data is read from tensor flow records. I have followed the exact way described in the tensorflow website.

def build(self):
    self.modelObj = Model(self.imagesize, self.targetSize)
    self.modelObj.model()
    self.global_step = tf.contrib.framework.get_or_create_global_step()
    self.opt = tf.train.AdamOptimizer(self.learningrate)
    if self.syncTraining:
        self.trainer = tf.train.SyncReplicasOptimizer(self.opt,replicas_to_aggregate=self.num_workers,total_num_replicas=self.num_workers)
    else:
        self.trainer = self.opt
    self.trainstep = self.trainer.minimize(self.modelObj.loss, global_step=self.global_step)
    self.saver = tf.train.Saver(max_to_keep=1)
    self.summary_op = tf.summary.merge_all()
    self.init_op = tf.global_variables_initializer()
    if self.syncTraining:
        self.sync_replicas_hook = self.trainer.make_session_run_hook(is_chief = (self.task_index==0))


def train(self):
    if self.syncTraining:



        with tf.train.MonitoredTrainingSession(master=self.server.target,
                                               is_chief=(self.task_index==0),
                                               checkpoint_dir=self.logdir,
                                               hooks=[self.sync_replicas_hook]) as self.session:
            step = 0
            try:
                while not self.session.should_stop():
                    # training

                    [trainx, trainy_] = self.session.run([self.trainx, self.trainy_])
                    feed = {self.modelObj.x: trainx, self.modelObj.y_: trainy_,
                            self.modelObj.batch: self.batch_size, self.modelObj.keep_prob: 0.7}
                    _, trainloss = self.session.run([self.trainstep, self.modelObj.loss], feed_dict=feed)

                    print("step: %d, training loss %f" % (step, trainloss))

                    step += 1

            except tf.errors.OutOfRangeError:
                print('training finished, number of epochs reached')

Answer 1

Yes, the ps should not be placed on gpu. I also had this question. I solve it by explicitly claim ps_device="/job:ps/cpu:0" in tf.train.replica_device_setter. The whole code is like:

with tf.device(tf.train.replica_device_setter(
                                 ps_device="/job:ps/cpu:0",
                                 worker_device="/job:worker/task:%d" % (worker_index),
                                 cluster=cluster_spec)):

Many thanks to @prateek agrawal

Answer 2

Found a solution.

Delay the start of chief worker by adding

time.sleep(5)

Also, do the same for parameter server and try running the parameter server on CPU instead of GPU.