tensorflow在队列训练时进行评估？

Question

I preprocessed my data as tfrecord. I feed my data by queue instead of feed_dict.

This is my code.

for i in range(100000000):
    sess.run(train_op)
    start_time = time.time()
    if i > 20 and i % 10 == 0:
        summaries_train = sess.run(Summaries)
        writer_train.add_summary(summaries_train, i)
        print("%dth batch took %fs, have scanned %d of %d" % (i, time.time()-start_time, i*cf.batch_size, all))
        print("loss: {0}, acc: {1}".format(sess.run(Loss), sess.run(Acc)))

My Question is: How can I evaluate while training?

When using queues, I don't have to write feed_dict. So How can I feed evaluating data to tensorflow ?

Answer 1

You can do something like eval_in_batches function here but where you take data from queue instead of feed_dict . Note that you will want to have a separate queue so that your evaluation doesn't take examples from training.

Another common pattern to evaluate while training is to start a separate CPU-only process which loads checkpoints and runs the evaluation continuously.

Answer 2

You can separate evaluation from training and even run it on a completely different machine/process. On your training machine (which has only training data) you periodically save your model to disk.

Your evaluation machine checks the folder with your models and once a new model appears it loads it and runs evaluation.

Answer 3

All is the same as the training process except that you should 1) separate the training data and evaluation data; 2) don't run the optimization operation, aka gradient descent. I hope this function may be of help for you.