在卷积神经网络训练期间定期评估大型测试集

Question

I created a small convolutional neural network with TensorFlow that I want to train.

During the training, I want to record several metrics. One of them is the accuracy on a test set independent of the training set.

The MNIST example shows me how to do it:

  # Train the model, and also write summaries.
  # Every 10th step, measure test-set accuracy, and write test summaries
  # All other steps, run train_step on training data, & add training summaries

  def feed_dict(train):
    """Make a TensorFlow feed_dict: maps data onto Tensor placeholders."""
    if train or FLAGS.fake_data:
      xs, ys = mnist.train.next_batch(100, fake_data=FLAGS.fake_data)
      k = FLAGS.dropout
    else:
      xs, ys = mnist.test.images, mnist.test.labels
      k = 1.0
    return {x: xs, y_: ys, keep_prob: k}

  for i in range(FLAGS.max_steps):
    if i % 10 == 0:  # Record summaries and test-set accuracy
      summary, acc = sess.run([merged, accuracy], feed_dict=feed_dict(False))
      test_writer.add_summary(summary, i)
      print('Accuracy at step %s: %s' % (i, acc))
    else: # Record train set summarieis, and train
      summary, _ = sess.run([merged, train_step], feed_dict=feed_dict(True))
      train_writer.add_summary(summary, i)

What it does is feeding the whole of the test set into the evaluation at every 10 steps, and print out this accuracy.

That's pretty cool and all, but my test set is quite a bit larger. I have about 2000 "images" of dimension 30x30x30x8, so feeding the all of this dataset into the evaluation at once would blow up both my core memory and the GPU memory.

As a workaround, I have this:

accuracy = mymodel.accuracy(logits, label_placeholder)

test_accuracy_placeholder = tf.placeholder(tf.float32, name="test_accuracy")
test_summary = tf.scalar_summary("accuracy", test_accuracy_placeholder)


# training loop
for batch_idx in enumerate(batches_in_trainset):

    #do training here
    ...

    # check accuracy every 10 examples
    if batch_idx % 10 == 0:

        test_accuracies = []  # start with empty accuracy list

        # inner testing loop
        for test_batch_idx in range(batches_in_testset):
            # get testset batch
            labels, images = testset.next_batch()

            # make feed dict
            feed_dict = {
                # ...
            }

            # calculate accuracy
            test_accuracy_val = sess.run(accuracy, feed_dict=test_feed_dict)

            # append accuracy to the list of test accuracies
            test_accuracies.append(test_accuracy_val)

        # "calculate" and log the average accuracy over all test batches
        summary_str = sess.run(test_summary,
                               feed_dict={
                                   test_accuracy_placeholder: sum(test_accuracies) / len(test_accuracies)})

        test_writer.add_summary(summary_str)

Basically, I first collect all the accuracies on the test set batches and then I feed them into a second (disconnected) graph that calculates the average of those batches.

This "kind of" works, in the sense that I am indeed able to calculate a test set accuracy on the required intervals.

However, this feels very awkward and has the serious drawback that I cannot record anything else other than the test set accuracy.

For example, I would like to also record the loss function value on the whole test set, the activation histograsm on the whole test set, and maybe some other variables.

Preferably this should work just like in the MNIST example. Check out the TensorBoard demo here: https://www.tensorflow.org/tensorboard/index.html#events

In this summary, all variables and metrics are evaluated both on the test and training set. I want that too! But I want that without somehow feeding the complete test set into my model.

Answer 1

It looks like this function was added with streaming metric evaluation (contrib).

https://www.tensorflow.org/api_guides/python/contrib.metrics