TensorFlow: How to perform image categorisation on multiple images

Question

Hoping that somebody can help out with a TensorFlow query. It's not a difficult one, I'm sure. I am just somewhat lacking in knowledge relating to TensorFlow and NumPy.

Without any prior experience of TensorFlow I have implemented Python code from a tutorial for doing image classification. This works. Once trained, it can tell the difference between a cat and a dog.

This is currently hard-wired for a single image. I would like to be able to classify multiple images (ie the contents of a folder), and do this efficiently. What I have done so far in an effort to achieve this is to simply add a loop around everything, so it runs all the code for each image. However, timing the operation shows that classification of each successive image takes longer than the previous one. Therefore there is some kind of incremental overhead. Some operation is taking more time with every loop. I cannot immediately see what this is.

There are two options to improve this. Either:

(1) Leave the loop largely as it is and prevent this slowdown, or

(2) (Preferable IMHO, if it is possible) Pass a list of images to TensorFlow for classification, and get back a list of results. This seems more efficient.

This is the code:

import tensorflow as tf
import numpy as np
import os,glob,cv2
import sys,argparse
import time

try:

  inputdir = [redacted - insert input dir here]

  for f in os.listdir(inputdir):

    start_time = time.time()

    filename = os.path.join(inputdir,f)

    image_size=128
    num_channels=3
    images = []
    image = cv2.imread(filename) # read image using OpenCV

    # Resize image to desired size and preprocess exactly as done during training...
    image = cv2.resize(image, (image_size, image_size),0,0, cv2.INTER_LINEAR)
    images.append(image)
    images = np.array(images, dtype=np.uint8)
    images = images.astype('float32')
    images = np.multiply(images, 1.0/255.0)

    # The input to the network is of shape [None image_size image_size num_channels]. Hence we reshape.
    x_batch = images.reshape(1, image_size,image_size,num_channels)

    sess = tf.Session() # restore the saved model
    saver = tf.train.import_meta_graph('dogs-cats-model.meta') # Step 1: Recreate the network graph. At this step only graph is created
    saver.restore(sess, tf.train.latest_checkpoint('./')) # Step 2: Load the weights saved using the restore method
    graph = tf.get_default_graph() # access the default graph which we have restored

    # Now get hold of the op that we can be processed to get the output.
    # In the original network y_pred is the tensor that is the prediction of the network
    y_pred = graph.get_tensor_by_name("y_pred:0")

    ## Feed the images to the input placeholders...
    x= graph.get_tensor_by_name("x:0") 
    y_true = graph.get_tensor_by_name("y_true:0") 
    y_test_images = np.zeros((1, 2)) 

    # Create the feed_dict that is required to be fed to calculate y_pred...
    feed_dict_testing = {x: x_batch, y_true: y_test_images}
    result=sess.run(y_pred, feed_dict=feed_dict_testing)
    # Note: result is a numpy.ndarray
    print(f + '\t' + str(result) + ' ' + '%.2f' % (time.time()-start_time) + ' seconds')

  # next image

except:
  import traceback
  tb = traceback.format_exc()
  print(tb)

finally:
  input() # keep window open until key is pressed

What I tried to do to modify the above was to create a list of filenames using...

images.append(image)

...and then taking the rest of the code out of the loop. However, this didn't work. It resulted in the following error:

ValueError: cannot reshape array of size 294912 into shape (1,128,128,3)

At this line:

x_batch = images.reshape(1, image_size,image_size,num_channels)

Apparently this Reshape method doesn't work (as implemented, at least) on a list of images.

So my questions are:

1. What would causing the steady increase in image classification time that I have seen as images are iterated?

2. Can I perform classification on multiple images in one go, rather than one-by-one in a loop?

Thanks in advance.

Answer 1

Your issues:

1 a) The main reason why it is so slow is: You are re-creating the graph for every image.

1 b) The incremental overhead is coming from creating a new session every time without destroying the old session. The with syntax helps with that. eg:

with tf.Session(graph=tf.Graph()) as session:
  # do something with the session

But that won't be a noticable issue after addressing a).

When thinking about the problem, one might realise which parts of your code depend on the image and which don't. The TensorFlow related part that is different per image is the call to session.run , feeding in the image. Everything else can be moved out of the loop.

2) You can also classify multiple images in one go. The first dimension of x_batch is the batch size. You are specifying one. But you may exhaust your memory resources trying to do that for a very large number of images.