Object recognition with CNN, what is the best way to train my model : photos or videos?

Question

I aim to design an app that recognize a certain type of objects (let's say, a book) and that can say whether the input is effectively a book or not (binary classification).

For a better user experience, I would like the input to be a video rather than a picture: that way, the user won't have to deal with issues such as sharpness, centering of the object... He'll just have to make a "scan" of the object, without much consideration for the quality of a single image.

And there comes my problem: As I intend to create my training dataset from scratch (the true object I want to detect being absent from existing datasets such as ImageNet),

I was wondering if videos were irrelevant for this type of binary classification and if I should rather ask the user to take a good picture of the object.

On one hand, videos have the advantage of constituting a larger dataset than one created only from photos (though I can expand my picture's dataset thanks to data augmentation) as it is easier to take a 10s video of an object rather than taking 10x24 (more or less…) pictures of it.

But on the other hand I fear the result will be less precise, as in a video many frames are redundant and the average quality might not be as good as in a single, proper image.

Moreover, I do not intend to use the time property of a video (as in a scan the temporality is useless) but rather working one frame at a time (as depicted in this article ).

What is the proper way of constituting my dataset? As I really would like to keep this “scan” for the user's comfort and if images are more precise than videos in such a classification is it eventually possible to automatically extract a single image from a “scan”, and working directly on it?

Answer 1

Good question: The answer is. you should train your model on how you plan to use it, So if you ask the user to take photos. train it on photos, If you ask the user to film the object. train on frames extracted from video.

The images might seem blurry to you, but they won't be for a computer. It will just learn to detect "blurry books", but that's OK, that's what you want.

Of course this is not always the case. The image might become so blurry that the information whether or not there is a book in the frame is no longer there. Where is the line? A general rule of thumb: if you can see it's a book, the computer will also see it. As I think blurry images of books will still be recognizable as books, I think you could totally do it.

Creating "photos (single image, sharp)" from "scan (more blurry, frames from video)" can be done, it's called super-resolution. But those models are pretty beefy, not something you would want to run on a mobile device.

On a completely unrelated note: try googling Transfer Learning: It will benefit you for sure.D.