Classify New Objects in Images Live

Question

I have a webcam, a microphone and a python GUI. The user shows the camera an object and using voice command asks "what is this object?". The webcam takes a photo of the camera frame and pushes it to a flask endpoint. I have a VGG16 model hosted in the flask app which responds with a object class based on the image from camera.

What I want to do now is, if the object is not recognised, a learning cycle should be triggered. In this cycle I will tell the model what the object is (voice to text), which will be the label for the object. I got that working.

What's not working is the next time (after the learning is complete), if I show the object to the camera, the model should be able to tell me what this object is.

Could someone please advise me on the following:

1. Is VGG16 (trained initially on 2 objects using transfer learning) the best model for this type of task? Currently, it is classifying unseen objects as one of the two classes.

2. How would you go about implementing this solution on the cloud (AWS, Azure etc.)

Thank you.

Answer 1

1. Is VGG16 (trained initially on 2 objects using transfer learning) the best model for this type of task? Currently, it is classifying unseen objects as one of the two classes.

Firstly the reason for "it is classifying unseen objects as one of the two classes " is simply that, you only allow it to classify between the two classes, so what happens is that even if you show it an unknown object it is supposed to fit it on either of the classes and post its best prediction, what you should be do is that train it on 3 distinctive classes {object1, object2, unknownObject}, so you may correctly predict unknown objects, now this may be a bit problematic, and would require retraining the model, the other thing that you could do is that you should set a threshold, basically when you give it an unseen object it could give the prediction confidence as follows {75%, 25%} or maybe {51%, 49%} you should set a threshold here, that unless your model is more than 90-95% sure of the prediction you take it to be an unknown object, now i am in no way saying this should be your threshold value, you threshold could either be .9 / .95 as mentioned above, or it could even be .75, tha t is a hyper parameter and you should employ ways to figure that threshold out

2. How would you go about implementing this solution on the cloud (AWS, Azure etc.)

You already have a flask server, just deploy the flask server on an AWS machine, and make it directly accessible via a static public external IP, hence you can post/get a request to that specific IP and run your model from any interface