JavaScript：检测画布中的形状

Question

I found out app called https://skinmotion.com/ and for learning purposes, I would like to create my own, web version of the app.

Web application work as follows. It asks user for permission to access camera. After that, video is caputer. Once every second, image is taken from the stream and processed. During this process, I look for soundwave patern in the image.

If the pattern is found, video recording stops and some action is executed.

Example of pattern - https://www.shutterstock.com/cs/image-vector/panorama-mini-earthquake-wave-on-white-788490724 .

Idealy, it should work like with QR codes - even small qrcode is detected, it should not depend on rotation and scaling.

I am no computer vision expert, this field is fairly new to me. I need some help. Which is the best way to do this?

Should I train my own Tensorflow dataset and use tensorflow.js? Or is there easier and more "light weight" option?

My problem is, I could not find or come up with algorithm for processing the captured image to make as "comparable" as possible - scale it up, rotate, threshold to white and black colors, etc.

I hope that after this transformation, resemble.js could be used to compare "original" and "captured" image.

Thank you in advance.

Answer 1

With Deep Learning

If there are certain waves patterns to be recognized, a classification model can be written using tensorfow.js.

However if the model is to identify waves pattern in general, it can be more complex. An object detection model is to be used.

Without deep learning

Adding to the complexity, would be detecting the waveform and play an audio from it. In this latter case, the image can be read byte by byte. The wave graph is drawn with a certain color that is different from the background of the image. The area of interest can be identified and an array representing the wav form can be generated.

Then to play the audio from the array, it can be done as shown here