detecting musical shapes with OpenCv on an android device

Question

i have an assignment on one of my classes. i need to "make" a musical instrument with a phone. I was thinking of making something like this:

there is a musical staff on a wall (or a board). then, the user can stick musical notes on that staff and point their phone camera to that wall. then the phone detects the musical notes (that could be crotchet or a half note, even a G cleff) and plays that succession of notes as sounds. the musical notes will have the same size and color because they will not be drawn, we will make them out of paper or cardboard.

my concern is if it's possible to detect several custom known shapes (in this case a bunch of musical notes), and their positioning regarding another one (in this case the lines on the staff, or the staff itself) using openCv; if not, any other idea of how could do this would be great.

maybe you could point me the direction on what to look for in google to get me started on this matter. i have 2 and a half months to do this project ( if it's relevant for you to know this ), so i would appreciate any help. thank you in advance.

Answer 1

What you're asking is a big picture idea of how this can be solved. Here are some thoughts to get you started.

First, you need to identify where the musical note in the image is. If you know the camera is always going to be static, you could do some background subtraction technique. A more rigorous approach would be to identify musical notes in the image.

Have a look at the MNIST dataset. It has a bunch of handwritten numbers and the corresponding ground truth. You could start here and extend this to musical notes. To begin, you can use k-nearest to classify these images.