Auto lips sync according to mp3/wav

Question

I'd like to create animated heads in my web apps. It seems that CSS3 transition, animation and background features with a little help of javascript web API is all I need. Using xface looks like an overkill to me, cartoon solutions is almost all I need. I need to make it cartoon.

I've made some progress already (beeing able to create voice controlled web app), but this time I need mp3/wav input, not direct voice from microphone using google servers through x-webkit-speech .

I am considering this approach:

1. record speech into mp3 or wav and write it's string contents
2. play the mp3 in browser and detect end of words using AnalyserNode to synchronize position in the string (I use Czech language which, unlike the English, has almost constant speech speed).
3. display the cartoon heads (see the link above) according to actual spoken letter

The question: Is there any lower effort (shorter development time for coder and designer) approach? Especially step 2 and English language in the future makes me worried. Maybe some karaoke tool could produce some speech sync file (which can I parse into CSS3 keyframes)? I am not aware of any such tool.

Answer 1

For something more involved you might try:

Step 1. Web speech API to text to voice...

http://updates.html5rocks.com/2013/01/Voice-Driven-Web-Apps-Introduction-to-the-Web-Speech-API

Step 2 try porting "papagayo" to js (uses dictionary to relate words to phonemes to mouth poses I believe)

http://anime.smithmicro.com/papagayo.html

The GNU source is available here: http://anime.smithmicro.com/update_files/papagayo/papagayo_1.2_source.zip

You might also refer to: http://www.adobe.com/devnet/flash/articles/lip-sync-smartmouth.html for an overview of what you're trying to achieve

Answer 2

也许您可以通过频谱分析快速而又轻松地完成一些工作：http: //0xfe.muthanna.com/wavebox/