Tag Archives: webaudio

Chuckles

Background

About 18 months ago, I had the idea of using Javascript to take live audio input and make it animate the mouth of a virtual ventriloquist dummy. The original seed of the idea was to play a prank on someone at SydJS. After a bit of research I was disappointed to find out there wasn’t any way to do it. The idea was pushed aside, but never completely forgotten.

Since then, there has been an explosion in new web technologies and standards. The ones that really piqued my interest were the Web Audio API and WebRTC.

The Web Audio API provides fine-grained control over sound processing and manipulation in JS.
WebRTC allows for peer-to-peer video communication between browsers, along with the associated access to webcams and microphones. Bingo.

Time to code

At the time I started playing with WebRTC, the few browsers that had implemented it only supported getting a webcam stream; microphone support was yet to come. I figured I could still work on getting the idea right using an HTML range input.

The first implementation was simple. I found a picture of a ventriloquist dummy online and hard-coded the image and drawing data, then set the position of the dummy’s mouth to be bound to the value of the input.

Then came the part I’d been waiting for: Google Chrome enabled getting microphone data in their Canary build.

All I had to do was get access to the microphone via the navigator.getUserMedia() API, pipe that input into a Web Audio API AnalyzerNode, then change the position of the dummy’s mouth based on the maximum audio level detected in the microphone. One final adjustment was made to lock the dummy’s mouth movement to regular “steps”, in order to give it a more old-fashioned wooden feel.

And thus was born the first demo of the library that has been christened “Chuckles”.

VIEW BASIC DEMO

More interaction

While the first version worked well enough, it still required hard-coding of all the data. So I built in some new display modes to make it more interactive:

  • Normal mode, the default state
  • Drawing mode, where you can draw on the image to define where the “mouth” segment is
  • Dragging mode, where you can drag the “mouth” around to set its position when fully open

A quick addition of drag-and-drop for adding your own image and you too can make your own ventriloquist dummy:

VIEW FULL DEMO

Next steps

The code is on GitHub at https://github.com/gilmoreorless/chuckles, and there are a few more things I’d like to do with it at some point.

  • Better definition for the mouth segment (possibly using a border).
  • Allow transforming the mouth segment using rotate, skew, etc.
  • Define eye regions that can move left/right independently of the mouth.

But adding those features all depends on whether I can be convinced the idea is actually useful, rather than just being a throw-away demo.

Presentation: WebRTC / Web Audio at SydJS

I gave a presentation about WebRTC and the Web Audio API (titled “WebRTC: Look Ma, no Flash!”) at the SydJS tech meetup on February 20, 2013.

There is no video available of the presentation (fortunately or unfortunately, depending on your perspective), but the slides are online at http://gilmoreorless.github.com/sydjs-preso-webrtc.

The presentation was really just me showing a heap of examples and demos of using WebRTC and Web Audio to do fun things. Most of the examples were written by other people, but a few of them were mine, showing things I’ve been doing in my experiments repository.

Instead of just being lazy and leaving my experiments hidden away, I’m going to force myself to write some blog posts explaining some of them and how they work.

The first one should be up tomorrow. Should.