Category Archives: Development

Chuckles

Background

About 18 months ago, I had the idea of using Javascript to take live audio input and make it animate the mouth of a virtual ventriloquist dummy. The original seed of the idea was to play a prank on someone at SydJS. After a bit of research I was disappointed to find out there wasn’t any way to do it. The idea was pushed aside, but never completely forgotten.

Since then, there has been an explosion in new web technologies and standards. The ones that really piqued my interest were the Web Audio API and WebRTC.

The Web Audio API provides fine-grained control over sound processing and manipulation in JS.
WebRTC allows for peer-to-peer video communication between browsers, along with the associated access to webcams and microphones. Bingo.

Time to code

At the time I started playing with WebRTC, the few browsers that had implemented it only supported getting a webcam stream; microphone support was yet to come. I figured I could still work on getting the idea right using an HTML range input.

The first implementation was simple. I found a picture of a ventriloquist dummy online and hard-coded the image and drawing data, then set the position of the dummy’s mouth to be bound to the value of the input.

Then came the part I’d been waiting for: Google Chrome enabled getting microphone data in their Canary build.

All I had to do was get access to the microphone via the navigator.getUserMedia() API, pipe that input into a Web Audio API AnalyzerNode, then change the position of the dummy’s mouth based on the maximum audio level detected in the microphone. One final adjustment was made to lock the dummy’s mouth movement to regular “steps”, in order to give it a more old-fashioned wooden feel.

And thus was born the first demo of the library that has been christened “Chuckles”.

VIEW BASIC DEMO

More interaction

While the first version worked well enough, it still required hard-coding of all the data. So I built in some new display modes to make it more interactive:

  • Normal mode, the default state
  • Drawing mode, where you can draw on the image to define where the “mouth” segment is
  • Dragging mode, where you can drag the “mouth” around to set its position when fully open

A quick addition of drag-and-drop for adding your own image and you too can make your own ventriloquist dummy:

VIEW FULL DEMO

Next steps

The code is on GitHub at https://github.com/gilmoreorless/chuckles, and there are a few more things I’d like to do with it at some point.

  • Better definition for the mouth segment (possibly using a border).
  • Allow transforming the mouth segment using rotate, skew, etc.
  • Define eye regions that can move left/right independently of the mouth.

But adding those features all depends on whether I can be convinced the idea is actually useful, rather than just being a throw-away demo.

CSS degrees – when standards aren’t standard

While reading through an article about CSS3 gradients the other day, I noticed that the angular degrees for linear gradients increase in an anti-clockwise direction. This seemed odd because the CSS3 rotation property increases degrees clockwise.

To check that this inconsistency was indeed the case, I made a quick test case: http://jsfiddle.net/gilmoreorless/9SVvm/1/

It looks to me like the two modules of the new specification were developed in complete isolation from each other, which doesn’t give me confidence in the “new saviour” of CSS3 standards. If the makers of the specification can’t be consistent, how can we expect the same of the browser implementations?