Tag Archives: sydjs

Webcam tile sliding game

Another WebRTC experiment shown at SydJS

My first experiments with WebRTC were during the early phases of its development, at a time when there were very few demos and references to work from. One of those early demos was Exploding Pixels (part of Opera’s “Shiny Demos” to coincide with the release of Opera 12) which inspired my very first WebRTC experiment.

We all know the old tile-sliding games that are often given out as cheap toys to kids, where you’re presented with a jumbled picture or pattern and need to slide the pieces around one-by-one until the puzzle is complete.
I wanted to build one of those using JavaScript, but with a live webcam feed as the picture.

The first step was to build a simple tile game library that took some basic parameters and handled the interaction and game state.

var tiler = new Tiler('game', {
    width: 640,
    height: 480,
    rows: 3,
    cols: 3


Hooking up a webcam

Next came the fun part — integrating with WebRTC. To do this I build the library in such a way that the game could be redrawn at any stage with a different background image. It also accepted a parameter for the background that could be a URL path to an image, an image element or a canvas element.

Then it was a simple matter of hooking up the webcam using navigator.getUserMedia, updating a hidden canvas with the video stream (I didn’t use a video element directly as I wanted to flip the image so it acted like a mirror), then setting it as the game’s background image repeatedly.

var tiler = new Tiler('game', {
    width: 640,
    height: 480,
    rows: 4,
    cols: 4
var video = document.getElementById('webcam');
var source = document.getElementById('source');
var ctx = source.getContext('2d');
navigator.getUserMedia({video: true}, function (stream) {
    // A custom helper method to set the webcam stream
    // as the video source - different for each browser
    setStreamSrc(video, stream);
    // Flip the video image so it’s mirrored
    ctx.translate(640, 0);
    ctx.scale(-1, 1);
    (function draw() {
        ctx.drawImage(video, 0, 0);
}, function () {
    console.error('Oh noes!');


Extra difficulty

Taking advantage of the fact that these aren’t real tiles, I added some extra difficulty by randomly flipping or rotating some tiles during setup. It obviously means that the game needs a “selection mode”, where clicking a tile selects it (at which point it can be flipped or rotated) and a double-click moves it.

var tiler = new Tiler('game', {
    width: 640,
    height: 480,
    rows: 4,
    cols: 4,
    move: 'dblclick',
    flip: true,
    rotate: true

This works for both static images and live webcam feeds.



All the code for these demos is on Github at https://github.com/gilmoreorless/experiments/tree/gh-pages/tiles

NOTE: The usual caveats apply for these demos (i.e. They were built as quick experiments for a presentation and are not guaranteed to work properly in all browsers). I’ve tested in Chrome, Opera and Firefox as they were the first browsers to enable WebRTC.



About 18 months ago, I had the idea of using Javascript to take live audio input and make it animate the mouth of a virtual ventriloquist dummy. The original seed of the idea was to play a prank on someone at SydJS. After a bit of research I was disappointed to find out there wasn’t any way to do it. The idea was pushed aside, but never completely forgotten.

Since then, there has been an explosion in new web technologies and standards. The ones that really piqued my interest were the Web Audio API and WebRTC.

The Web Audio API provides fine-grained control over sound processing and manipulation in JS.
WebRTC allows for peer-to-peer video communication between browsers, along with the associated access to webcams and microphones. Bingo.

Time to code

At the time I started playing with WebRTC, the few browsers that had implemented it only supported getting a webcam stream; microphone support was yet to come. I figured I could still work on getting the idea right using an HTML range input.

The first implementation was simple. I found a picture of a ventriloquist dummy online and hard-coded the image and drawing data, then set the position of the dummy’s mouth to be bound to the value of the input.

Then came the part I’d been waiting for: Google Chrome enabled getting microphone data in their Canary build.

All I had to do was get access to the microphone via the navigator.getUserMedia() API, pipe that input into a Web Audio API AnalyzerNode, then change the position of the dummy’s mouth based on the maximum audio level detected in the microphone. One final adjustment was made to lock the dummy’s mouth movement to regular “steps”, in order to give it a more old-fashioned wooden feel.

And thus was born the first demo of the library that has been christened “Chuckles”.


More interaction

While the first version worked well enough, it still required hard-coding of all the data. So I built in some new display modes to make it more interactive:

  • Normal mode, the default state
  • Drawing mode, where you can draw on the image to define where the “mouth” segment is
  • Dragging mode, where you can drag the “mouth” around to set its position when fully open

A quick addition of drag-and-drop for adding your own image and you too can make your own ventriloquist dummy:


Next steps

The code is on GitHub at https://github.com/gilmoreorless/chuckles, and there are a few more things I’d like to do with it at some point.

  • Better definition for the mouth segment (possibly using a border).
  • Allow transforming the mouth segment using rotate, skew, etc.
  • Define eye regions that can move left/right independently of the mouth.

But adding those features all depends on whether I can be convinced the idea is actually useful, rather than just being a throw-away demo.

Presentation: WebRTC / Web Audio at SydJS

I gave a presentation about WebRTC and the Web Audio API (titled “WebRTC: Look Ma, no Flash!”) at the SydJS tech meetup on February 20, 2013.

There is no video available of the presentation (fortunately or unfortunately, depending on your perspective), but the slides are online at http://gilmoreorless.github.com/sydjs-preso-webrtc.

The presentation was really just me showing a heap of examples and demos of using WebRTC and Web Audio to do fun things. Most of the examples were written by other people, but a few of them were mine, showing things I’ve been doing in my experiments repository.

Instead of just being lazy and leaving my experiments hidden away, I’m going to force myself to write some blog posts explaining some of them and how they work.

The first one should be up tomorrow. Should.