Canvas pixels #3: getUserMedia

getUserMedia() is a proposal for one of the most desired device APIs that can give HTML and JS access to the user's camera and microphone. It's already available in Chrome Canary in the form of navigator.webkitGetUserMedia(). It's also available in Opera without a prefix.

In part #1 of this miniseries I talked about manipulating pixels in canvas, let's do the same but this time using video data from your own webcam instead of a static image.

Demo

I've only tested in Chrome so if you want to see the demo, you need to:
1. install Canary
2. go to chrome://flags and Enable PeerConnection

If you don't want to go through the trouble, here's a snapshot of what you're missing: a little video element showing your video camera stream and 4 canvas elements where the image data is manipulated in some way.

And the demo

Hooking up the cam

Getting a video stream is pretty straightforward:

navigator.webkitGetUserMedia(
  {video: true},
  iCanHazStream,
  miserableFailure
);

You declare what type of media you want (video in this case) and provide success and failure callbacks. The browser then prompts the user to allow access:

If the user allows, your success callback is called.

Here's mine:

function iCanHazStream(stream) {
  var url = webkitURL.createObjectURL(stream);
  $('video').src = url;
  webkitRequestAnimationFrame(paintOnCanvas);
}

I have a <video id="video"> element on the page and I set its src to be the stream URL. (In Opera you assign the stream directly, not some made up URL. In webkit the URL ends up being something like https://www.phpied.com/files/canvas/blob:http%3A//www.phpied.com/c0d155b9-f4f8-4c4f-b2bc-694de68d74f2. Anyway, not terribly important)

So this is all you need to do in order to display the camera stream in a VIDEO element. Easy right?

Then I have another function paintOnCanvas() which I schedule with the new requestAnimationFrame hotness instead of old school setInterval()

Setting up the canvas

For the image manipulation I'm using the same CanvasImage() constructor from part #1.

During page load I initialize 4 canvas elements with a placeholder image.

var transformadores = [
  new CanvasImage($('canvas1'), 'color-bars.png'),
  new CanvasImage($('canvas2'), 'color-bars.png'),
  new CanvasImage($('canvas3'), 'color-bars.png'),
  new CanvasImage($('canvas4'), 'color-bars.png')
];

And I have 4 simple pixel manipulators like you saw already:

var manipuladors = [
  {
    name: 'negative',
    cb: function(r, g, b) {
      return [255 - r, 255 - g, 255 - b, 255];
    }
  },
  {
    name: 'max blue',
    cb: function(r, g, b) {
      return [r, g, 255, 255];
    }
  },
  {
    name: 'max red',
    cb: function(r, g, b) {
      return [255, g, b, 255];
    }
  },
  {
    name: 'noise',
    cb: function(r, g, b) {
      var rand =  (0.5 - Math.random()) * 50;
      return [r + rand, g + rand, b + rand, 255];
    },
    factor: '(0 - 500+)'
  }
];

Painting on the canvas

Finally, the paintOnCanvas() function. Here's what happens:

function paintOnCanvas() {
  var transformador = transformadores[0];
  transformador.context.drawImage(
    $('video'), 0, 0, 
    transformador.image.width, transformador.image.height
  );
  var data = transformador.getData();
  for (var i = 0; i < 4; i++) {
    transformador = transformadores[i];
    transformador.original = data;
    transformador.transform(manipuladors[i].cb);
  }
  webkitRequestAnimationFrame(paintOnCanvas);
}

First we need to take the image data from the video element and draw it on a canvas. Then read the image data from the canvas, play with it and paint it back. This seems like a hassle, there might be an easier way to get image data from the video or the stream without going stream-video-canvas, but I don't know it. In any event I only do it once for the first canvas, then remember this data and use it for all the 4 canvases.

It's surprisingly easy to draw a video data in canvas, just using context.drawImage(video_dom_element, ...). From there I read the image data into data and loop through the 4 canvas instances, transforming the image using one of the manipulators I have set up.

Once again, for your entertainment, the demo is right here.

Comments? Feedback? Find me on Twitter, Mastodon, Bluesky, LinkedIn, Threads