2012-Sep-13 OLDER NEWER
Occlusion Video Chat

What if your webcam captured depth? The Kinect already does. Occlusion Video Chat tickles the question: "If we used such a technology to share the same space virtually, what would happen?"
The answer: lots of slow motion fist fights.

Download the Source

Occlusion Video Chat is available as a public repository on Bitbucket:

Occlusion Video Chat on Bitbucket

Photos

Watercolored posters on the wall

A couple of dancers. (Photo credit: Phil Smith)

Watercolored posters on the wall

Watercolored posters on the wall

Two bros posing together.

Two bros posing together.

My dad and Zach in a fist fight. (there were a lot of fist fights!)

My dad and Zach in a fist fight. (There were a lot of fist fights!)

The first time <em>Occlusion Video Chat</em> compiled with all of the features operating correctly.

The first time Occlusion Video Chat compiled with all of the features operating correctly.

Both cameras pointed at me from opposite directions.

Both cameras pointed at me from opposite directions.

An Homage to Z-sorting

The overall goal of Occlusion Video Chat was to make something that was personal, intimate, and inspiring to the imaginations of adults and children alike. The project was also subtly an homage to one of Computer Graphics' first major accomplishments: The solving of the "visibility problem" with Z-sorting. The Kinect has single-handedly made depth data accessible to the masses and therefore bridges one of the remaining gaps in the world of special effects. Special effects teams combine live footage with animated objects on a regular basis, painstakingly rotoscoping subjects because of the lack of depth information in their images. Occlusion Video Chat is a lo-fi example of how we may be approaching the end of an era.

Calibrating the Depth Map

Since Occlusion Video Chat takes advantage of both the color and the depth cameras on the Kinect, the two images needed to be synthetically aligned. Most applications blur the depth data slightly to achieve better results. I wanted nice sharp masks and to achieve the above level of detail. Using a technique similar to parallax mapping, the depth image was broken up into slices. Those slices were shifted to match the angle of the color camera, and then layered back on top of one another. To fill in holes, I used a background plate which is currently captured manually but could also be derived from the farthest recorded points on-screen.

holes versus no holes

A depth map with holes still present and my filtered depth map without holes.

Depth to Alpha Conversion

The above video shows two example video feeds A and B. Depth from feed A is converted into transparency before being sent over the network. This was because some streaming video protocols implement alpha support with MPEG.

Stability

For SPARKcon, for any other future gallery spaces, or for home use, stability was very important. I wanted to be able to leave Occlusion Video Chat to operate autonomously. The original concept was written in Processing in about three days but it ran very slowly. Refactoring in C++ took sixteen weeks. Instead of relying on the computers' CPUs, OpenFrameworks was picked as the best candidate for its ability to use the GPU. The remaining slowdowns are the result of problems I found with transmitting video over a network. Evan Kinney attempted to help me understand MPEG encoding but we have so far been unsuccessful. If you can help, please get in touch via Github.

On rare occasions when Occlusion Video Chat would break, it was because of network latency. We would usually take these opportunities to make clone photos.

Comparisons to Occlusion Photo Booth

Occlusion Video Chat is the second piece in the Occlusion Series. It was done entirely in my free time, and unlike the first installment Occlusion Photo Booth (OPB), Occlusion Video Chat received no grant funding. OCV was very successful in getting across the core concept without the need for explanation, whereas OPB had to be explained to patrons repeatedly. OCV drew from OPB thematically, but shared no other connection in its technology stack or its core concept.

Exhibitions

  1. Digital Motion Showcase @ Fish Market Student Gallery (2012-09-18)

Who Did What?

Wray Bowling Videography openFrameworks
Wray Bowling Shaders openFrameworks
Wray Bowling OOP openFrameworks