Blacklight Gaming
Overview
Camera-based interfaces such as the Sony Playstation's EyeToy are beginning to become popular
input devices for video games. Although computer vision algorithms to detect 3D depth using stereo cameras
are reasonably well known, this has not been done in gaming due to the high computation expense. In this
project, the computational power required is reduced by having the user wear a glove that glows under a blacklight.
The glow of a white cotton glove under a blacklight can be cheaply detected in a moderately dim room (afternoon on a cloudy
day or darker).
Inspiration
Sony EyeToy
NailJet Pro
Equipment
The equipment required is fairly simple. My setup contains the two cheapest USB cameras I could buy at Radio Shack
and a 24" blacklight purchased at Spencer's Gifts. The cameras give feeds of 320x240 at 15fps. In total, the equipment cost was about $70. I'm sure at a
commercial level, this could be produced much more cheaply. The processor power required from the computer makes
a moderate dent in it's overall load (about 30% of a P4 3.0 ghz), but this code was written in C# with no real
attempt at optimization, so I'm hopeful a modern game console could handle the load. The only equipment limitation
that worries me is the bandwidth on a USB hub. Presently, I need the cameras connected to different USB ports on the
computer motherboard because the bandwidth requirement is too high to connect them both to a single hub.
Cat Juggling
Cat Juggling is the prototype game for this interface. In the game, the user moves a 3D cube using hand movements
in front of the cameras. The cube is used to keep two falling cats juggled in the air. If the user lets a cat fall
it howls and is restarted in the air.
Implementation
This project was implemented in 4 days by one person. So, it is admittedly rough around the edges. The video capture
and analysis is built upon the DirectX.Capture library by Brian Low at The Code Project
and its frame capture modification by HP Ang. The 3D engine is the
Hazy Mind 3D Engine by Michael Schuld with some simple physics added.
The code also contains the AForge Neural Net library by Andrew Kirillov
but is isn't turned on. Finally, there is some code in there from my own Basics library which I haven't released, but contains code
by Tim Miller and Chris Maloney that was created in the process of writing ChemPad.
New Formulas
The detection algorithm basically reads the luminance of each pixel and thresholds it based on a value set
during calibration. To accurately detect the blacklight-powered glove, a new formula for luminance is
used that gives more power to the blue glow that white gloves produce under a blacklight.
luminance = red * 299 + green * 587 + blue * (114 + blueBonus)
At calibration time, the blue bonus is set along with the luminance threshold to filter out all pixels but the ones
glowing with the blacklight.
Additionally, although standard 3D depth detectors require intensive calibration of the cameras, generating a simple
3D depth detector only requires the cameras to be roughly pointed in the same direction. The depth is relative to
the disparity between the locations of the hand in each image. At calibration time, the user places their hand forward
to read the disparity at the "near" plane and then on the chest to read the disparity at the "far" plane. The 3D location
formula then becomes
Z = (disparity - near) / (far - near);
Future Possibilities
One of the original intentions of this project was to be able to capture user input and detect specific motions made
by the user. These pantomimes could be used to generate commands for another game to use as input. For example, in
a vampire game, a grasping motion could trigger the avatar's command to attempt to grab another avatar and feed off of
it. The program contains all the code necessary to forward keystrokes to another game running in the foreground, but
lacks the neural net necessary to do the recognition. I just ran out of time.
Also, it would be reasonable to create this interface to work with two hands and even the user's head for a wider range
of input possibilities.
Team Members
Just me, Dana Tenneson.