Technology

Discover how Virtual Drums works.

How, from the sticks' motion, can we reproduce realistic drum sounds depending on your positions, speed? How can we handle differentlighting , background behind the player, different distance between the player and the cameras? Discover the 7 main steps:

Step 1

Video acquisition of the 2 cameras simultaneously, at a frequency of 30Hz. At this step, image data are huge: 640 * 480 * 3 bytes * 2 cameras * 30Hz = 55Mb/s transferred on the firewire bus!

Step 2

Setup the camera parameters: gain, saturation, white balance, shutter speed. The resulting image resulting is then easier to analyse.

Step 3

Algorithms designed to process the colors and detect shapes are processing the images. At this step, we have to ignore the noise and all other colors but the sticks. We finally have the 2d positions of the sticks on the screen.

Step 4

Because we have the 2d positions of the sticks on the screen, and because we know the position of the cameras in 3 dimensions, we can determine via geometrical algorithms the 3d positions of the sticks. We also handle the distortion of the lenses.

Step 5

Once the sticks are detected, we process their speed. We also look for other pertinent information such as apparition frequency, percentage of theoretical object similarities, etc.

Step 6

Using a positioning algorithm, we determine if the stick is hitting a drum element. We then compute the sound localization and the volume, depending on the speed.

Step 7

The 3d Interface displays in real time the sticks and the whole drum kit. Drums are highlighted each time the player hits it. The fun is maximum!

More information

You can look at the presentation of the Virtual Drums project for the Shaastra festival at the IIT Chennai, in India. Download the presentation PDF

Discover Frozen Cameleon, the color calibration software especially developed for the virtual drums project.