Precise Pinch Hand Tracking in Virtual Reality
/When mounted to the front of a VR headset such as the Oculus DK2, the Leap Motion provides good enough hand and finger tracking to maintain immersion. However, the pinch-gesture requires user to be obtuse in order for the camera to recognize the hand clearly, which makes interacting with 3D objects frustrating. To improve this a mechanical switch mounted in a 3D printed ring provides a near 100% reliable input device. It's better suited for interacting with virtual objects consistently, while maintaining full hand and finger presence.
The end goal is browsing the web in Virtual Reality and the keyboard/mouse is required to do this. A ring doesn't get in the way when typing.
This is the first test of interacting with web "panels" by pointing and "clicking" with the ring device, and doing text input with a keyboard.
The best user interface device is our hand, adaptable and precise with no learning curve. Having to pick-up a controller just for interacting in the virtual world breaks immersion, while wearing a glove gets sweaty fast. Furthermore, my intent is to integrate keyboard/mouse into my virtual reality apps and wearing a glove when typing isn't the best. This is how I ended up on a ring with a mechanical button embedded inside. The first iteration (blue as seen below) was as simple as possible, and the second iteration aimed for as small as possible with the tools available to me.
For now the device is connected to the computer with a long USB cable. An Arduino flashed as a USB HID device enumerates to the computer as a joystick device and detects the button press. Ideally the entire device would be just a wireless ring, but I only have an ESP8266 wifi board on hand and the power draw would require a large battery anyway. It was fastest to build a wired version since I wasn't sure about the idea yet.
The mechanical button was pulled from a broken optical mouse. The wrist mount is a Fitbit band with a hole pierced through it. A 3D printed plug replaces the internal electronics and adds a mounting post for the Arduino case (also 3D printed). Thin gauge wires are used to connect the mechanical switch inside the ring to the wrist mounted Arduino, with globs of hot glue as wire strain relief. Moving the ring from one finger to the next is convenient for testing.
The most comfortable finger to press on was the middle finger as my thumb needs to stretch the least to reach it. This poses minimally annoying issues when the hand tracking becomes confused and the virtual hand glitches out. The Leap Motion has trouble tracking the fist-like shape of the thumb pressing on the middle finger, as compared to it more easily recognizing the pinch gesture. The best hand tracking occurred when the ring-button was on the thumb, which allows for fast and precise enough control, as seen in the video. Pinching your fingers around virtual objects feels the most intuitive when grabbing virtual objects, so this lesson inspired the following open-ring design.
Along with making it all wireless I'm excited to experiment with using the button on the forefinger and being able to press on real world objects. Being able to switch input devices and making it all modular also seems obvious to pursue. Maybe magnetic quick disconnect connectors...