Complex Hand-Tracking Is the Key to Better Mixed Reality Interfaces

Keyboards and mice work fine for computers, but in a holographic environment you'll want to do more than just point, click, and type. While we can still benefit from these input devices, complex hand-tracking methods are necessary for the evolution of mixed reality user interfaces.

Microsoft's HoloLens only has two gestures: air tap and bloom. It's not hard to learn them both, but some struggle with air tapping because the headset looks for a specific motion that doesn't always feel natural. Once you get the hang of it, you can generally get by, but even then you'll run into some trouble when dragging and adjusting. The gestures just don't offer enough precision right now.

Fortunately, Microsoft is well aware of the problem and has found efficient ways to track and understand natural hand movements. As you'll see in the video below, the tracking method is remarkably efficient and demonstrates how software can already understand complex movements. When put in context, those movements are easy to understand as a button push, lever pull, or whatever else developers can dream up.

This level of interaction provides an important metaphor for the user. While holograms can't provide tactile feedback, the immediate visual response is enough to create a vastly more immersive experience. If you can touch a rendered object and see it react accordingly, your natural movements gain far more meaning.

In mixed virtual reality setups like the Void (and, to a lesser extent, the HTC Vive), hand and motion tracking can do even more. Headsets can see physical objects in the room and render similar ones in the virtual environment. Through this matchup, you'll touch a physical object in the real world but you'll see the rendered version in your headset. This can effectively fool your brain into feeling like the virtual world is a real one.

But the language of gesture isn't limited to holographic headsets. As Microsoft demonstrates in the video above, hand tracking can help expedite common tasks. We've already seen how multitouch trackpads can help provide better and often more efficient control of the computing environment. This is just an untethered evolution that should breathe new life into the world of webcams.

Right now, Microsoft is only scratching the surface of what's possible, but it's already feeling awfully close to the common dream of a Minorty Report-style interface.