Facebook's Oculus Quest Turned into Augmented Reality Guitar Hero Experience via This Developer App

The augmented reality passthrough abilities of Facebook's Oculus Quest were only fully opened to independent developers recently.

But despite the recency of the shift, the change is already yielding interesting AR applications that few would have considered for the primarily VR-centric device.

• Don't Miss: Facebook Finally Opens Oculus Quest Passthrough Feature to Developers, Making the VR Device an AR Playground

Austria-based indie developer Dominik Hackl has come up with a clever way to study playing the piano using AR overlays as a guide. The project, called Magic Keys, was first deployed by Hackl for the HoloLens as an experiment a couple of years ago.

"Similarly to Guitar Hero, the notes of any piece you would like to play [move toward the piano] keys in real-time, making it possible to play along [with] pieces without any prior practice or experience. I started working on this project about two years ago... intended to be a small AR app running on a tablet," says Hackl on his personal website.

"After it turned out that it could be of actual use, I ported the application to the HoloLens alongside with some major improvements. I continued to add features and tried to enhance the visualization of the music but I was greatly limited by the small field-of-view of the HoloLens. In early 2019, I finally got access to a brand-new Magic Leap One, which was a huge step forward and enabled me to add some cool new features, including in-app feedback on note hits or misses (using MIDI input and output)."

Now that Facebook has opened the Oculus Quest's passthrough function, Hackl decided to port his project to the wireless VR device, and the results are impressive. Hackl posted some of the test footage from the Quest version to Reddit to show off how it works.

Dominik Hackl/Reddit

If you're not familiar with the Oculus Quest and its passthrough function, essentially the device uses outward-facing cameras to help track controllers, your hands, and the environment around you. While in the VR environment, you have the option to turn the cameras on in a way that allows you to see your surroundings without taking the headset off.

The first hints at AR applications using the Quest passthrough function appeared in 2019 via the boundary guidelines that are overlaid on the real world to help you set up your play area. From that moment on it was fairly obvious that Facebook would likely begin testing out the AR possibilities through the wireless VR headset.

Now that developers like Hackl are beginning to test the Quest's AR limits, it's likely that these experimenters will also give Facebook a few ideas as the company continues to work on its dedicated AR smartglasses that will likely follow its Project Aria (aka Gemini) initiative.

Cover image via Dominik Hackl