NR30: The AR Software Development Leaders of 2018

One could argue that, at least for the moment, software development is more important to the augmented reality experience than hardware. Since a viable augmented reality headset has yet to emerge for the broader, mainstream consumer market, currently, the same devices that make texting and selfies possible are leading the charge to enable easy-to-use AR experiences.

Even some utilitarian smartglasses targeting enterprise businesses run on Android, which is yet another nod toward the supremacy of existing mobile platforms.

Maybe this is more a testament to how powerful mobile devices have become, but even the most powerful headset is just a paperweight without a compelling software experience. Consider the breadth of augmented reality experiences that are available today, and it becomes evident that it's not easy to make a genuinely magical AR app. It takes technical prowess, creative content, design competency, business savvy, great ideas, and overall execution. We've seen great ideas fall flat on their faces through lackluster content. A flawlessly-coded app can still fail to gain traction due to a convoluted interface.

Previous: NR30's Top AR Hardware Leaders to Watch in 2018
Next Up: NR30's Top AR Investment Leaders to Watch in 2018

Which leads to another complexity of developing AR, namely, just because we can project digital content into the real world doesn't mean that we always should depend on the real world context. The same game that's entertaining to a user on a 2D screen may not have the same impact in AR.

Nevertheless, the 3D content tools used to create realistic video games and immersive virtual reality experiences are, for the most part, the same ones used to build augmented reality experiences. As a result, Unity and Unreal are the engines behind the majority of AR apps. As these engines become more sophisticated, the AR experiences they power become more capable of twisting our world into the fantastical. At the same time, tools are emerging that are democratizing development and making it easier for non-coders to create AR experiences as simply as they would put together a PowerPoint or Keynote presentation.

The app developers on this list have demonstrated their understanding of the difficulties involved in balancing the qualities of a good app and how to apply those attributes to augmented reality. From the apps to the tools that developers build them on, the following are the NR30 software developers, executives, and entrepreneurs who are making our AR dreams a reality.

Tony Parisi — Unity

In some ways, Tony Parisi (pictured in the cover image above), Unity's global head of VR/AR brand solutions, represents the collective conscious of the AR and VR community. A veteran of immersive computing, Parisi pioneered the 3D web years before anyone outside the lab had seriously considered using AR and VR in everyday settings. As the co-creator of the Virtual Reality Markup Language (VRML) in the '90s, he helped give many developers and end users their first taste of what it could be like to interact with virtual objects in personal computing environments.

Today, Parisi sits atop the most popular development platform for truly immersive AR and VR experiences, while simultaneously acting as a cheerleader for immersive computing with a passion and engagement that belies his many years working within the space. Whether you're working with the HoloLens, Magic Leap, and Meta 2, or the Oculus Rift and HTC Vive, there's a strong chance you're using Unity. According to Unity, more than "60% of all AR/VR content [has been] developed with Unity."

Last year, the company quickly added support for both ARKit and ARCore, and earlier this year, it launched an official partnership with Magic Leap to help smooth the way for developers adopting the new AR headset. Back in 2015, before the Oculus GO, and before ARKit, Parisi wrote a post wondering why no one was building the "iPod Touch of VR." Even then, it was clear to Parisi that mobile was the future of immersive computing. And similar to his prescience from decades past regarding VR, Parisi is already positioning Unity for the day when mobile immersive computing is ubiquitous.

"I don't want my primary phone to be stuck in a VR headset," wrote Parisi. "I want it to make calls, and all the other stuff it already does. So I'd rather have another device." What that device will be is unknown — Magic Leap, Apple Glasses, or something else — but one thing is clear, whatever it is, Parisi will be there, likely a few steps ahead.

We spoke to Parisi about what drives his dedication to the AR space. Below are his lightly edited responses.

Describe the first time you experienced AR or VR?

I was privileged to see a demo of the VPL system when VR pioneer Jaron Lanier visited my wife's art school in 1991! It was crude — very low polygon scenes and a very large, cumbersome device — but it showed the possibilities. Just a few years later, inspired by VPL, I got into VR myself, though by way of working on technology that brought 3D to the web without headsets (VRML).

The moment you knew AR was the future?

Even though I've worked in immersive tech for over two decades, until recently, I had a lot of skepticism about widespread consumer use of AR. Sure, there were compelling industrial and enterprise use cases, but a lot of the consumer-facing work I was seeing fell flat, literally, they were just 2D images overlaid on the camera ...
All the good stuff (true 3D AR) was happening on high-end headsets like the HoloLens and Meta hardware, which meant that not nearly enough consumers would be able to experience it. It wasn't until the announcements of ARKit and ARCore last summer that the lightbulb went on for me. Now we have a true AR technology that can scale to reach everyone with 3D content seamlessly integrated into the world around us, using the device you already have in your pocket.

The thing you dislike most about currently available AR hardware or software?

Currently, I have two frustrations, both of which I hope will be solved soon. First, with AR headsets like the HoloLens, Meta, and Magic Leap, we are still early on: each system has limitations that prevent it from being ready for all-day use, and they're still a bit high in price. With phone-based AR, the price is right, but the form factor also limits prolonged use, i.e., holding up the phone for an AR experience is something you can do for maybe a minute or two, about the same amount of time you can use your phone to capture a video or live stream an event. The answer to both of these frustrations is going to be smartglasses ...
Then you'll have a form factor that allows for all-day use, at consumer price points. I'm no hardware expert, but predictions range from three to 10 years as to when such glasses will be readily available. Until then, we'll keep working with what we have, learning and refining the craft as creators, and educating the consumer and enterprise marketplaces about the wonders and benefits of 3D and augmented reality.

What's the most important thing that happened in AR in the last 12 months.

Without a doubt, it was the introduction of ARKit and ARCore. With these technologies, we have the first platform that can deliver immersive content at scale to both the Apple and Android ecosystems. Yes, it's limited compared to what can be done with a headset, or eventually with smart glasses. But it runs on the phone in your pocket today.

Michael Abrash — Facebook

While Hugo Barra leads Facebook's augmented reality efforts from a business perspective, Michael Abrash, chief scientist of Facebook Reality Labs, oversees the development of the technology itself.

Interestingly, Abrash earned his undergraduate degree in geography from Clark University in the 1970s, though he spent much of his spare time (and, by his own admission, some of the time he should have been studying) on programming. He continued his studies in the Ph.D. program for energy management and policy at the University of Pennsylvania, where he also began to develop video games. After receiving an $8,000 royalty check from Datamost, Inc. for his first published game, Space Strike, he dropped out and began his career in video game development, which includes the seminal game Quake among its credits.

On March 28, 2014, Abrash was introduced as the chief scientist for Oculus, following his Quake cohort John Carmack, who serves as the company's chief technology officer. Just three days before Abrash's arrival, Facebook had announced its intention to acquire Oculus.

In May of this year, Oculus Research changed its name to Facebook Reality Labs, reflecting its mission to develop augmented and virtual reality technology for Oculus and all of Facebook. Much of the group's published research still focuses on VR, but innovations such as six degrees of freedom tracking, advanced signal processing for light field characteristics, and facial performance sensing for head-mounted displays bodes well for Facebook's eventual foray into AR smartglasses, for which the group has already developed a patent.

Kyle Roche — Amazon

Before 2017, Amazon's augmented reality efforts had been relatively meager compared to other top tech companies, namely, Microsoft, Google, Apple, and Facebook. Nonetheless, Jeff Bezos' company arrived on the scene in a big way in November of last year with the unveiling of Amazon Sumerian, which is overseen by general manager Kyle Roche.

Roche, who holds a bachelor's degree in mathematics from the University of Mexico and another in information technology from the University of Phoenix, co-founded 2lementary with Chris Chiappone in 2011. The Denver-based company launched a platform for enterprise connected devices in 2013, with Honeywell among its more notable customers. According to TechCrunch, the startup also developed a facial recognition technology that triggers actions based on identified age and gender.

Amazon acquired 2lemetry in 2015 and folded the company into its Amazon Web Services organization; both founders remained on board, with Chiappone serving as software development manager.

The duo led the development of Sumerian, with job postings hinting in June 2017 that the platform would be coming soon.

Roche has demonstrated his knowledge on the subject of AR as far back as 2011, when he published a book titled Pro iOS 5 Augmented Reality for Apress. In the book, Roche guides readers in creating augmented reality apps using MapKit, accelerometer, magnetometer, facial recognition, and Facebook data.

The Sumerian platform, which is now open to all interested users, is among a growing segment of development tools designed for non-programmers to create immersive experiences.

Keiichi Matsuda — Leap Motion

Most of us got our introduction to the mind of Keiichi Matsuda back in 2016 when he released Hyper Reality, the concept film that painted a future world full of augmented reality advertisements that covered entire cities in a nightmarish kaleidoscope of virtual advertising imagery. Since then, the short film has become a staple of conversations about AR and what it may look like in one possible future. The film was so impactful that it led to Matsuda joining the team at Leap Motion in the waning months of 2017 as the company's vice president of design and global creative director.

Working within a large team of innovators at Leap Motion, Matsuda has become one of the front-facing windows into what the company is creating. In recent months, Matsuda has been tweeting early peeks at some of the AR experiments the company is working on, from stunning interface developments to open source platforms that will allow anyone to construct their own AR device via Project North Star. Although he's just one part of a team working on these developments, it's Matsuda's penchant for imagining the future of AR that has gripped many in the immersive computing community.

Now, with his talents situated within the Leap Motion team, which was already known for delivering cutting-edge immersive computing innovations, Matsuda has found the perfect platform to imagine bigger and, in the process, help the rest of us imagine along with him.

We spoke to Matsuda about his imaginative approach to AR and some of the inspirations behind his vision. Below are his lightly edited responses.

Favorite movie depicting the future of AR or VR?

A lot of the aesthetics for VR were established in the '80s and '90s with Tron, The Lawnmower Man, and Star Trek's holodeck. My 15-year-old mind melted when The Matrix came out. Although I'm really nostalgic for all of these films, I think my favorite take on VR is probably in Summer Wars, as it's not built as a proxy to our physical world, but exists as something entirely new. There are so many great movies in the genre, but VR novels beat VR films. Vernor Vinge's True Names is a favorite of our CTO David Holz, and required reading at Leap Motion.

The thing you dislike most about currently available AR hardware or software?

It's frustrating to me that the industry is so hardware-obsessed. That the experience of AR sometimes feels like an afterthought. To me, this feels strange and has led to an industry with no strong product vision or purpose. We need to make the technology accessible to more developers and designers outside of the tech giants. And we need to invest time to properly investigate the kinds of experience we want to build, instead of rushing into a land grab to set up walled garden ecosystems and app stores.

Robert Sumner — Disney Research

Disney chairman and CEO Bob Iger may set the company's direction for preferring augmented reality over VR, but Robert Sumner, associate director at Disney Research, is driving the company's innovation of AR technology.

At Disney, Sumner heads up research in animation and games, including augmented reality. According to the Disney Research website, Sumner and his team endeavor to "bypass technical barriers in animation and game production pipelines with new algorithms that expand the designer's creative toolbox in terms of depiction, movement, deformation, stylization, control, and efficiency."

In this capacity, Sumner has co-authored several publications for research into augmented reality innovations, including AR face projection, a mobile app that lets users re-color paintings in AR, and intelligent AR characters that can react to real-world obstacles.

Sumner's work in advancing technology also extends outside of Disney's doors. After studying computer science at the Georgia Institute of Technology, Sumner earned his M.S. and Ph.D. from the Massachusetts Institute Technology (MIT). Before joining Disney, Sumner worked as a postdoctoral researcher at ETH Zurich, where he now serves as an adjunct professor for the Game Programming Laboratory. He founded the ETH Game Technology Center in 2015; a group focused on fostering research, education, and outreach in game technology.

Robert leads the lab's research in animation and games. His research group strives to bypass technical barriers in animation and game production pipelines with new algorithms that expand the designer's creative toolbox with respect to depiction, movement, deformation, stylization, control, and efficiency.

We spoke to Sumner about what sparked his path toward working on the cutting edge of immersive computing. Below is his lightly edited response.

Describe your first time experiencing AR or VR?

My first experience was around 1996. I was a student volunteer at the SIGGRAPH computer graphics conference and had the chance to try a virtual reality experience in Disney's exhibition space where you fly a magic carpet through the animated world of Aladdin. Even in this early day, the experience was incredible, and I've never forgotten it. Fortunately, a research paper describing the concept is still available.

Jeff Powers — Occipital

Long before there was ARKit and the TrueDepth camera on the iPhone X, there was Occipital, the computer vision company co-founded by Jeff Powers that gives iPhones and iPads the capabilities of the HoloLens.

Backed with venture capital from Techstars, Powers and co-founder Vikas Reddy opened Occipital's doors in 2008. By 2010, the company created RedLaser, a barcode scanning app for the iPhone, which he sold to eBay in 2010.

In 2013, the company launched the Structure Sensor, a 3D sensor that gives Apple mobile devices spatial mapping abilities. The product's Kickstarter campaign earned nearly $1.3 million, far surpassing its $100,000 goal. The company followed that up with the Bridge headset. When combined with the sensor and an iPhone, the Bridge becomes a head-mounted display for pass-through augmented reality experiences. The company also released the Structure SDK so that developers could easily build apps with six degrees of freedom tracking and real-world occlusion and 3D scanning apps for photogrammetry and home improvement.

Now that ARKit exists, Powers and company continue to apply their computer vision acumen via TapMeasure. Using ARKit, the app enables users to not only measure rooms in augmented reality but also export those measurements as CAD drawings. The company has also diversified its portfolio with the acquisition of 3D scanning company Paracosm.

Powers graduated from the University of Michigan, where he studied computer science, engineering intelligent systems, and electrical engineering. Aside from his work in AR, he also acts as an angel investor in technology, with investments including mobile robotics company Orbotix and the TechStars Boulder accelerator.

We spoke to Powers about the motivations behind his bold moves to shape the future of AR. Below are his lightly edited responses.

Describe your first time experiencing AR or VR?

I had two firsts in VR: one in 1995 with a product called i-Glasses by a company named Virtual I/O. They were 640x480 mini CRT displays in front of your eyes, with a 28 degree FoV, and cost thousands. I still remember playing Descent and getting butterflies in my stomach thinking about flying up and down through virtual tunnels. This was a device way ahead of its time, and it would be years before GPUs could actually keep up with the promise of something like this.
My second first was with the Oculus Rift and the Tuscany demo scene in 2014. The low resolution was more than offset by the immersion with a high-angle display. I felt actually afraid to step off of ledges and wanted to accelerate my work developing ways to capture the world and bring it into VR.

What is your favorite movie depicting the future of AR or VR?

Altered Carbon has a cold but perhaps realistic future of AR/VR where you can fluidly move between not only virtual bodies, but also real ones. It is a good series to watch if you want to figure out what we need to prevent from happening as VR and life converges. If you're a Westworld fan, they're pretty similar.

When was the moment you knew AR was the future?

Circa 2005, I was reading a paper by a researcher named Marc Pollefoys where he was showing 3D reconstructions of statues from only a sequence of photographs. It clicked to me then that AR would solve some fundamental problems like enhancing our memories by reconstructing objects and then displaying things back to us on top of the real world.

If you were making this list, who would be one person you'd be sure to include on it?

I would include Mark Billinghurst, AR pioneer and investor.

Alban Denoyel — Sketchfab

It takes a lot to drive the immersive computing space forward, and one of the most passionate and hardworking startup founders in the space is Alban Denoyel. Originally from France, and now based in New York City, Denoyel's company Sketchfab specializes in punching above its weight. With a tight-knit team of developers, Denoyel's company has managed to amass over 1.5 million registered users and 6.5 million monthly visitors. Those users access over 2 million 3D models that they can integrate into AR and VR applications and experiences, and the number of virtual assets is only growing.

But rather than resting on the traction he's already earned with Sketchfab, Denoyel and his team can smell the impending explosion of immersive apps, and ramped up their efforts to a new level in the last year. Just months ago, the company launched the Sketchfab Store, a vast trove of 3D content that allows creators to earn cash from their creations, while simultaneously adding a more professional resource for app makers beyond the free content provided via Sketchfab.

Not long after that, Denoyel also announced a partnership with Facebook that integrates Sketchfab's download API into Facebook AR Studio. Although there are larger players in the 3D resource space, such as Google and Microsoft, Denoyel has kept the company nimble and on the cutting edge of nearly every development in immersive computing, which is even more impressive considering that much of this work is being done in relative obscurity as AR has yet to see its true breakout moment. But when it comes to the future of AR, Denoyel has made sure that Sketchfab will figure prominently in the overall picture.

We spoke to Denoyel about what sparked his relentless efforts to smooth the path of AR development for developers and artists. Below are his lightly edited responses.

Describe the first time you experienced AR or VR?

I experienced mobile AR for the first time in 2012. I experienced actual AR with a very early prototype of the HoloLens, about two years before Microsoft unveiled the device.

Favorite movie depicting the future of AR or VR?

I love the movie "Kingsman: The Secret Service," and I think the part where they do a team meeting with AR glasses is a good way of depicting the future of AR.

What's the most important thing that happened in AR in the last 12 months.

Probably ARKit.

Matt Miesnieks — 6D.Ai

As a managing partner at Super Ventures and CEO of AR startup 6D.ai, Matt Miesnieks contributes to the augmented reality landscape from the finance and software development angles. But it's his role with 6D.ai, an AR cloud platform maker, that excites the AR community.

Emerging earlier this year, 6D.ai can enable persistent, shared AR experiences with just a smartphone camera. The platform constructs a real-time dense mesh from crowdsourced data, even while working in the background, with performance as good as some depth sensors. Users can apply the technology in 3D mapping and multi-player AR gaming. Recently, Miesnieks shared the platform's occlusion capability, which puts the platform on par with the Niantic Real World Platform, and beyond the features available in either ARCore or ARKit.

At Super Ventures, Miesnieks applies his industry knowledge to help startups target their offerings to the appropriate markets. Miesnieks has raised approximately $15 million for augmented reality companies over the past five years and served as an advisor to three startups, 30Mhz, Yuno Juno, and Fantasmo, another AR cloud company.

A computer science graduate from Wollongong University, Miesnieks has an extensive resume in augmented reality. Notably, he previously led the augmented reality research and development team at Samsung. He also co-founded and served as the CEO of Dekko, makers of 3D AR content for use in iOS apps. Miesnieks also handled worldwide customer development of Layar, leading to the AR app coming as a pre-loaded feature on most Android smartphones shipping in 2010. And, like other AR pioneers on the NR30, he has a few potentially important patents pending devoted to AR technology.

We spoke to Miesnieks about what's behind his decision to work on the frontier of AR technology. Below are his lightly edited responses.

When was the moment you knew AR was the future?

A while before I discovered Layar. I was thinking about how interfaces were becoming more sensory, and phones were becoming more wearable — I realized that whoever captured the interface between our sense of sight and the Internet would be the ultimate winner (until brain interfaces).

If you were making this list, who would be one person you'd be sure to include on it?