Facebook Chief Mark Zuckerberg Talks Immersive Remote Video & AR Smartglasses Following Major Reveals from Snap & Google

The last few weeks were fairly busy in the realm of augmented reality and remote meetings developments. Most of that activity was generated by some mammoth announcements from Snap and its Spectacles AR smartglasses, and Google, with its Project Starline experimental holographic video conferencing system.

Now, right on cue, Facebook chief Mark Zuckerberg has emerged to give some (veiled) responses to some of what we've seen from the competition in the last few weeks.

• Don't Miss: Facebook CEO Zuckerberg Uses Latest Appearance to Triple Down on Augmented Reality & Creator Revenue-Focused Future

As soon as Google took the wraps off Project Starline, the first thing I thought was: Oh, here's our Facebook Portal challenger! Amazon's Echo Show hasn't really caught on in a big way, and most people use their Apple iPhone to FaceTime rather than a dedicated/stationary device. So it seemed like Facebook's family-friendly device might quickly take over this video conferencing void in the market.

That hasn't really happened, yet. But while some may have forgotten that Portal is a thing, Zuckerberg is very much pushing it forward.

"A lot of norms got established during the pandemic that I think are going to be around for a long time. So people are more likely to just casually want to do video calls with friends and do social gatherings that way when people can't be together," said Zuckerberg during a public chat on Instagram with Facebook Reality Labs boss Andrew Bosworth.

"We have an exciting roadmap of stuff that basically weaves into what I think are gonna be some of the new social norms around video presence. And that certainly connects to the bigger picture of all the AR and VR stuff that we do. A lot of people think about VR and AR like they're these technologies…really what they are is tools and platforms to deliver a sense of presence. And video does that, too. It's not ideal, it's 2D. When I see you on a screen, I can kind of get a sense of what's going on with you, but I sort of still need to trick my mind into feeling like I'm there with you. Whereas VR and AR really just make you, in a very native way, feel like you're there with the person. But video, for a lot of people, it's the best that we have for presence today. And we view this as all just kind of one extended product category."

It's not difficult to see that, based on these comments, Zuckerberg plans to continue to use the Portal device as a gateway drug to immersive experiences via AR. But the real question many are asking is: How will we interact with these virtual experiences in the near and distant future beyond the 2D screen? Well, it's here that the company's research into gesture control and neural interfaces—both teased by Facebook in recent years—take center stage once again.

"You'll have voice assistance, but you're not always gonna want to use voice because there are privacy issues with that. Hands are going to be a thing. But you're not always gonna be walking around with your hands outstretched in front of you doing stuff… I think controllers are going to be one interesting dimension of this, too. For things like writing, you want a stylus, it's super helpful. But in some ways, the holy grail of all this is a neural interface," said Zuckerberg.

Andrew Bosworth/Instagram

"Basically just think something and your mind kind of tells the computer how you want it to go… You don't actually want to read the person's mind, you're not trying to understand what they're thinking, what you're trying to do is give the person an ability to have their brain send signals to the rest of the body about how this works. We have a system that does this with motor neurons, where your brain basically sends signals to your hands and your body, telling them when you want to make movements, and how to control them. And it turns out that we all have some extra redundant capacity for that, right? It's part of the neuroplasticity. So if one pathway gets damaged, your brain can kind of get rewired. But you can train those extra pathways to control, for example, a second set of virtual hands."

This is a dynamic that those already familiar with the Oculus VR system are already accustomed to using. The focus on virtual hands in Oculus VR is paramount and quite effective. However, the idea of bringing this dynamic to AR is even more exciting.

"Basically you have your virtual hands, [they're] typing and controlling what you're doing in VR and AR, and then you don't need to actually have a physical controller or anything like that," said Zuckerberg. "So I think when you get to that, we're gonna have this whole kind of constellation of inputs. But that's perhaps one of the more ambitious projects that we have going on. But I think it's really promising long term."

It's around this point that the talk moves into the real goal of Facebook: turning its social media empire into a true operating system platform via the Trojan horse of VR and, more importantly, AR. This is also the area where we get to see, in a roundabout way, what he thinks of Snap's latest Spectacles play.

"If you're building a pair of [AR] glasses that need to look like normal glasses, you need to have the system be tightly optimized. So you can basically do all the computation that you would expect from a modern computer, but do it on someone's face within a thermal envelope in a power envelope that can last all day long," said Zuckerberg.

"That's a very big challenge. The team is pretty far along in this, and we're building a microkernel-based operating system, which is the architecture that you want to basically segment the pieces to make it as secure as possible… At the end of the day, we need to be able to design and customize every layer stack, in order to build out the performance and efficiency that we need in order to deliver these systems. We'll have more to share for developers and some folks at some point."

Perhaps the biggest reveal in the way of strategy is that fact that this new AR operating system will indeed be advertiser-powered, at least to an extent. Of course, we already knew this, but the Facebook founder is making it clear just how fundamental this aspect will be to making the AR hardware affordable enough to achieve mainstream escape velocity.

"If your mission is to serve as many people as possible, then fundamentally you're not trying to charge a premium for your devices," said Zuckerberg. "Or, you're trying to drive the price down as much as possible, including potentially even doing what consoles have historically done…they subsidize it a little bit upfront with the hope that they'll basically make that up on app sales and on other experiences in the rest of the economy around it. So you know, that's going to be more our plan, in VR and AR."

He went on to predict that the immersive industry will continue its current trajectory of being divided along wired and unwired hardware lines, with the tetherless devices likely winning the lion's share of consumer interest. "If you're trying to deliver a sense of presence, you don't want a wire wrapped around your neck, it really breaks the whole thing," said Zuckerberg. "So I think that that is going to be the bar for VR and AR products of kind of high quality going forward."

This is an interesting take considering the fact that at least two of the most high-profile AR hardware systems of the last few years, Magic Leap and, more recently, China's Nreal and 0glasses, all feature a tethered solution dedicated to delivering an immersive experience. However, the sheer physics of 2021 would seem to indicate that normal looking glasses with no tether delivering immersive AR is a high hurdle to traverse. Even Snap's Spectacles, while alluring in a futurist design sense, don't look like "normal glasses" and only have 30 minutes of battery life—a less than practical solution for mainstream usage.

So does this mean that Facebook has cracked the tetherless AR smartglasses code at its Seattle and Palo Alto labs? Are these the first hints at what Project Aria will evolve into?

"Some of the things that we have coming not too far down the line, which I think is gonna be really exciting, help you deliver the sense of presence. It's not just the spatial audio, it's like you physically feel like you're in a place, even if you're you're not actually physically together. It's really cool," said Zuckerberg. "It's tough because we think pretty long term. When you have a 10-year roadmap saying that something is coming kind of soon it might be easy to get people's hopes up. But, I mean, if we're doing demos in it, it can't be that far off."

Cover image via Facebook Reality Labs