Forget Face Masks, Facebook Is Working on Full-Body AR Camera Effects

Jan 26, 2018 11:22 PM

Jan 26, 2018 11:24 PM

Facebook may have shamelessly copied Snapchat and its camera effects for faces (as well as its World Lenses), but it might beat its social media competitor to virtual body augmentation.

The AI Camera Team at Facebook has developed a method for detecting body poses and separating people in photos and videos from their backgrounds. In other words, the technology could make motion capture and green screen effects accessible to anyone with a Facebook app. Take that, Hollywood!

Facebook

The research team also envisions the technology being applied to gesture controls for video games (sort of like Kinect for mobile devices). It could also be used to visually anonymize people to protect their privacy.

As with AR face masks, the body mask technology relies heavily on computer vision and artificial intelligence. However, while face masks require tracking and the detection of the mouth, eyes, and other facial features, the full-body version must be able to detect body movements, poses, and clothing, as well as have the ability to identify multiple people in a scene.

Don't Miss:Snapchat Claps Back at Facebook by Letting Users Create Custom AR Lenses with Lens Studio

To accomplish this feat, the team used the Mask R-CNN framework for object detection and segmentation. On Monday, the team offered a detailed explanation of the technology via a blog post.

With the methodology in place, the team plans to work on "new model architectures" that can make the technology more efficient and lightweight for mobile devices.

Facebook

Consumers have a seemingly insatiable appetite for AR front-facing effects from Facebook and Snapchat, so expanding such effects to include full body capture could open up compelling opportunities for creative expression. Moreover, the method's potential for gesture recognition could make it a standard for future AR interfaces.

And while this just research for now, and there's no indication as to when this might make it into an actual app, it's nevertheless an exciting development to watch.