News: Google Researchers Develop Method for Capturing 'Relightable' Volumetric Video

As the demand for realistic volumetric video for AR experiences begins to grow (along with the available facilities and services for capturing it), researchers at Google have figured out how to improve upon the format.

The team has devised a system, called "The Relightables," which consists of a spherical cage holding 330 programmable LED lights and approximately 100 cameras designed to capture volumetric video.

• Don't Miss: Microsoft Launches Its First Mixed Reality Capture Studio in Hollywood

According to the team, the Relightables system achieves three innovations. First, the system's cameras operate as depth sensors, capturing 12.4MP depth maps of its subject. Second, the team created a geometric and machine learning reconstruction pipeline to synthesize video from the captured data.

Finally, the system also captures reflectance maps, time-synced to the depth maps, that enable the lighting of the subject to be manipulated in an AR or VR scene instead of maintaining the lighting from the original studio. This information is obtained by alternating flashes of two different color gradient patterns, which the system uses to infer the reflective properties of the subject during video reconstruction.

Having recently published the research via Association for Computing Machinery, the team is presenting its findings at SIGGRAPH Asia 2019.

"While significant progress has been made on volumetric capture systems, focusing on 3D geometric reconstruction with high-resolution textures, much less work has been done to recover photometric properties needed for relighting," wrote the team in its abstract for the paper.

"In contrast, a large body of work has addressed relightable acquisition for image-based approaches, which photograph the subject under a set of basis lighting conditions and recombine the images to show the subject as they would appear in a target lighting environment. However, to date, these approaches have not been adapted for use in the context of a high-resolution volumetric capture system. Our method combines this ability to realistically relight humans for arbitrary environments, with the benefits of free-viewpoint volumetric capture and new levels of geometric accuracy for dynamic performances."

The volumetric video segment has seen steady growth over the past two years. In addition to Microsoft's Mixed Reality Capture Studios, Sony has established its own studio, while Verizon acquired Jaunt for its dive into volumetric capture technology.

The use cases for volumetric video in augmented reality experiences, both now and in the future, are plentiful. The New York Times has demonstrated how volumetric video can enhance augmented reality content for immersive storytelling, while 8th Wall has extended support for the format to its web-based AR platform.

Meanwhile, Magic Leap's acquisition of Mimesys and its real-time holographic video calling technology shows how volumetric video capture will change how people will communicate in the near future.

But Google's Creative Lab, in collaboration with the Opera Queensland, has devised yet another use-case: virtual opera. The prototype experience, which debuts at SIGGRAPH, features three performers captured via the Relightables system.

With Google's contribution, the additional realism of 3D content from Relightables volumetric video will make it even harder to distinguish AR from reality.