Apple's Latest Patent Generates AR Content from Real World Images

Christmas came early for Apple, as the company was awarded more than 40 patents by the US Patent and Trademark Office on Tuesday, including one covering a system for compositing an augmented reality construct based on image recognition.

The patent (US 9,824,495) covers a system for capturing a real-world image, detecting an object in that image, "geometrically reconstructing" a virtual representation of that object, and then placing that virtual object in the real-world view as an AR object.

Apple's Latest Patent Generates AR Content from Real World Images — *Images via USPTO*

Various methods and applications for this system are described in the patent documentation. For example, it could be used to embed an owner's manual or assembly instructions into a product's image, provide information about a country on a map, or present an architect's rendering based on scanned design plans.

Discovered via Patently Apple, the awarded patent comes from work by Netanel Hagbi and Oriel Bergig from Flyby Media, the AR company acquired by Apple for the ARKit platform, along with Israeli computer scientist Jihad El-Sana.

While patents for future technology often leave the implementation up to the reader's imagination, in this case, we have actual video footage that demonstrates the patented system in action.

El-Sana presented research described in this patent in a seminar at the University of Canterbury in 2012. During his presentation, he demonstrated how AR content can be embedded into hard copies of a ski map and then rendered based on the camera's recognition of that map. This example is presented in Fig. 4 of the published patent, although the patent goes on to describe how a skier can be shown traveling down the mountain based on simulated physics.

Don't Miss: Apple Acquires AR Hardware Maker to Jumpstart Its Own Headset Aspirations

Further into his presentation, El-Sana described how computers can recognize shapes and poses of an icon and extract the 3D model that corresponds to it. He demonstrated this method through a video, which is still available on YouTube. The objects are recognized based on the silhouette of the image and hand-drawn shapes, upon which the 3D model is placed. These methods are also mentioned in the patent description.

Another YouTube video from the inventors demonstrates another method for generating AR content on sketches. As described in the patent, the system is used to take a picture of a sailboat on the open seas and animate the waves depicted in the picture with the boat riding them. In the video demonstration, a whiteboard sketch inserts water features, plants, buildings, and other animated content onto a racetrack. Cars navigate the landscape, and topple over when they encounter obstacles they can't physically traverse.

Considering FlyBy Media's role in ARKit and the software-based nature of the patent, there's a chance that this functionality could make its way into a future edition of the toolkit. Much like ARKit in its current incarnation, these image recognition-based features could be very helpful in customer service-focused apps, as well as games and entertainment.

Moreover, this patent could place a chilling effect on other companies that build AR tools. For example, Vuforia is introducing a similar feature called Model Targets into Vuforia 7. We've reached out to Vuforia for comment.