Apple AR: ARKit & Machine Learning Come Together Making Smarter Augmented Reality

The world is a massive place, especially when you consider the field of view of your smartglasses or mobile device. To fulfill the potential promise of augmented reality, we must find a way to fill that view with useful and contextual information. Of course, the job of creating contextual, valuable information, to fill the massive space that is the planet earth, is a daunting task to take on. Machine learning seems to be one solution many are moving toward.

Tokyo, Japan based web developer, Frances Ng released a video on Twitter showing off her first experiments with Apple's ARKit and CoreML, Apple's machine learning system. As you can see in the gifs below, her mobile device is being used to recognize a few objects around her room, and then display the name of the identified objects.

Apple AR: ARKit & Machine Learning Come Together Making Smarter Augmented Reality — *Images via France Ng, Frances Ng*

In case you are unclear on the term "machine learning" — as it is thrown around everywhere in the tech world these days — let's define it. On the base level, machine learning a computer model created using pattern recognition and some search criteria. The model is then given data sets, at which point it will respond when its recognition is triggered. It can then build a better model as it recognizes more patterns, or learns.

In the examples above, the machine learning model could have been fed pictures of space heaters, fans, keyboards, and jeans and the words associated with those pictures. Then using the ARKit, it is told to draw the associated word with the picture. Notice how it corrects itself from fan to space heater. Of course, this sample is very basic and will not likely become Skynet, so don't worry... too much.

This object recognition solution and its odd cousin, computer vision, are being leveraged a bit more every day for augmented reality. More accurate alternatives to GPS are being explored, and when one is ready for prime time, the blending of technologies will allow not only useful contextual information but — depending on who creates the system — it will be crowd sourced information as well. Which of course, will fill that massive space up quickly.