Robot Swarms Could Be Controlled by Your Smartphone, Thanks to AR

Have you ever wanted to control a swarm of robots? Well, now you can! Robotics researchers at New York University (NYU) have created an app which controls 'bots from your smartphone, using augmented reality. This AR app would certainly come in handy when you want to make a coffee from the comfort of your couch, but let's not get ahead of ourselves!

Jared Alan Frank, a Ph.D. candidate in mechanical engineering at NYU's Tandon School of Engineering, built the app which uses AR to create a robot control interface. This can be run from a smartphone or tablet, which is certainly impressive. Speaking about his app, Frank told Next Reality:

Unlike the methods that are conventionally used to interact with sophisticated teams of robots, our approach does not require purchasing or installing any additional hardware or software and does not require the interaction to be in a traditional laboratory environment. This is because our app relieves the dependence on laboratory-grade and industrial-grade equipment and instead leverages the capabilities of the mobile device in tracking and controlling the robots.

As we know, AR works by layering additional information on top of real surroundings; this system works accordingly. Your smartphone camera is used to capture the scene details and virtual objects are then overlaid. But Frank's app makes the process even easier as users simply tap and swipe the screen to control the robots. We're going a little power-mad just thinking about it ...

The app options (see captions)

In a detailed email, Jared described how the data from your device's cameras and its inertial sensors ("e.g., its accelerometers, gyroscopes, and magnetometers") also render augmented reality graphics on the interface screen, as well as tracking the robots. He said through the manipulation of these augmented reality graphics, users "can intuitively command the robots to perform a task."

Apple's software development platform Xcode 8 was used to design the interface of this app. This creates a virtual grid, in which robots and objects can be easily detected on the screen. Basically, you can sit back, command your 'bots from the app using Wi-Fi, and watch those metal minions do your bidding in the real world. Mwahahaha ... Er, excuse me.

The current version of Frank's app is using Raspberry Pi, a "credit-card-sized computer that plugs into your TV and a keyboard," to carry out these commands.

Visual tags called Fiducial Markers are placed on both the robots and the objects they are asked to move. The device captures the scene using its camera and the app detects these tags.

The concept

Frank, whose dissertation is entitled "Mobile Devices as Platforms that Enhance Measurement, Control, and Interaction with Systems," said that mobility is a distinct advantage here. The easy operating system on a phone means that robotics can soon be liberated from the lab. The doctorate student told Next Reality that his application is rapidly becoming relevant:

Enabling everyday people to control a small swarm of robots is of great practical value, since the list of applications in which people and groups of robots may need to interact is expect to grow steadily (e.g., in education and training, construction and manufacturing, and recreation).

This innovative application may be coming to a construction site or factory near you, to test out its capacities in a real world environment.

Meanwhile, the savvy robotics expert isn't stopping there: Jared confirmed that he has other robot-controlled applications which manipulate objects in the works, as well as "laboratory experiments for learning scientific and engineering concepts."

Let's hope the software and hardware Frank is using can operate and flourish in real world conditions. After all, who wouldn't like to control a herd of robots at work?

• Follow Next Reality on Facebook, Twitter, and YouTube
• Follow WonderHowTo on Facebook, Twitter, Pinterest, and Google+

Cover image via Jared Alan Frank