Over the past decade, Marvel Studios has been a dominant force at the box office, raking in more than $21 billion dollars. Averaged out over that span of time, the yearly earnings of those movies outweigh the gross domestic product of some countries.
And while those films have earned a reputation for massive, effects-driven epics laced with humor — a seemingly unbeatable box office formula — there's a supporting player that has helped drive the narrative for most of those movies: augmented reality.
From the advanced technology of Iron Man to the cosmic computers of the Guardians of the Galaxy, augmented reality has contributed to the sense of wonder and awe that the movies inject into their audiences.
Now, as Spider-Man: Far from Home officially brings the Infinity Saga of the Marvel Cinematic Universe to a close, we take a look back at the 23 films from Marvel Studios and how augmented reality has served as the sidekick to its superheroes. So, suffice to say, spoilers abound.
Iron Man
The movie that started the entire saga, Iron Man, also set the stage for augmented reality's role throughout the series. Considering that the man inside the Iron Man suit, genius billionaire playboy philanthropist Tony Stark (played by Robert Downey, Jr.), is a technological wizard capable of building an advanced suit of armor in a cave with a box of scraps, it makes sense that Stark has the best tech.
The most pervasive example of augmented reality comes courtesy of Iron Man's heads up display (HUD), which is first introduced with Tony Stark's first flight in the Mark II suit. From a storytelling standpoint, the HUD serves as a unique point-of-view for the audience, but it also gave us a tantalizing look at how augmented reality headsets and smartglasses will eventually assist users in the real world.
Stark's fictional HUD not only provided him with critical flight data but it also leveraged computer vision to identify objects and individuals within his field of view. During his first flight, he identified landmarks like a Ferris wheel at the Santa Monica Pier. In the heat of his first battle in the suit, the HUD assisted Stark in targeting enemy combatants instead of civilians.
In addition, Stark employed a kind of virtual interface that enabled him to manipulate engineering designs for the Iron Man suit. While designing one of the suit's gauntlets, Stark was able to interact with the design with a light pen controller, as well as directly with his hands. In one scene, for example, we see him putting his arm into a wireframe gauntlet.
But is this realistic? Even with today's best AR projection systems, a headset or smartglasses of some kind are still required to deliver the imagery. Maybe Marvel is taking a few too many liberties here in order to avoid having their star's face obscured. In any case, the Marvel effects team repeatedly blurs the lines between AR and standalone holograms, but it all adds up to a visually stunning world of virtual interfaces.
With Stark reappearing as Iron Man in seven more films in the Marvel Cinematic Universe (plus one post-credits cameo in the next MCU film), these examples of augmented reality or holograms would continue to reappear, with advancements in the (generally fictional) technology added along the way.
The Incredible Hulk
To my best recollection, The Incredible Hulk did not include a single sample of augmented reality. But, upon further review, like Tony Stark, the virtual interface did make a cameo.
Midway through the film, General Thaddeus "Thunderbolt" Ross (William Hurt) and his teams converged on Dr. Bruce Banner (Edward Norton), who morphed into the Hulk during the encounter. The skirmish concluded with a helicopter firing on the Hulk with the assistance of a HUD in its targeting system.
This wasn't one of the more advanced examples of augmented reality, but it kept the technology from striking out completely in the film.
Iron Man 2
Augmented reality made a triumphant return in the sequel to Iron Man. The HUD conceit continued to appear throughout the film, which introduced the War Machine suit, piloted by Colonel James Rhodes (portrayed by Don Cheadle, who infamously replaced Terrance Howard from the origin story).
Stark also expanded his use of his holographic computer interface in his personal laboratory. In this installment, he had multiple designs for his suits anchored throughout his space. At one point, he even grabbed and crumples one idea and tossed it into a virtual trash can, which scored his made shot with a celebratory flourish.
This type of interaction already exists in the real world, to an extent. For instance, the Magic Leap One and the HoloLens both allow users to anchor content throughout their space. More recently, the HoloLens 2 enables users to "grab" 3D content and manipulate it naturally, without any specific gestures, much like we see in the Iron Many films.
The virtual computer interface plays a big role during a pivotal scene in the movie. Stark uses the display to scan a Stark Expo diagram from his father Howard Stark's archives and decode it to discover a new element. Stark is then able to pick up and replace the diagram, expand and contract it, walk within it, and pick up components of the design with his hands.
One new example of augmented reality comes earlier in the film via Stark's smartphone, which has a see-through display. Stark is able to point his smartphone at computer displays and hack into them to gain access. Transparent displays might be a bit of a pipedream right now, but Apple does have a patent for such a display, so this may eventually come to fruition.
The Avengers
After all the virtual interface magic of the last film, augmented reality takes a break in the next two installments of the MCU, by way of the origin stories for Thor (Chris Hemsworth) and Captain America (Chris Evans), but it returns when those characters team up with Iron Man, Hulk (with Mark Ruffalo replacing Norton), Black Widow (introduced in Iron Man 2 and played by Scarlett Johannson), and Hawkeye (introduced in Thor and portrayed by Jeremy Renner) to form the Avengers.
Once again, Stark's boundary-defying holographic computing interface returned, along with a laptop with a see-through tablet interface and an expandable AR workspace. To review a dossier of files on the Avengers, Stark used hand gestures to expand the display into his real space (which would make a lot more sense in reality if he was, again, wearing a headset/glasses/or future AR contact lens), filling his workspace with files, videos, and interactive 3D models.
A new and somewhat gruesome application of augmented reality popped up in a scene featuring Loki (Tom Hiddleston), who assaulted an individual and used a contraption to scan his eyeball and stream a real-time 3D model of it to a mind-controlled Hawkeye in order to gain access to a secure facility.
Iron Man's reprise means his HUD returned as well. In one scene, the HUD warns Stark about an impending surge of power from Thor's hammer, which enables him to deploy more powerful blast from his gauntlets. In a world where interconnected devices deliver data to be read via an AR interface, there's some plausibility here. In another scene, the dynamic enables him to target alien forces and intercept a nuclear warhead. Even though US Army personnel will eventually wear customized HoloLens 2 devices, I'm not sure that it will have the capability displayed by Iron Man here.
Iron Man 3
Phase Two of the MCU kicked off with the third installment of the Iron Man trilogy. And, once again, Stark leveraged his familiar AR tools, but with some unique twists.
Despite playing fast and loose with what's feasible in the real world of tech, at this point in the series, the AR projection interface settled in as a problem solving tool for Stark. In Iron Man, it helped him iterate his original Iron Man suit, built MacGuyver-style with spare parts, into the sleek, form-fitting familiar suit we see today. In Iron Man 2, it assisted him in finding an alternative fuel source for the arc reactor that was poisoning him.
In Iron Man 3, he used the computing device to walk through a virtual reconstruction of a crime scene (something actually happening on a rudimentary level today via the HoloLens), the latest in a series of bombings. He is then able to cross-reference his findings from the crime scene with data sourced from national databases in order to discover a pattern matching a previously unidentified case that helps him to eventually solve the case.
In a somewhat new twist on the holographic projection interface, the film's villain, Aldrich Killian (Guy Pearce) uses an AR projection device that consists of three marble-like projectors that display a rendering of the universe and then the neuron activity in his own brain.
The familiar HUD trope takes on a new role as well. In this film, Stark has developed an AR headset that enables him to control a prototype suit remotely. The remote HUD ends up being the punchline to two somewhat humorous moments, where Stark needs to be in two places at once.
At least in this instance, Marvel felt that it was necessary to give Stark a wearable to achieve this feat of engineering.
Thor: The Dark World
The second installment of Thor's trilogy has the ignominy of being the lowest-rated of all of the MCU films, inching out Hulk's solo outing for the bottom of the barrel.
But the version of augmented reality on Asgard stands out from the pack. In the film, when the Asgardian medical staff examines Jane Foster (Natalie Portman), they use what they call a "soul forge," which is essentially a quantum field generator, according to Foster. Whatever it is called, the device is able to scan Foster's body and project the resulting data into the space above her.
Already, we're seeing applications in the real world where MRIs can be viewed via the Magic Leap One and the HoloLens, but, here on Earth, we use AR headsets, not witchcraft.
Captain America: Winter Soldier
In Marvel's take on the spy thriller, the HUD typically reserved for Stark's armored suits finds a home in the spy car of Nick Fury (Samuel L. Jackson). Paired with an AI assistant, the HUD conducts phone calls, navigates traffic, and displays critical information for the driver.
The film fits in another tantalizing example of augmented reality when Fury's boss/nemesis, Alexander Pierce (Robert Redford) conducts a holographic video call with the World Security Council. Pierce is able to see each member of the council in real time via body-sized glass panels onto which each person's likeness is projected.
The best part of this film's depiction of AR is that it's not totally out of bounds. While armored suits might be a bit out of reach, HUDs in automobiles are within the realm of current real-world possibility. Magic Leap subsidiary Mimesys has already achieved video calling with volumetric video. Although a headset is required in real life, Winter Soldier compensates with glass booths that act as displays.
So, kudos to Winter Soldier for being grounded in its approach to AR!
Guardians of the Galaxy
Marvel's maiden voyage into outer space presented several cosmic imaginings of augmented reality technology. During the introduction of adult Peter "Starlord" Quill (Chris Pratt), we find him employing a hand-held projector to scan the environment of Morag and cast a virtual overlay of the planet's past civilization. The projection also served as AR navigation to the location of the Orb. And, because Starlord wears a helmet here, we have practical excuse for this to work.
Later, Rocket (voiced by Bradley Cooper) took a page out of Stark's tech playbook with a transparent tablet that used facial recognition to learn more about passersby, through which he discovers a bounty on Quill. The tablet is nearly all screen; alas, even in space, the notch is ever-present.
Elsewhere in the film, The Collector (Benicio Del Toro) delivered exposition regarding the Infinity Stones via a holographic presentation. As we progress through the rest of the MCU, we'll see that immersive technology increasingly becomes a tool for delivering exposition and, since proponents of the technology often tout it as a new storytelling medium, this makes total sense.
Much of the rest of the examples of augmented reality come courtesy of the Nova Corps, the police/military force of Zandar. In one scene a smart window displays biographical and biological information of the arrested Guardians as they are booked, doubling as our introduction to the characters. During the movie's climactic battle, the Nova Corps also monitor the action via a holographic display.
We've taken Stark to task for holographic displays that defy convention, but, since we're talking about advanced alien civilizations here, who's to say they haven't figured it out already?
Avengers: Age of Ultron
For a film named after its AI supervillain, there's not a lot of augmented reality to speak of. However, it is deployed to a great extent to depict elements that would not otherwise translate well on film.
As Stark and Banner are attempting to build Ultron (voiced by James Spader) as a peacekeeping system, Stark uses a holographic projection to visualize the AI of Jarvis (Paul Bettany) and the neural activity of what is later revealed to be the Mind Stone. It's a similar concept to the trick we saw in Iron Man 3 in that it exists purely to help set a foundation for the plot, but strains credulity in its practical feasibility.
Captain America: Civil War
Despite several heaping spoonfuls of science stuff, Antman, which was released after Ultron, was devoid of augmented reality, but the technology returned in the next edition of Captain America's trilogy, which is essentially another Avengers movie.
Aside from the usual appearance of the HUD via Iron Man and War Machine, Stark manages to introduce a radical new version of Marvel's brand of holographics/AR called binarily augmented retro framing (aka B.A.R.F.)
As he explains to the MIT audience, the fictional technology reads the subject's memories and, with a combination of light projectors and a pair of glasses, renders a more pleasant version of events into the physical space as a means of psychotherapy.
That's not all. Stark's smartphone and smartwatch are outfitted with projection displays that enable Stark to conveniently show images to others. In both instances, the displays expand via hand-gestures. Can you image the battery power this kind of display would require?
Augmented reality took a break during Doctor Strange and Guardians of the Galaxy Vol. 2 (unless you count magic and celestial conjuring).
Spider-Man: Homecoming
The third film depiction of a live-action Peter Parker (Tom Holland) is essentially a surrogate son of Stark's, so he also gets the benefit of a more technologically advanced suit. (Ironically, his own homemade suit pales in comparison to his predecessors, but, then again, he's just in high school.)
Along with the Stark-powered suit, Spider-Man gets an Iron Man-style HUD, complete with computer vision and his own AI. While we haven't seen this before in a big-screen Spider-Man, it's become old hat as far as AR in the MCU is concerned, though it does supply a fair amount of humor and narrative tension.
So, we've picked at a couple of notions of AR in the MCU that are outlandish, but, on this one, I'll focus on one particular foible: how does Spider-Man's suit power its HUD? I don't see any solar panels, much less where batteries would fit in his form-fitting suit.
Thor: Ragnarok
It's a bit ironic that Thor and the Hulk, the owners of the two lowest-rated solo films in the MCU, were able to combine forces in Thor Ragnorok (which could easily be titled Thor & Hulk's Excellent Adventure) for one of the more critically-acclaimed entries.
Alongside its cosmic hijinks, the film manages to pack in two examples of augmented reality and holographics, and both come within the scene that finds the dynamic duo, along with new ally Valkyrie (Tessa Thompson), making their escape from Sakaar.
First, the planet's ruler, The Grandmaster (Jeff Goldblum) implores his followers to chase down the escapees. He does this via a larger-than-life holographic projection.
Second, in a twist on the HUD displays featured throughout the MCU, the spaceships piloted by the protagonists and antagonists in the chase scene include virtual displays that deliver information to the passengers.
Black Panther
Taking place mostly in the highly advanced kingdom of Wakanda, home of the miracle metal vibranium, Black Panther introduces some fairly fantastic takes on augmented reality.
Basically, instead of projected light displaying 3D images or video, what appear to be vibranium particles (or maybe nanobots) form 3D representations of the subject matter. In one instance, T'Challa (Chadwick Boseman) uses the technology to view footage of a convoy. He is able to interact with the display, picking up one of the trucks to view the passengers inside.
- Don't Miss: Walmart Shoppers Can Now Use Their Smartphones to Become Marvel's Black Panther via Augmented Reality
Elsewhere, portable versions of this technology, embedded in a vibranium-powered bracelet, are used to conduct video calls and deliver 3D presentations for the purpose of exposition.
Of all of the AR examples throughout the MCU, the vibranium displays seem the most far-fetched, until you consider that the microdrones developed by Intel, which can be programmed to fly in patterns, could eventually take this form with enough cycles of Moore's Law.
Avengers: Infinity War
A movie as big as Infinity War does not have room for a lot of fluff, but the film manages to pack several AR instances into its storyline.
Augmented reality teleconferencing at Avengers HQ has taken a big leap forward since Winter Soldier. (I presume we have Stark to thank for that.) In this film, an entire remote room is projected into the Avengers HQ, and callers on the remote end are able to walk around the space. And, with a simple hand gesture, the call ends. There's also a brief view of a holographic display at Avengers HQ.
In the cosmic realm, we find out that abused cyborg Nebula (Karen Gillan) is able to project holographic video playbacks of her memories from her eye. I don't see that one hitting the stores on Earth anytime soon.
When the story moves to Wakanda, we get to see Shuri (Letitia Wright) conduct a full-body scan of Vision (Paul Bettany) using the vibranium-based bracelet computer and then display the results via a holographic projection.
Ironically, Iron Man's HUD doesn't get much screentime, but Banner gets an opportunity to utilize it as he mans the Hulkbuster armor.
I'd be remiss if I didn't at least address the Reality Stone, which Thanos (Josh Brolin) deploys to literally remake reality into his vision. He uses the gem's ability for subterfuge, counterattacks, and exposition. Is this the ultimate in augmented reality? Reality is whatever I want it to be!
There's isn't any augmented reality in Antman and the Wasp, the film released just a few months after Infinity War, as the film is mostly preoccupied with such heady concepts as quantum mechanics and truth serums.
Captain Marvel
Since Captain Marvel takes place in the 1990s, there isn't much AR on display on Earth.
However, the advanced Kree have already mastered immersive computing because they're from outer space.
Rather than just retread previous examples of holographic displays, the movie does take the opportunity to introduce a more novel use of the technology via the wrist-mounted touchscreen computers worn by each Starforce member.
The smart gauntlets, as I'll call them, are capable of making and receiving 3D calls, with a light engine mounted on each finger to project a 3D video of the caller. They can also project a larger display for group battle plan presentations that also serve as exposition. The visibility of the light source lends some credibility to the concept, but lacking a display or glasses of any kind, this is an alien concept (pun intended).
The projection also reads hand gestures and touch input. Finally, the smart gauntlets can scan and identify species and rate their threat levels.
- Don't Miss: Marvel Expands Roster of Avengers Playmoji for Pixel Playground Ahead of 'Avengers: Endgame'
In addition, the Kree spaceships use the 3D projection displays throughout the film, particularly for receiving volumetric video calls. One ship even has a HUD for displaying a video feed. As for the lack of an AR headset? I guess we'll give alien technology a pass here.
Even the mid-credit scene, which serves as a bridge between Infinity War, Captain Marvel, and Avengers: Endgame, includes augmented reality displays at Avengers HQ, which will figure prominently in the next film.
Avengers: Endgame
Taking place (mostly) five years in the future, the climax to the Infinity Saga packs more augmented reality examples into its three-hour runtime than any other MCU installment.
Of course, where there's Iron Man, there's a HUD. And for a short period of time at the beginning of the movie, Banner, wearing the Hulkbuster armor, has a HUD as well. But that's not all, as each Avenger gets a HUD when they don their time travel suits.
Tony Stark's glasses appear ordinary in 'Infinity War,' but their smart properties are revealed in 'Endgame.'
But, not to be outdone, Stark also has an everyday HUD in a pair of smartglasses. While there are hints that the glasses are of the smart variety in Infinity War, that status is confirmed in Endgame. In addition to x-ray vision, the smartglasses have robust computer vision capabilities, identifying objects in a lab in the year 1970.
I won't even touch x-ray vision, but, since smartglasses computer vision, particularly by means of AR cloud with crowdsourced environmental mapping, is on the verge of being a reality, I'll note that such a system would probably not have the machine learning capacity to have mapped what existed in the 1970s, much less within a secure military complex.
Holographic tables and displays have also become commonplace throughout the Avengers compound. In this film, the remaining heroes use tabletop holographic displays, projected from an Amazon Echo-sized cylinder, to review data on missing people (similar to the mid-credit scene in Captain Marvel) and visualize planetary scans.
They also use 3D interface to assemble a delicate piece of bejeweled armor, and use a holographic display in place of a security camera monitor, complete with a swipe-to-answer mechanism to answer the intercom.
Stark also has a holographic table in his home, where he uses it to design yet another piece of advanced tech.
Also, in this film the holographic video conferencing technology received an upgrade, as it now allows five simultaneous callers, including two from deep space, to converse with each other.
It's all an impressive leap forward in fictional technology, particularly considering that half of the world's engineering talent has been reduced to dust. And they have time travel, which is an absolute win.
In the cosmic realm, Thanos's ship is outfitted with augmented reality as well, with his own brand of holographic displays on board. Nebula's implanted projector also makes a return appearance to replay more memories.
Later in the film, Stark manages to sneak in a new example of AR in (remember, this is a spoiler-packed feature, so you've been warned, again) his swan song. At the beginning of the film, he uses the remains of his helmet to record what he believes to be his final farewell (as shown in the official trailer). The helmet scans him with light before he records his message. At the end of the film, we find out that scan was used to record a volumetric video, as his truly final farewell is projected into the physical space for his friends and family to view.
Spider-Man: Far from Home
If Avengers: Endgame is the climax of the MCU's Infinity Saga, then Spider-Man: Far from Home serves as the denouement. And while Stark does not physically appear in this edition, his technological legacy looms large over the film, as evident from its trailers and TV spots.
In a vignette released by Sony Pictures, we learn that Stark has left behind a 3D printer with a little more of that magical holographic interface that enables Parker to create customized suits.
Parker also inherits Stark's smartglasses, which get a chance to show off some more of its AR superpowers. Clearly, Stark wasn't very forthcoming about what the smartglasses could actually do, as Parker is surprised to hear an AI voice when he puts them on. He then proceeds to accidentally eavesdrop on his fellow travelers as they message and browse on their smartphones. While facial recognition via smartglassesis really a thing, the level of power these smartglasses exhibit, such as gaining instant access to smartphones, probably requires lofty NSA security clearance.
The holographic table makes its return as well, again as an expositional tool as Mysterio (Jake Gyllenhall) explains the genesis of the current threat against Earth.
I've already had the opportunity to see the film, but to explore AR and holograms any further would venture into spoiler territory, so I won't address them yet. But, I can say that the film actually makes the firm distinction between augmented reality and holograms. Stark's smartglasses are classified as augmented reality. Elsewhere, projected imagery are defined as holograms.
Real World Examples
While much of what the MCU depicts as augmented reality is either too outlandish to consider or just too far down the road to fathom, some of the recurring concepts are almost within arm's length.
The spatial computing capabilities of the HoloLens and the Magic Leap One give users the ability to accomplish some of the same feats depicted throughout the MCU. And while the fictional depictions of Marvel films conveniently omit the use of any headsets, with the right real world equipment, practical examples of some of these dynamics can allow users to manipulate engineering designs, conduct scientific research, and even examine crime scene data.
Meanwhile, smartglasses equipped with cameras and computer vision software can already accomplish navigation, object and facial recognition, voice input, and even remote control of drones. It might be a few more years before they are available in the form factor Stark's smartglasses present, but, in the real world, this kind of device is within sight. Also, AR clouds in which the world is mapped out by multiple crowdsourced cameras will be a big step on the journey toward arriving at the capabilities that Stark-engineered HUDs envision. The AR cloud dynamic will be particularly potent in the realm of computer vision, which will be used to identify people and landmarks in real time.
As famed science fiction author Issac Asimov once said, "Today's science fiction is tomorrow's science fact." Thanks to Marvel, we have some great examples of augmented reality to look forward to. And, since the movie juggernaut is sure to continue well into the future, now that the rights to Fox's film properties have returned to the mothership, get ready to expand your AR imaginations even further.
Cover image via Marvel Studios
Comments
No Comments Exist
Be the first, drop a comment!