This could be the way we watch movies in the future

Marvel Studios’ What If…? — An Immersive Story is tricky to describe. Part interactive game, part narrative-driven movie and part 3D comic book, it puts you — the viewer? the player? — at the center of a narrative that reimagines the fates of superheroes and villains from the Marvel Cinematic Universe.

To experience What If...? you must wear an Apple Vision Pro headset, which is considered "mixed reality." That means that it incorporates both virtual reality, or VR — transporting you to a different world — as well as augmented reality, or AR — where other video is layered on top of the actual room that you are in. The story toggles between scenes in AR in which characters appear to materialize in the user's real room, and a series of dizzying VR landscapes in the Multiverse.

Some parts involve passive viewing, like a sequence in which the supervillain Thanos is on trial for theft. But there’s also a lot of interactivity: one character explains how to use hand gestures, like making a fist, to defend yourself against enemies and cast magic spells.

This is a much different experience than traditional TV or movie watching, and industry insiders think it will change the face of entertainment.

"It's kind of creating a new canvas," said ILM Immersive’s Shereif Fattouh, the executive producer of the immersive version of What If...?.

Although interactivity is at the heart of the experience, viewers can opt out.

"There's a lot of audience that are traditional gamers that really want to shoot things," Fattouh said. "And there's folks that don't play games and want to just see a great story."

Catering to a wide variety of tastes

Gamers have been using VR systems for decades. But in the last 10 years or so, new headsets — with more powerful graphics and motion tracking technologies — have started to broaden audiences.

The current entertainment offerings cater to a wide variety of tastes. For example, Meta headset users can sit with friends at an NBA basketball game with its Xtadium app, explore a haunted Irish castle in The Faceless Lady, a VR live action horror series, or take in pop star Sabrina Carpenter’s recent immersive VR concert.

"Sitting courtside at your favorite basketball game or seeing your favorite superhero is a completely different experience [in virtual reality]," said Jason Thompson, the creator and host of The Construct, a consumer VR-focused YouTube channel. "It can't really be compared to watching things on a flat screen."

Thompson said he uses apps like Bigscreen to watch traditional TV shows and films in his headset. If he chooses, it can be a social experience; users can chat with others at a watch party or mute them if there's too much conversation. They can also change their surroundings, so a living room couch transforms into what seems like a plush movie theater, complete with virtual popcorn. Thompson said he sometimes watches in Bigscreen's bedroom setting, lying flat on a bed.

"The screen is actually on the ceiling," said Thompson. "And you have to lay back to see the screen."

Lying down to watch content in VR isn’t just whimsy. It serves a practical purpose. Most of today’s headsets are heavy and awkward to wear. Thompson said reclining to watch takes the load off the head and neck.

"In order for VR to excel it has to become comfortable," he said.

An industry finding its feet

Tech players are working on it.

"As every year passes and we improve the technology, make it easier to set up, more seamless to what you do on other devices, we see more and more people actually adopt," said Sarah Malkin, director of immersive entertainment at Meta, the current market leader in consumer VR. "We knew we would be investing in what is essentially the future of computing and that it would take time."

Malkin said Apple's arrival in the market — it launched the Apple Vision Pro in February — is a good sign that headsets are becoming more mainstream, even as they cost anywhere from $300 to more than $3,000. However, Apple and Meta are not disclosing specific sales figures, so it's hard to know for sure how the market is developing.

"The key for this market is consumer adoption," said Ben Arnold, a consumer technology analyst with the market research firm Circana. "Because that is something that makes development of the applications more attractive."

Adoption depends on both tech and content

Filmmakers say that the technology also needs to be friendly to creatives, so they can tell better stories in VR.

"The basic interaction methods have yet to be figured out," said VR film writer and director Eugene Chung of Penrose Studios, the company behind several VR movies, including Arden’s Wake, a post-apocalyptic ocean adventure which won the first Lion Award for Best VR at the Venice Film Festival in 2017. "It should feel as natural as using your iPhone. And we're so far away from that."

Chung said it’s easy for people to get frustrated with many of the current TV and film VR offerings because users don’t know where to direct their attention in a given scene, or they want to fully interact with a character but often can’t.

"You see stuff happening, but things aren't reacting to you the way that you think they should," Chung said. "For example, you can't just go up to a character and talk to them about Shakespeare or ask them about what they ate for lunch." 

Yet he’s excited about continuing to explore the creative potential of this new medium, especially since many young people today are growing up as VR natives.

"I have no doubt that this will be the future of all of entertainment and really all of computing," he said.

Jennifer Vanasco edited the audio and digital versions of this story. Isabella Gomez Sarmientomixed the audio version.

With thanks to Will Mitchell and James Mastromarino