Why VR still isn’t as immersive as it should be

One of my most immersive experiences was set in a forest. Daytime, the sky a clear blue with large white clouds floating over, pine trees stretching up, orange needles covering the ground with a smell of sap. On the ground a man is dying, but preparing for his last action, an ambush to cover the escape of his dear friends, fighters for the Republican side in the Spanish Civil War. The man thinks long about death and eternity and everything he could not do, the horror of finitude. I finish the book and throw it across the room. The experience was too real for me.

Reading a novel, such as Ernest Hemingway’s For Whom the Bell Tolls, is an immersive experience. Never did I think that I was actually Robert Jordan, that I was actually speaking Spanish among Republican guerrillas, nor did I feel exactly what he felt blowing up a bridge, or having a tank shell explode below my horse. But I still know this world, know what it looks like, smells like, feels like. I know many of the characters better than people I’ve met in person, I know the geography of the world, I know what matters and doesn’t matter to each of them. So even if I don’t inhabit the body of Robert Jordan, I do know what it feels like to be him. I know his knowledge, and I think his thoughts.

Different media excel at different experiences, and VR and books should never try to replace each other. But there is much that this new medium can learn from older ones, and hopefully guide it away from some key errors. Much of the excitement for VR comes and came from the promise for immersion, feeling like you are actually there in an experience. VR has the highest visual fidelity of an actual physical environment ever experienced and allows for realistic head and body movements. But even with directly photographed content and big budgets, VR content has not grabbed people like the great works of other media. VR experiences are often neat — short and compelling but not particularly memorable. Those initial few minutes are incredible, immersing you quickly, but the effect and its novelty fade before long. In contrast, a book can be extremely hard to begin and get into, but after 400 pages from a skilled author, the immersion can be just as deep if not deeper. VR should be capable of doing much more than this, but so far it has failed to.

Books, movies, games, and all other media usually become more immersive the more time you spend with them. As you get to know the world better and accept it, the thoughts of the characters become your own. Usually such immersions break apart only when something goes wrong, where the creator makes some key mistake that ruins the suspension of disbelief and forces the audience to think about the media rather than within it. A huge plot hole, a broken boss fight, or poorly written dialogue all pull a user out of immersion. But with VR, people aren’t yet going deep enough for these to be a real problem.

What is missing from VR? It has incredible visuals, issues with motion comfort are improving, sound is improving, there is experimentation with haptics, but none of these improvements seem to be getting VR closer to enabling true immersion. The chains of persuasion that help us feel a place is real still seem to be missing some crucial link. This is because the key ingredient that makes both physical and virtual experiences real is meaning.

If I am in a familiar room, it is not the visuals or sounds that make it feel real, but what I can do in it. I know where my chair is; I reach for my glass, knowing its exact weight and shape as I lift it, knowing it contains the water I got from the sink behind me. This background knowledge is what immerses me, not the direct perception of these objects. Meaning is how other media work with immersion. Books are thick with meaning, and the more time you spend immersed, the more real the characters, locations, and logic of the world all become in a way that is deeper than the senses.

Video games have deeply internalized this lesson. When computers were still simple and slow, games could simulate action far better than they could simulate visuals, and so game designers focused on making worlds full of meaningful action. Text adventures drew from the descriptive power of books but allowed for interaction and exploration. Because they require reading, it takes time and dedication to become immersed in such a low-resolution experience, but it can be highly rewarding. Once games were able to simulate 3D graphics and movement, developers pioneered kinaesethetic experience design, allowing for complex motions and environments. Many of the best VR experiences to date are works like Half-Life: Alyx and Resident Evil 4, which fully borrow the meaning structures of video games but deepen the experience with VR features.

When I control Mario, I am immersed. It doesn’t matter that it is on a small screen separate from my face, the graphics are blocky, the sound non-spatial, or that I control him through button presses, because I have learned this world and what I can do in it. Just as I do not think to myself “extend arm” when reaching for a glass, I never think “press A” but only “jump” “jump” “triple jump,” moving nimbly through this world. Kinaesthetic experiences are highly absorbing, requiring great concentration, which in turn facilitates greater immersion in the world.

Simulating actions simply is also more realistic to the experience of meaning than simulating them in detail. If you asked me to punch rather than pressing a button for a game of Street Fighter, I would punch poorly. My form would be completely wrong, I would fail to extend correctly, my muscles not used to the movement. I would have to be thinking every moment about the unfamiliar act of punching and what exactly I’m doing right and wrong. But for a martial arts master such as Ryu, punching is second nature, his body and mind trained to make it subconscious. He would not think “extend arm” but would think “punch,” which is also what I think when pressing the button on my controller. Simulating an action exactly can be a highly valuable design tool, but an action that fails even one tenth of the time destroys immersion in a way that an abstracted button press would not. Immersion should give you the feeling of being in a world, and no world will feel right without consistent control of your actions.

This careful attention to the meaning of actions should also extend to the meaning of the environment. If there is nothing to do in a world, then it does not feel real, just a series of unrelated images. Good experience design teaches a user how to navigate a space based on goals and actions. Playing an RPG like Dark Souls, I know there is a shortcut below a shrine unlocked by a key I’ve gotten from an enemy, but getting there means I’ll have to face several other enemies. Every decision requires strategy and thought, but I am fully thinking within the world itself — my thoughts are the same as the character’s. I know where I am and what I am doing. Even if it looks to an outsider like I am just pressing buttons, because I am immersed my sword strikes carry meaning, moving towards the goals of this fictional world I inhabit.

VR represents an enormous leap forward in media technology, which will undoubtably deliver the best visual, aural, and haptic experiences. But more fundamental to immersion is the meaning within a world, the meaning of movement and navigation and objective. To create the best immersion, VR needs to build on a foundation of carefully designed meaningful interaction that guides the player into making the virtual world their own.

Ethan Edwards is a Creative Technologist at EY.