XR Expectations Part 3
In Part 2 of our series introducing Immersive Design principles, we introduced the idea that Expectation Models are built and developed as a result of any and all exposure to a real-life object, interaction, or event. The more familiar we are with a real-life experience, the stronger and more robust our expectation model becomes until we get to a state where we become hyper-sensitive to any little thing being amiss with that experience.
Before founding Realised Realities, I worked at Sony Interactive Entertainment as their very first Immersive Experience Specialist. I was brought onto the project that eventually became PlayStationVR in 2010, way back when the hardware was cobbled together from a head massager, iPhone screens, circuit boards and lots and lots of cables, with a single PSMove mounted on top for tracking. But from humble beginnings... this prototype evolved into the 'Project Morpheus' R&D effort, which blossomed and debuted as PlayStation VR at the 2014 Game Developers Conference. Almost everyone else who joined the project in the first few years was an engineer first and foremost. They'd got the tech working already, but were immediately running into all the problems we're familiar with now as VR designers. My role was figuring out the gameplay and design implications of VR to see if it was worthwhile to even pursue the tech. Was making VR games for a consumer audience something that was even viable?
All the team saw was one challenge after another. The tech was hugely promising but as a user experience it was a long way from being ready for PlayStation players. So for several years I explored, researched, hypothesized, and figured out what we needed to understand before we could teach developers how to make actual commercial games for this new device. To understand the time-frame; this started a couple of years before Palmer Luckey launched the Oculus DK1 in 2013, and so there was practically zero VR content out there initially. If we wanted to try something, we pretty much had to build it in a prototype, or try to convert an existing game's code to work in VR and on our headset. So we built and ran lots of small experiments to get an experiential reference point of what different real-life experiences might feel like in VR, and thus develop our understanding of what was possible and viable for developers to explore.
In one such protoype at Sony PlayStation, we tested two groups of users on a virtual driving game, a very very early conversion of PlayStation 4's DriveClub game (which I'd been in charge of once-upon-an entirely different incarnation), into Virtual Reality. We knew driving games would be tough in VR for various reasons, but we knew it needed solving, just like every other game type that developers would want to build for PSVR. Indeed, years later, DriveClubVR ended up being a real game and one of the top-selling launch titles for PlayStationVR, so the knowledge we learned in these experiments turned out to be valuable for more than one reason!
We were testing how much more immersive the driving experience was in VR using different interfaces. The first was playing the game with a high-end force-feedback steering wheel controller with foot pedals. The second approach was playing the game the way most players would encounter it; using the PS4’s standard out-of-the-box DualShock4 controller. What we found went squarely against our expectations. But it gave us a fascinating insight into the way expectations work in VR.
Users found it more immersive to drive with a wheel, just as we'd expected. However, this interface also caused a notable increase in the levels of discomfort, something that we didn't expect. Some of our users couldn’t make it round a full lap using the wheel in VR before they were suffering from the onset of what we then called motion sickness, but that we now know more specifically as VR sickness.
But confusingly, in a separate test the next day, all of our test users could manage 3 or more laps using the DualShock controller much more easily. Given what we understood at the time we would have expected the wheel to feel more natural, intuitive, and thus less likely to cause discomfort for the users. Something didn’t add up, and our assumptions had been proven very wrong. This meant it was time to science! This finding sparked several rounds of investigation and discovery that proved hugely useful in our understanding of user expectations and the effects they can have on the plausibility of a virtual experience.
Driving is a very familiar activity for many of us in real life. Because we know that the more familiar we are with a real-life experience, the stronger and more robust our expectation model becomes, we can understand why driving a car in VR might be very hard to pull off successfully. Experienced drivers will easily spot all the mismatches between their virtual experience what they’re expecting from real life experience. If we drive often, then our expectation models for driving a car in real life are very rich and very detailed. As well as seeing the turn in the road, we expect to feel the lateral g-forces pushing us over in our seat. As well as hearing the engine respond to the accelerator, we expect to feel the acceleration forces push us back. We're dependent on different types of feedback through the pedals, wheel and the body of the car. There's a whole symphony of sights, sounds and feelings that all come together as a package and collectively confirm that we're driving a car.
We're so familiar with this that we can quickly spot when something's sketchy. It’s easy to understand why things can feel 'not right' in a driving experience that looks and sounds just about realistic enough to fool our eyes and ears, but where almost all of the other sensory inputs we're used to are missing. And yet there are lots of VR driving games out there that players enjoy, and that don’t cause those users untenable amounts of discomfort – so how can that be?
Not everyone who is reading this might have driven a car in VR, so let's use a different, more common immersive experience to explain how this effect is caused because familiarity changes our expectations.
Loads of people loved Wii Sports Tennis. It was an unimaginable success for a video game - lifetime sales of 82.9million, still the highest ever sales for a single platform game, and to this day the 4th best selling game of all time. And of the 5 sports it offered, Tennis was amongst the best loved. It was lauded for its immersive qualities, of how it successfully emulated the 'feel' of real tennis for many people. But not for everyone.
If you were an experienced tennis player in real life, then it was probably much harder for you to become immersed in Wii Sports Tennis, even if the next guy was finding it super-immersive and believable. You see, that next guy probably had a much looser and more vague set of expectations going in. Maybe they played some tennis as a kid. Maybe they've watched a few Wimbledon finals on TV, and played a tennis videogame or two over the years. As far as they're concerned, it it walks like a duck and quacks like a duck then it's a pretty close emulation of Tennis - and many found it to be surprisingly immersive and believable. But as a more experienced Tennis player, your greater experience with the real life sport is going to carry a more detailed set of expectations for what tennis looks and feels like. And Wii Sports Tennis never tried to replicate all of the real sport. You play standing still, not running around the court. You play looking at the court from an unnatural 3rd Person perspective with a high camera more like a TV broadcast. You're not feeling the impact and feedback as you thwack the wall. The controller doesn't feel anything like a tennis racquet. And so on and so forth.
That other guy has a much looser set of expectations that need to be satisfied for his brain to buy into the illusion - as long as the racket feels and behaves enough like the racket he remembers holding as a kid, and sounds like it does on TV when he swings it and thwacks the ball back to his opponent, then that may be enough to satisfy his expectations, and buy into the illusion. This is the magic that Wii Sports managed to conjure for so many players back in 2006. Real tennis players thought little of it, but for millions of other players with looser expectations, it ticked all the (limited) boxes and felt believable and immersive.
And this is a key consideration for anyone working in VR. The most immersive moments in virtual reality are those where you’re not stopping to question the validity and realism of what you’re doing.
In VR science this is called the Plausibility Illusion, and it's one of the components we need in order to maintain a 'Presence' state where the user forgets they're in an artificial reality. It’s an illusion that’s easier to achieve when the activity is less familiar to the user.
So when you’re driving a virtual car or playing virtual tennis or doing just about anything else in VR, remember that your subconscious mind is evaluating the inputs against that pre-formed expectation model. It needs to test its plausibility and validity to make sure it is what it thinks it is. That expectation model is built from a whole bunch of influences, not just first-hand personal experience.
The less familiar an activity is for a user, the easier it is to plausibly reproduce in VR for that user.
But every user is going to be different, with different expectations.
That’s one reason why some racing game enthusiasts might find it easier to accept the sensory mismatch of driving in VR, while those primarily familiar with driving in real life, but not so much in video game, might find it causes them discomfort. If you play a lot of flat-screen driving games, you already have a rich expectation model not only about driving a real car, but also about driving a videogame car. You already have two different sets of expectations that are quite distinct. When your cerebellum is trying to figure out what on earth is happening (“it looks like we’re driving a car but we’re not getting the sensory inputs that normally go alongside that?!”), it already has a more appropriate context to consider. You have a rich set of expectations about what videogame driving feels like. There are already 2 experiences your brain can match this against, and it's closer to a videogame in many ways. That also means it’s easier for your brain to accept this is something else again, a third type of experience that fits between those two models.
Here's the cool, fascinating insight that we had in our DriveClub experiment, where the majority of users who found playing with the PlayStation controller to be less immersive but more comfortable than the much more true-to-life steering wheel controller. What we came to realise was that the interface you have with the experience is hugely significant when your brain is trying to contextualize the experience and match it against other activities you’ve experienced. The visual and spatial audio inputs from a racing game in VR can be convincing – it sure can look and sound like you’re driving a car, and this will only become more convincing as VR tech continues to evolve. But the abstracted interface of the game controller in your hand, which feels nothing like a real car but just like a videogame, is a strong signal that you’re playing a game, and that what you’re seeing and hearing is not real.
Framing the experience as a videogame invokes a comparison with a different set of expectations. Videogames being as broad and multifarious as they are,
this frame of comparison is far more flexible and accommodating.
This truth was borne out in the responses players gave in the DriveClub experiments. We saw a trend of users describing the controller-based experience as ‘the coolest driving game I’ve ever played’. But the word ‘game’ was conspicuously absent when talking about the exact same race played through the steering wheel, where the responses were more often variations on ‘it feels off somehow ... it's not like driving a real car’. Simply changing the input device re-framed the whole context of the experience for users, evoking a different set of expectations, and that directly affected their enjoyment and comfort levels.
What we went on to find was that there were other ways of abstracting away from realism, that also seemed to help users re-frame the experience to positive effect. There are a slew of techniques and tricks that can be used to do this, more often than not without damaging the plausibility of the illusion. Since the designer’s goal in VR is most often to achieve sustained immersion for the user, finding ways to manage and steer the user's expectations can help sustain immersion and give a smoother, more comfortable ride.
We'll look further into this in a future post.