UX for XR — User Experience for

Extended Reality

 There’s no doubt that VR and AR are extremely innovative experiences, allowing us to use our movements and gestures to manipulate and interact with virtual objects. However, most wrongly assume that this immersive tech is ready to use out of the box.  Evidence has been consistently found that proves the user experience of AR and VR remains in its infancy and that there are many opportunities for improvement.

What is XR and Why is it Important?

Extended reality (XR) does not refer to any specific technology. Rather, it’s an umbrella term for any technology that alters reality by adding digital elements to the physical or real-world environment to any extent, whether that be by blending the two together or creating a fully virtual experience. This includes, but is not limited to, augmented reality (AR), mixed reality (MR), and virtual reality (VR). Since the“X” in XR stands for any variable or letter of the alphabet, the term also leaves room for future technologies as well.

XR includes the entire spectrum, from "the complete real" to "the complete virtual.” The connotation of the phrase refers to the extension of human experiences, particularly those relating to the senses of existence (represented by VR) and the acquisition of cognition (represented by AR). However, with the continuous development of human-computer interactions, this definition continues to evolve too.

The Importance of XR

The power of extended reality comes from its ability to stretch, bend, and reshape conventional ways of seeing and engaging with the world. For example, virtual reality can allow architects to envision buildings without any physical materials or help workers to understand the repair and assembly of jet engines more thoroughly and safely by giving them a 3D model to work with.

With augmented reality, a synthetic overlay viewed through smart glasses allows users to see their environment with new virtual elements. For example, a technician may view instructions for repairing a real-world element, eliminating the need for an instruction manual, or a consumer may view products like furniture in their home before buying.

In addition, AR and VR can be combined to create intriguing possibilities, categorized as MR (mixed reality). For instance, people could meet virtually in a collaborative workspace and view physical objects, such as the prototype of a proposed new product, or an engineer could enter a building that’s under construction and see what the wiring will look like once it’s completed to assess potential problems.

With these experiences, a head-mounted display, a pair of glasses, or even a web browser or smartphone becomes a window into a new way of viewing the world, allowing users to see, hear, and explore their environment like never before. This innovative way of interacting is more immersive and interesting, kicking stagnant slide presentations and similar materials to the curb in favor of “hands-on” virtual engagement. With this technology, people from all over the globe can connect, collaborate, and interact with one another in real-time on incredible projects and experiences.

The field of XR is full of endless possibilities for businesses and organizations of all kinds to connect to their customers and clients. And many are catching on to this potential. In 2020, the XR market was valued at $26.05 billion and continues to grow. In fact, according to XR Today, experts believe that XR will grow to be a $463.7 billion market by 2026.

What is UX?

According to a study from the Oxford journal Interacting With Computers, UX design is all about “improving customer satisfaction and loyalty through the utility, ease of use, and pleasure provided in the interaction with a product.” In other words, user experience (UX) is about striving to create a valuable and positive experience for customers, which involves designing products—digitally or physically—so that they are not only useful but also pleasant and easy to use.

UX is a broad term that applies to any and all interactions a person has with company’s products, whether that be services, websites, or physical merchandise. This requires considering a users’ preferences, perceptions, emotions, and more. Good UX design also strives to find the balance between the business's goals and needs and the needs and desires of the user.

User Experience for Extended Reality

Creating a positive user experience for XR is very different than traditional UX design. Since the experiences aren’t screen-bound, the environment that the user interacts with is much larger, not to mention multifaceted. The 3D elements and spaces must be taken into consideration as well as the physical comfort and safety of the user. With all of these points of concern and the fact that these technologies are still an emerging and growing field, there are few set standards for crafting suitable UX for extended reality (XR) experiences. A few proposed frameworks have emerged, but in comparison to other mediums, these guidelines are minimal and could be expanded. Some of these guidelines include:

•   Organize the spatial environment to improve efficiency
•   Create flexible interactions and environments
•   Prioritize users’ comfort
•   Do not overwhelm the user
•   Design around hardware capabilities and limitations
•   Develop cues to help users through their experience
•   Create a compelling XR experience
•   Build upon real-world knowledge
•   Allow users to feel in control of the experience
•   Allow for trial and error

In the book UX for XR, Cornel Hillmann suggests that object-oriented UX methodology (OOUX) might be a good approach for solving design problems in the XR world. A typical UX process starts with user research and user flows and then progresses to wireframes and mockups, usually defining the flows, interactions, and features before defining the objects. But OOUX breaks down the complexity of a design problem by focusing on the core content—the objects—first and assigning actions to them afterward. For example, using the OOUX design process, you would first focus on the button itself and then what action to assign to it. This approach typically has four phases:

1. Discover objects
2. Define objects
3. Establish relationships
4. Force rank objects

UX for Hand Gestures

Even with such writings and guidelines, UX for XR spaces is still largely uncharted territory. Such experiences are complex and require careful attention and smart strategies for the best results. A strong foundation is essential when defining gestures in your vocabulary, which is the crux of user experience in an XR experience. The guidelines below will help to refine this process and can be applied in any situation, regardless of context, devices, or location on the XR spectrum.

•   Use affordances and signifiers: Affordances and signifiers are just as important in an XR context as they are in any other, if not more so. What can be done? What can be pressed? Provide a real-world analog for these actions. Relying on assumptions or for something to simply “look like” it can prompt an action isn’t enough. Users should be clear about what they can do with elements, especially when using such new technologies that they may not be familiar with.
•   System feedback is imperative: When a user pushes a button, pulls a lever, selects an object, etc., they should receive feedback to show them the action has been successfully taken, even if in a partially-immersive environment. In the best-case scenario, the user understands that their gesture is being tracked and when the system has successfully processed the action to prompt a result. This can include visual, audio, or haptic cues or a combination of any of the three.
•   Have a sandbox: Gesture vocabularies will vary from one context/platform to another. Some gestures might carry over while others may not. Give the user time and space to learn and practice these gestures to gain proficiency so they can proceed more comfortably and easily into the experience.
•   Onboarding is necessary: You can’t assume a user knows anything about your experience and your gesture vocabulary. In an XR context, disorientation is an even greater issue than it would be in a non-immersive experience, as it can be more frustrating and uncomfortable in such an environment. Therefore, present onboarding as a default, and allow the user to choose whether or not to see it again in the future, making tutorials easily accessible at any point during the experience in case it may be needed. Of course, onboarding and your sandbox should be completely accurate and up to date at all times, and you should have in-system notifications for any changes to these fundamental pieces.
•   Keep the user physically safe: You should do everything possible to keep the user safe and not put them in a position that could result in discomfort, injury, or harm. For example, striking the body or any movement that obscures vision should be avoided as it can lead to a precarious situation. Likewise, don’t spawn important objects behind other objects, as they can be difficult to track and cause issues. Elements shouldn’t sneak up or seek to startle users, either. It may seem that these types of experiences would be impactful, but they can cause discomfort, distress, and mistrust that leads to poor user experience. Only in rare cases, when the user can anticipate them, such as in a horror game, can these types of interactions be used. But there should never be any actions or gestures that could put the user in danger, even in these situations.
•   Don’t be unduly exclusive: Accessibility is a necessity. Making an experience exclusive to only a certain group of people not only creates a poor user experience but minimizes potential clientele. An experience should be available to as many people as possible. Don’t make assumptions about what a user is capable of or has at their disposal. For example, don’t assume a user has use of five fingers, doesn’t wear glasses, or has a certain body shape. When creating an experience, the technology should be able to adapt to any type of person or ability. This can be achieved by creating gestures and contexts that are open to all or that offer accommodation. Of course, making an entirely inclusive experience can be difficult, but it is important to take the time and care to accommodate as much as possible.

Key Challenges of UX for XR

Good UX does not simply emerge on its own, it requires a great deal of effort, analysis, user research, and testing to get right. The solutions that work best are rarely those that come to mind first.

There are many challenges to creating a good UX for XR. The list below is by no means exhaustive, but it does help to illustrate the importance of adopting rigorous UX design processes when developing XR applications and experiences.

•   Selecting an area of focus for interactions

A user needs to be able to indicate which item in the interface they want to interact with. This can be a challenge due to how the system defines which specific application, window, or field should be the main focus in a user interface when there can be multiple layers open at once and several interactive panels arranged 360° around the user. Up until now, this has been done with pointer tools such as Raycast, but smarter and more efficient solutions are emerging.

•   Direct manipulation

There’s a promise of better hand tracking and full-finger articulation thanks to developments being made by those like Leap Motion (recently acquired by UK company Ultrahaptics) and HoloLens 2. It has always been a goal of XR to mimic the natural movement of hands, and this advancement has the potential to be a game-changer in the near future. But for now, other methods must be used to try to imitate these gestures as realistically as possible.

•   Communicating interactive affordances

Users don’t always understand what parts of the experience are interactive. In many cases, users try to interact with non-interactive items or miss elements. That’s why it’s crucial that the user interface (UI) indicates what can be interacted with and what can’t be to make for the best and most complete experience. Such functionality should be familiar and as intuitive as possible to make for the smallest learning curve and most natural interaction, but this can be a challenge considering that the technology is often an unfamiliar medium.

•   Ensuring intuitive interaction via controllers

Hand-held controller mapping and on-screen actions are not always clear for users, who need to know when, how, and where to press buttons in the experience. This issue is exacerbated in VR games where the controller is difficult to visualize in the 3D environment. Some users may even end up removing their headset to find the button they’re looking for, which does not make for a pleasant or immersive overall experience. Therefore, ensuring that these controls are clear and easy to handle is essential for keeping users engaged.

Final Thoughts

The field of XR will continue to progress in these areas and work through these challenges, among others. Quickened by the pandemic, the medium is growing and evolving quickly, with new products and innovations cropping up regularly. The industries interested in using XR are continuously expanding; categories such as entertainment, tech, auto, consumer packaged goods (CPG), and more are discovering the potential of XR as a tool to grow their businesses and interact with customers in new ways every day. With such exponential growth, the tools and guidelines to create the best UX possible for these experiences will be refined and become more readily available. Together, these developments will help to evolve and expand XR as we know it.