UX for XR — User Experience for
Extended Reality
There’s no doubt that VR and AR are extremely innovative experiences, allowing us to use our movements and gestures to manipulate and interact with virtual objects. However, most wrongly assume that this immersive tech is ready to use out of the box. Evidence has been consistently found that proves the user experience of AR and VR remains in its infancy and that there are many opportunities for improvement.
Extended reality (XR) does not refer to any specific technology. Rather, it’s an umbrella term for any technology that alters reality by adding digital elements to the physical or real-world environment to any extent, whether that be by blending the two together or creating a fully virtual experience. This includes, but is not limited to, augmented reality (AR), mixed reality (MR), and virtual reality (VR). Since the“X” in XR stands for any variable or letter of the alphabet, the term also leaves room for future technologies as well.
XR includes the entire spectrum, from "the complete real" to "the complete virtual.” The connotation of the phrase refers to the extension of human experiences, particularly those relating to the senses of existence (represented by VR) and the acquisition of cognition (represented by AR). However, with the continuous development of human-computer interactions, this definition continues to evolve too.
The power of extended reality comes from its ability to stretch, bend, and reshape conventional ways of seeing and engaging with the world. For example, virtual reality can allow architects to envision buildings without any physical materials or help workers to understand the repair and assembly of jet engines more thoroughly and safely by giving them a 3D model to work with.
With augmented reality, a synthetic overlay viewed through smart glasses allows users to see their environment with new virtual elements. For example, a technician may view instructions for repairing a real-world element, eliminating the need for an instruction manual, or a consumer may view products like furniture in their home before buying.
In addition, AR and VR can be combined to create intriguing possibilities, categorized as MR (mixed reality). For instance, people could meet virtually in a collaborative workspace and view physical objects, such as the prototype of a proposed new product, or an engineer could enter a building that’s under construction and see what the wiring will look like once it’s completed to assess potential problems.
With these experiences, a head-mounted display, a pair of glasses, or even a web browser or smartphone becomes a window into a new way of viewing the world, allowing users to see, hear, and explore their environment like never before. This innovative way of interacting is more immersive and interesting, kicking stagnant slide presentations and similar materials to the curb in favor of “hands-on” virtual engagement. With this technology, people from all over the globe can connect, collaborate, and interact with one another in real-time on incredible projects and experiences.
The field of XR is full of endless possibilities for businesses and organizations of all kinds to connect to their customers and clients. And many are catching on to this potential. In 2020, the XR market was valued at $26.05 billion and continues to grow. In fact, according to XR Today, experts believe that XR will grow to be a $463.7 billion market by 2026.
According to a study from the Oxford journal Interacting With Computers, UX design is all about “improving customer satisfaction and loyalty through the utility, ease of use, and pleasure provided in the interaction with a product.” In other words, user experience (UX) is about striving to create a valuable and positive experience for customers, which involves designing products—digitally or physically—so that they are not only useful but also pleasant and easy to use.
UX is a broad term that applies to any and all interactions a person has with company’s products, whether that be services, websites, or physical merchandise. This requires considering a users’ preferences, perceptions, emotions, and more. Good UX design also strives to find the balance between the business's goals and needs and the needs and desires of the user.
Creating a positive user experience for XR is very different than traditional UX design. Since the experiences aren’t screen-bound, the environment that the user interacts with is much larger, not to mention multifaceted. The 3D elements and spaces must be taken into consideration as well as the physical comfort and safety of the user. With all of these points of concern and the fact that these technologies are still an emerging and growing field, there are few set standards for crafting suitable UX for extended reality (XR) experiences. A few proposed frameworks have emerged, but in comparison to other mediums, these guidelines are minimal and could be expanded. Some of these guidelines include:
In the book UX for XR, Cornel Hillmann suggests that object-oriented UX methodology (OOUX) might be a good approach for solving design problems in the XR world. A typical UX process starts with user research and user flows and then progresses to wireframes and mockups, usually defining the flows, interactions, and features before defining the objects. But OOUX breaks down the complexity of a design problem by focusing on the core content—the objects—first and assigning actions to them afterward. For example, using the OOUX design process, you would first focus on the button itself and then what action to assign to it. This approach typically has four phases:
Even with such writings and guidelines, UX for XR spaces is still largely uncharted territory. Such experiences are complex and require careful attention and smart strategies for the best results. A strong foundation is essential when defining gestures in your vocabulary, which is the crux of user experience in an XR experience. The guidelines below will help to refine this process and can be applied in any situation, regardless of context, devices, or location on the XR spectrum.
Good UX does not simply emerge on its own, it requires a great deal of effort, analysis, user research, and testing to get right. The solutions that work best are rarely those that come to mind first.
There are many challenges to creating a good UX for XR. The list below is by no means exhaustive, but it does help to illustrate the importance of adopting rigorous UX design processes when developing XR applications and experiences.
A user needs to be able to indicate which item in the interface they want to interact with. This can be a challenge due to how the system defines which specific application, window, or field should be the main focus in a user interface when there can be multiple layers open at once and several interactive panels arranged 360° around the user. Up until now, this has been done with pointer tools such as Raycast, but smarter and more efficient solutions are emerging.
There’s a promise of better hand tracking and full-finger articulation thanks to developments being made by those like Leap Motion (recently acquired by UK company Ultrahaptics) and HoloLens 2. It has always been a goal of XR to mimic the natural movement of hands, and this advancement has the potential to be a game-changer in the near future. But for now, other methods must be used to try to imitate these gestures as realistically as possible.
Users don’t always understand what parts of the experience are interactive. In many cases, users try to interact with non-interactive items or miss elements. That’s why it’s crucial that the user interface (UI) indicates what can be interacted with and what can’t be to make for the best and most complete experience. Such functionality should be familiar and as intuitive as possible to make for the smallest learning curve and most natural interaction, but this can be a challenge considering that the technology is often an unfamiliar medium.
Hand-held controller mapping and on-screen actions are not always clear for users, who need to know when, how, and where to press buttons in the experience. This issue is exacerbated in VR games where the controller is difficult to visualize in the 3D environment. Some users may even end up removing their headset to find the button they’re looking for, which does not make for a pleasant or immersive overall experience. Therefore, ensuring that these controls are clear and easy to handle is essential for keeping users engaged.
The field of XR will continue to progress in these areas and work through these challenges, among others. Quickened by the pandemic, the medium is growing and evolving quickly, with new products and innovations cropping up regularly. The industries interested in using XR are continuously expanding; categories such as entertainment, tech, auto, consumer packaged goods (CPG), and more are discovering the potential of XR as a tool to grow their businesses and interact with customers in new ways every day. With such exponential growth, the tools and guidelines to create the best UX possible for these experiences will be refined and become more readily available. Together, these developments will help to evolve and expand XR as we know it.