How to direct and maintain attention in augmented reality applications.

Co-Authors: Cat Hodges, Hannah Nye, and Kay Stanney

Augmented reality (AR) is a powerful tool that enhances our real world with overlaid virtual information. ARs use ranges from virtually testing the look of new furniture in our homes before purchase, to exploring virtual worlds layered onto our own (and catching fantasy creatures while doing so!), to viewing notifications and data that correlate to our world, and beyond. AR expands our environment into the extraordinary and is exploding in both the commercial and government sectors, with an estimated CAGR of 43.8% from 2021 to 2028.

Of particular importance may be the potential AR has to completely restructure the way we educate, train, and operate. Opportunities to enhance the enterprise for personalized instruction, improved quality of work, and increased productivity are endless. But, in order for us to harness this power, we have to ensure that our users are correctly attending to relevant AR information and not getting lost in the shiny new toy that AR presents itself as. The bottom-line is we must be careful to design our AR applications in such a way that our users attention is directed appropriately.

Attention, Attention!

Before we can direct and maintain our users attention in AR apps, we first have to understand exactly what attention is. In general, attention is an extremely complex phenomenon that encompasses many processes of cognition. The earliest definition of attention dates back to 1890 when William James argued that attention was:

taking possession by the mind, in clear and vivid form, of one out of what seem several simultaneously possible objects or trains of thought.

Since then, modern psychology has simplified the definition of attention as:

the act of intentionally focusing on specific stimuli (objects, features, thoughts, etc.) while intentionally ignoring other stimuli (Goldstein, 2018).

This argument assumes that attention is a conscious and intentional choicemeaning we actively choose what we do or do not pay attention to. There are many ways we can pay attention to the world around us: we can divide our attention between multiple things, sustain it for a really long time, selectively choose to focus on one thing over another, and even alternate and switch our attention between many objects.

Designing for Attention within AR Applications

Now that we know what attention is, lets discuss some best practices for designing our AR applications with the intention to direct and maintain our users attention.

Create a Narrative

Humans have an innate desire to connect, see patterns, and seek meaning in the world around us. As UX practitioners, we can use the power of narrative to capture attention and keep people engaged in the augmented world around them. Maybe you dont think youre a storyteller, but you can be! Narratives often follow the same formula, with some differences in-between genres. To severely oversimplify how to create narrative: (1) start by setting the scene, (2) create steady rising tension that leads to a climax, and then (3) wrap it up at the end.

Some experiences, like video games, lend themselves to narrative storytelling, but what about more utilitarian applications? We can still use elements of narrative to sustain attention by giving the voice of our application a consistent tone and personality.

Use Stimulating Visuals

Sight is the sense we tend to lean the most on to gather information about the world around us. With the current AR hardware situation, AR tends to be a mostly visual experience. In order to attract and maintain our users attention, we should thus invest a large portion of our design time creating a visually stimulating experience. A key aspect of attention-grabbing visuals is motion. You see, way back when humans were evolving, motion = possible predator = death (yikes!). Weve even got structures in our eyes, called cones, that help us with our spatial acuity. As such, were hard-wired to hold our attention to visual motion. We can use this biological preparedness to leverage visual motion in AR apps in order to direct attention to an area of interest. Much like our eyes, the field of view for AR headsets is limited. So, what do you do if you want the user to interact with something outside the field of view of an AR head worn display (HWD)? Use visual motion! Even though our peripheral vision is not good at most things, it is good at detecting movement. By animating an AR object, UI, etc. on the perimeter of your users peripheral vision, you can get them to quickly disengage from whatever they are currently interacting with and turn towards the thing you want them interacting with now.

Spatial Audio

Within human sensation and perception, sound has an extreme impact on our experiences. This is apparent in the TV shows and movies that we watch. A slight change in the tone of sound can warn the audience of something suspicious that is about to occur; while a fun, playful song can be used to set the tone for a care-free car ride with two friends. Thus, beyond visuals, audio is also critically important when drawing attention in AR. With the capability for spatial audio, we can direct users attention to a direction in world space via sound. For example, in a narrative experience that encourages exploration, we can direct the users attention to a physical table behind them by playing music from a virtual radio overlaid onto the real table. Alternatively, a chime or alarm sound spatially connected to an AR object could be used to orient the user to a particular object. A notable use of spatial audio is the Windows mobile application, Soundscape, that provides audio cues for users who are blind or vision impaired. During navigation, it plays a friendly beat when the user is oriented in the correct direction. When the user goes off course, the sound stops. This helps to keep the user on track and is an innovative way to maintain attention via sound cues; in such an application the absence of sound would be a signal for gaining a users attention. Whether for exploration, orientation, or navigation, spatial audio is a simple but effective way to direct a users attention in AR applications.

Avoid Overload

When it comes to attention, less really is more. If an AR app is designed such that everything is vying for the users attention, then likely nothing will get attended to. Its thus important to be cognizant of what else is going on in your users AR environment when youre trying to get them to pay attention to something in particular. We wrote a whole article on guarding against overload in AR that can be found here. To summarize it quickly, you can prevent overload by i) balancing the modality of cues and information you present in AR, ii) mimicking users existing mental models, and iii) taking a minimalistic approach to your AR app designs.

Breaking Conventional UX ParadigmsThe World is a Stage

The early days of computing were like the Wild West. Sure, human factors had been around for ages, but people werent certain about how to apply it to this brand-new computing medium (sound familiar?). It wasnt until the ingenious minds at Xerox PARC started abstracting real-world metaphors to computer systems that our current UI paradigm of WIMPS (Windows, Icons, Menus, and Pointers) was born. Since that time, weve gone from skeuomorphism (derivative 3D design) to flat (2D minimalistic) design, and now neomorphism (3D minimalistic). Now that going three-dimensional is all the rage, we should, as Xerox PARC did, once again look back to the real world for new metaphors that explain the capabilities and bounds of our AR systems.

With respect to attention, we can borrow concepts from the well-established medium of theatre. Practitioners of stagecraft have been using narrative, lighting, stimulating visuals, audio cues, and sometimes, even smells, to direct and maintain attention for centuries. We can easily apply these concepts to AR apps. In this context, the audience is our user, the stage is our real world that is being augmented, the actors are the AR overlays we are to attend to, the orchestra is the voice and tone of our AR narrative, and the proscenium is our users field of view in the AR HWD. There are so many parallels between the two! One aspect that is heavily exploited in theatre, but severally underrepresented in AR is lighting. Lighting is a subtle way to direct attention without using distracting (and possibly mental workload inducing) visual indicators. We want our users to direct their attention to the intended AR target, NOT some type of WIMP icon (e.g., an arrow) directing us to look at the AR entity. In fact, there is even a spotlight theory of attention that describes how our visual systems operate like a spotlightit moves around our field of vision and notices important details (sounds just like theatre spotlights, right?)! When combined with color theory and sound, we can also use lighting to create a specific mood (remember your narrative!). For example, a red spotlight over an object coupled with daunting music could invoke a feeling of danger, while a blue spotlight coupled with lighthearted music could invoke feelings of tranquility. What lighting and sound cues have in common is that they can be designed into an ambient user interface, often seen as the holy grail of UI design, which seamlessly directs the AR UX in a hyper-subtle and highly intuitive manner. The point being that the same cues used by stage directors on Broadway and the West End can also be leveraged in AR applications to direct and maintain our users attention. I dont know about you, but I could have watched at least two more hours of Hamilton content (I needed more Layfette/Daveed Diggs) and those fantastic visual, auditory, and color cues would have made sustaining my attention as easy as overthrowing the British Government.

TL;DR

Intentional attention in AR can be designed and there are several design practices for implementation:

• Develop a narrative (or a personality). The first step to gaining and maintaining attention in AR apps is to keep your user engaged in a compelling story or consistent tone and personality.
• Create stimulating visuals. Our brains are hard-wired to notice visual motion, even in our periphery; use this innate human capability to garner attention in AR apps.
• Dont forget spatial audio. Incorporating spatial audio into AR apps can help to orient a user in the correct direction without imposing high levels of mental workload.
• Avoid overload. Less is more when trying to direct attention in AR appsa balance between visual, sound, and color cues and information can help users focus on whats important.
• Consider theater. This creative form of narrative provides a plethora of intuitive and highly effective concepts for directing attention in 3D space and could be readily applied to AR appsincluding lighting and sound as attention directors.
• Aim to wrap all of these elements into an ambient UI. An ambient UI is the holy grail, as it has the potential to provide a means of seamlessly and subtly directing the attention of AR users in augmented real-world spaces.

How might we better lead our users through AR? A combination of the given design practices will help to give users a directed experience. Guiding users through AR is importantit is a new medium that can be confusing if not designed correctly and users who are confused can ultimately write off AR as a futuristic technology not ready for real-world application. But we know this is not true! Its exciting to realize that virtually everyone has the tool to experience AR right in their pockettheir Smartphonethe only hurdle we face now is to design these applications such that they truly empower their users.

References

James, W. (1890). The Principles of Psychology. New York: Henry Holt.

Goldstein, B. E. (2018). Cognitive Psychology: connecting mind, research, and everyday experience. Cengage Learning. IS

https://medium.com/re-form/what-is-narrative-ux-9400664660af

https://www.smashingmagazine.com/2010/01/better-user-experience-using-storytelling-part-one/

Maintaining attention in AR was originally published in UX Collective on Medium, where people are continuing the conversation by highlighting and responding to this story.