Augmented Reality – Interactive and Immersive Design

Introduction:

We have probably witnessed the applications and possibilities it brought about in many aspects of daily life, such as entertainment and gaming. But Augmented Reality is more than a BeautyCam filter or cutting fruits on Fruit Ninja, AR applicability extends to practical fields including medical, military and education. Although we might have unintentionally encountered some common applications of AR already, we are not necessarily aware that they are AR-based because the terminology seems elusive and abstract, for example, what exactly is being augmented? What are the ways of augmenting? What is the ultimate purpose of the augmentation?

The definitions of AR vary, but in essence they all indicate a characteristic, Augmented Reality can be perceived as a medium where digital information overlaps with the physical environment (Craig, 2013). In Craig’s work Understanding Augmented Reality: Concepts and Applications, he proposed that “the ultimate goal of augmented reality is to provide the user with a view of the surroundings enriched by virtual objects”. Indeed, humans have been modifying the surrounding conditions of the reality to make living easier since day one. However, it was not until the emergence of Information Age did the majority of the alteration shift from sufficing survival to gaining as much as information as possible. Today, digitalized computers allow enormous amounts of information to be retrieved, saved and available for manipulations speedily. One can easily find traces in this respect in AR applications, let’s take the simplest example, the digital maps allow us to gain information of a certain place that we are not physically placed at. While we are using the application, we get a faster comprehension (than actually getting to the place to gain information), a possibility of gaining information. In Engelbart’s 1962 work Augmenting Human Intellect: A Conceptual Framework, the author defined the concept “augmenting human intellect” as increasing capability to face a complex problem, to gain comprehension to suit particular needs, eventually to resolve the previously complex problem. Based on this connection, the ultimate goal of AR is to challenge and redefine the existing reality, to derive corresponding solutions to the emerging problems. From this ongoing process, not only the amount of information is augmented, but also human intellect.

In this paper, I will discuss two of the essential design principles developers adopt to improve the usability of Augmented Reality applications. Sorted by hardware devices, software bases, applicable fields and so on, the number of applications can be innumerable. Applications and settings can be infinite, depending on human’s initiatives and technological bedrock. For this reason, this paper will only focus on mobile augmented reality (MAR) experience.

1.1 Interaction design

In any interactive design, it takes computer intellect to form a platform and human intellect to comprehend. Perhaps we can refer this experience to the metaphor of watching a movie, while the lighting, angles and tones of the set can be as compelling as possible, it is the interpretation of how the observation constructs meaning in real world situation that helps viewers understand the story it intends to convey.

1.2 Elements to interact

Although AR is designed to be interactive, this process is not always visible. It is hard to be fully aware of the interactions going on in the space and time, for instance, it remains ambiguous to most people about what “reality” is being augmented and what the virtues of the augmentation are. To better understand AR and take an active role in participating in the interactive process, one must determine what is there to interact with and the underlying design techniques that enable it.

The definition of Interaction Design (IxD) is abstract yet self-explanatory in the title. To successfully interact, both product and user need to contribute their share of effort. As Gillian Crampton Smith proposed, Interaction Design consists of 5 dimensions, 1) words, 2) visual representations, 3) physical objects/space, 4) time, 5) behavior. The first four dimensions encompass what products and services (digital/non-digital) have to offer, while the fifth dimension (behavior) stresses the importance of the user interface, in this respect, users are encouraged to realize their goals and objectives as much as possible by using the products.

1.1.1 Words

History and culture endow characters and letters with specific meanings. In IxD, words serve as one of the essential elements to improve the usability. In common with any other application, A successful AR application should have enough words to explicate the instructions and elucidate the usage, allowing users to form an understanding of what the next step is and what goals can be achieved using the application. The amount of words should be concise enough to make clear the objectives instead of providing overwhelming information.

1.1.2 Visual Representations

“Humans are visual animals”, this statement holds true in the context of using application. In line with the first element, visual representations adorn applications with cognitive symbolism. For instance, we have long figured out that hands can be utilized to grab and drop objects, so when the cursor turns into a hand shape, we know it means the targeted files can be moved to almost any other spots on the screen. In a word, affordance that suits its intended usage is appreciated in an application. Most of the time, instead of giving out wordy instructions to proceed, simply put out a button-like representation and the usability is enhanced by the suited affordance. (See Fig. 1)

(Figure. 1. On apple’s measure app, words as descriptions and cinemagraphs are presented to indicate the possible movements the application can recognize. Visual representations such as images and videos deliver an instant instruction for users.)

 1.1.3 physical objects/space

The third dimension takes context into account, it is the physical environment within which users will be interacting with the products. Since this paper mainly discusses about design principles of Augmented Reality applications on mobile devices, the object (the device as virtual window through which users experience the products) is the mobile devices such as laptop and smartphones, the range of space (physical environment where users use the products) can be as broad as desired enhancement of environment can be achieved. (See Fig. 2)

(Figure. 2. Real-time maps allow users to garner information from anywhere on the maps, this process can be done by almost any digital devices and anywhere as it is a personal context.)

1.1.4 Time

 This concept can be interpreted as the time that users spent on interacting with the application. Users get feedback (audible/visual) from the application and over time, participate in a complete interactive process with the application. Reasonably timed feedback from products is crucial in constituting this dimension, as users gain further instruction and information from feedbacks to their actions hence the steps of interaction unfold. The amount of time depends on the capability of specific applications and the depth of purpose that users intend to obtain. (See Fig. 3)

(Figure. 3. Amazon’s Augmented Reality function allows users to view products on the intended surface before purchase. The products move around the room with the motion of the user’s fingers and when placed, devices vibrate to indicate that the product has “dropped” on the surface. When surfaces are not detected by the device, there will also be words and visual representations on the screen explaining the error occurred.)

1.1.5 Behavior

 In relation to the user interface, behaviors are considered as a range of actions conducted by users to interact with the product, including operation, presentation and reaction (Kevin Silver, 2007). In IxD, the first four dimensions integrate at this step, shaping users’ behaviors (i.e., the predefined possibilities or constraints of command) and encouraging users to create a personalized experience. 

1.1 Summary

 Computers are participatory medium (Murray, 2012). As a computational system, interactive is an innate feature of augmented reality experience. The level and quality of interactions depend both on computer and human interface, how effective does an application present its ideas and provide clues to users, the first three dimensions (words, visual representations, physical objects/space) can be directly improved in design processes, while the last two dimensions (time, behavior) are engaged with user interface, thus they are influenced but not straightforwardly altered by the technical modifications. However, they can be directed to develop positive and compelling interactions with the application if the elemental designs are successful (e.g., timely feedback).

2.1 Immersion technology

Most forms of media utilize certain senses of human body. We can read a book, listen to the radio, however, within the framework of Augmented Reality (AR), using only eyes, ears and hands would not have achieved an optimal experience for users. That is what differentiates AR from other forms of media – an immersive user experience (Craig, 2013). 

 As far as the current development of AR applications on mobile devices is concerned, to assert that AR provides a complete immersion would be somewhat unrealistic. Unlike Virtual Reality (even VR has its limitations, e.g., locational mobility), Augmented Reality leaves the users with connections with the physical world, meaning it has sensual boundaries of the environment it shapes. Applications and settings of AR can be infinite, depending on human’s initiatives and technological bedrock. However, full immersion has yet to be achieved.

 Notwithstanding the efforts in progress, the total immersion in most AR applications has not been successfully registered in the physical world, due to both the limitations in design techniques and the human factors (Aukstakalnis et al, 2016). So far, there’re few academic works and pragmatic studies on AR’s total immersion theme. Whether MAR total immersion is a realistic goal remains in question, however, feasible means are discussed in existing literature about how enhanced immersion in AR applications can be achieved by ameliorating organizational/modular system (hardware and software components). Given that this paper only discusses MAR, the ensuing discussions will be centered on the software layer. 

2.1.1 Sensual design

 Visual design covers a range of standards in software components such as image processing and recognition. Studies have indicated that humans garner information mostly (80-85%) by visual system, largely exceeding that by other senses (Politzer, 2015). Human eyes receive lights reflected by objects, then stimulate the cognitive system in brains to process and recognize objects. For this reason, visual design is a pivotal factor in AR immersion because it decides whether users are able to establish beliefs in the digitized environment while creating little maladjustment. (See Fig. 4 and 5)

As Fig. 4 and 5 presented, the digitized information (the animation character and spider) is a virtual layer overlaps the actual environment captured by the camera. Because the figures do not blend in seamlessly with the real-world environment, the degree of immersion (the level of disbelief in virtual environment) is low compared to those applications that take in-depth simulation (e.g., brightness, contrast similar to that in actual environment) into consideration. (See Fig. 6)

(Figure. 6)

 In contrast, Civilisations AR, an application launched by BBC, packs more pixels into given screen areas, improving the quality and authenticity of the digitized information, thus telling a more compelling story. 

 Other improvements to the software layer can also be made to generate a more immersive experience for users, such as feedback timing, which affects the latency in human-computer interaction (UCI).

2.2 Limitations

 For immersive goals, developers must consider improvements both in hardware (e.g., head-up displays, two-handed panel, etc.) and software components. For the limited scope in this paper and a few instances and cases in existing business and academic fields, the last part was not able to develop comprehensively. Moreover, for the lack of time, other feasible software improvements were not presented (e.g., audio effect, object recognition, etc.). Moreover, since the users’ interaction plays a vital role in creating experiences, human factors also need to be taken into account, because AR is essentially a hybrid image of the digitalized and the physical, so users might choose certain information but not all to process (Bolter et al, 2013).

3.1. discussion

 As we discussed in the previous paragraphs, augmented reality is both an interactive and partially immersive experience. Although whether total immersion of AR remains unsolved, the subject itself is designed to be heuristic, which means that in AR design process, user experience is not the main concern but rather the challenges it can impose on the reality and the derivative solutions to the problems. With further development, AR has the potential to reach a ubiquitous level, advanced immersion design (images, audio, feedback) relates to interaction design and provide users with better AR experience, conversely, interaction design helps users develop refined immersion experience.

———————–

References:

Engelbart, D. C., and Friedewald, Michael. Augmenting Human Intellect a Conceptual Framework . Fremont, CA: Bootstrap Alliance], 1997., 1997. Print.

Interaction Design Foundation, The Encyclopedia of Human-Computer Interaction, 2nd. Ed. https://www.interaction-design.org/literature.

Sziebig, Gabor. (2009). Achieving Total Immersion: Technology Trends behind Augmented Reality- A Survey.

Fischer, Jan & Bartz, D. & Strasser, W.. (2005). Stylized augmented reality for improved immersion. IEEE Proceedings. VR 2005. Virtual Reality, 2005.. 2005. 195-325. 10.1109/VR.2005.71.

Jacobs, Marco, Livingston, Mark, and State, Andrei. “Managing Latency in Complex Augmented Reality Systems.” Proceedings of the 1997 Symposium on Interactive 3d Graphics. ACM, 1997. 49–ff. Web.

Aukstakalnis, Steve. Practical Augmented Reality: A Guide to the Technologies, Applications, and Human Factors for AR and VR. Old Tappan, NJ: Addison-Wesley Professional, 2016.

Dunleavy, Matt. “Design Principles for Augmented Reality Learning.” TechTrends: Linking Research and Practice to Improve Learning 58.1 (2014): 28–34. Web.

Murray, Janet H. Inventing the Medium : Principles of Interaction Design as a Cultural Practice . Cambridge, Mass: MIT Press, 2012. Print.

Craig, Alan B. Understanding Augmented Reality: Concepts and Applications. Waltham, MA: Morgan Kaufmann / Elsevier, 2013.

Bolter, Jay David, Maria Engberg, and Blair MacIntyre. “Media Studies, Mobile Augmented Reality, and Interaction Design.” Interactions 20, no. 1 (January 2013): 36–45. https://doi.org/10.1145/2405716.2405726.

Choudary, Omar et al. “MARCH: Mobile Augmented Reality for Cultural Heritage.” Proceedings of the 17th ACM International Conference on Multimedia. ACM, 2009. 1023–1024. Web.