There have been several attempts to theorize computation and cognitive technologies as we approach an age of ubiquitous computing. While there is a large body of literature that focuses on how to maximize the efficiency of interface design as we approach such an era, there is a lack of theoretical discourse. As interface is the mechanism by which we engage with computers, a revision to theoretical frameworks of interface is necessary in an age of ubiquitous computing. Semiotics lends much in the way of theorizing generative meaning making as well as recursive symbol processing. Applying the study of semiotics to a discussion of interface leads to a move away from the desktop metaphor and the prominence of the GUI interface, and towards more integrated, multi-faculty interfaces that map to distinct layers of information in space. Furthermore, it lends to a process of unlimited semiosis in interface design and diversity in interface metaphors.
Semiotics has offered much in the way of how we theorize about our cognitive technologies. From symbolic logic, to cultural institutions, semiotics has provided conceptual frameworks for analysis as well as for conducting research, such as communication in computer mediated environments; the mapping of signal processing in machines and symbol processing in humans; and the role of Semiotic Engineering in issues revolving around HCI (de Souza 415-418). In what follows, I will use semiotic approaches to interface design to contribute to a discussion of interface in a paradigm shift of computation from a single user with a single computer system to models of computation that take growing ubiquity into account.
Cognitive Technologies as Artefactualization of Conceptual Sign Systems
The triadic model in Peircean semiotics breaks down sign processes as such: “A sign, or representamen [the material-perceptible component], is something which stands to somebody for something in some respect or capacity. It addresses somebody, that is, creates in the mind of that person an equivalent sign, or perhaps a more developed sign. That sign which it creates I call the Interpretant of the first sign” (Peirce 16). We can think of the three components of sign processes (Representamen, Object, and Interpretant) as such: sense data, concept, and percept. Each component of a sign process is a sign system in itself, in much the same way Ray Jackendoff treats the different components of spoken language, to be discussed later. As much semiotic literature has found, the fundamental components of human cognition break down to this system of sign processes.
Of interest to a discussion of interface in digital technologies is the instantiation of sign systems through sign processes. To interpret a sign process is to reinforce the sign systems that form the foundation of that sign process. A single painting reinforces the medium (or sign system) of painting, which reinforces visual art, which reinforces artistic expression, which reinforces and preserves human culture. All our cognitive technologies and symbolic expressions exist in an ongoing continuum: a historically constructed network of artifacts, ideas, languages, cultures, media, technologies – anything constructed by humans (Irvine 43). This idea is critical in a discussion of interface in cognitive technologies, because the interface mediates the sign systems of our cognitive faculties with system architecture sign systems, to be elaborated on further later.
Most importantly, cognitive technologies are artefactual instantiations of sign processes. Unlike spoken language, computer mediated sign processes are tokenized instantiations of meaning making. This ties in with the concept of “vital materiality,” or the concept that our artifacts are not a product of our symbolic meaning systems but rather mediate our relationship with the world and are indicative of the human species’ shared experiences (Barrett 10). In other words, artifacts are codified with meaningful and interpretable human data, and furthermore are a necessary component in meaning making in the continuum of symbolic cognition. Cognitive technologies afford artefactualization of conceptual sign systems, and thereby an artefactual network of meaning systems. The more this concept is realized in computational technologies, the more pervasive our artefactual representations of sign systems; the more we computationally remediate sign processes and sign systems, the closer we get to a major paradigm shift in the role of cognitive technologies.
The New Paradigm
One perspective on dealing with this new paradigm in HCI is a transition from thinking about direct manipulation and object-oriented computing to navigation of information spaces (Benyon 426). Thinking about computing as navigation of information spaces frees up the study of HCI from a single user and single system to a larger system of information spaces. In the words of Benyon, “As computing devices become increasingly pervasive, adaptive, embedded in other systems and able to communicate autonomously, the human moves from outside to inside an information space” (426). What this refers to is how computation is mediating more of our sign processes, from the workplace, to museums, to politics, to entertainment, and far beyond. Furthermore, the information in each individual system is unique to that system, which is how we move between various information systems. While the topic of ubiquitous computing is a bit out of the scope of this paper, it will point its eye toward the idea of ubiquitous computing as a motivator for this paradigm shift in HCI.
This idea of technical systems as information spaces ties directly with the notion that we offload symbolic information into artifacts that become part of our cognitive processes, in a process known as “external symbolic storage,” (Renfrew 4). The idea is that our artifacts are part of the cognitive process of preserving culture and information, both in terms of writing as well as architecture, art, pottery, etc. This idea is applicable to the idea of technical systems as information spaces as technical systems are cognitive artifacts. As Peter Bogh Andersen writes, computer systems are layers of sign systems: “If we continue this descent through the different layers of the system, passing through the operating system and the assembly code, down to the actual machine code, we will encounter signs most of the way” (Andersen 6). Even down to the machine code, we can treat sequences of electrical signals as a sign system. Because of this, when we encounter an information system, we are effectively navigating into an information space and its network of information spaces. This is true whether we are navigating the World Wide Web from a personal computer, accessing client information through an employer’s intranet, or ordering a sandwich at a deli kiosk, just as much as the spaces themselves flood us with information about whether it’s a home environment, professional environment, or commercial environment through architecture, décor, and social contracts within those spaces.
The globality of sign systems, especially as it permeates technical computer systems, is indicative of how we offload and automate those sign systems. In the words of Jeannette Wing, “Computational thinking is using abstraction and decomposition when attacking a large complex task or designing a large complex system” (Wing 33). If we think of our artifacts, including technical systems, as organizing complex computational tasks in the everyday generation and preservation of culture, the emergence of computer systems playing central roles in human culture is an extremely confluent event (not to fall into any mystic determinism about emerging technologies). And it is no mistake that computers are layered with sign systems much like our own cognition – computation is a type of human logic. We use computer systems to more efficiently compute, store, and organize the processes and products of this logic.
Further speculation on this perspective of HCI leads to the realm of augmented space or reality. In a ubiquitous computing environment, augmented space refers to the dynamic information spaces layered over physical spaces – more a cultural and aesthetic practice than technical (Manovich 220). This refers to more than just the virtual data in a given space, but the layers of abstraction in a given space. As Manovich writes, “If previously we thought of an architect, a fresco painter, or a display designer working to combine architecture and images, or architecture and text, or to incorporate different symbolic systems into one spatial construction, we can now say that all of them were working on the problem of augmented space – the problem, that is, of how to overlay physical space with layers of data” (Manovich 226). We can think of each layer of data as a layer of symbolic abstraction. For example, in an art exhibit with four walls a floor and a ceiling, one layer of abstraction may be the paintings on the wall; another layer may be the construction of the room itself; still another may be the barriers, walkways, plaques, and other informational behavior modifying objects in a room; yet another abstraction are the people in the room – be they employees or other patrons. To add digital layers of information is to incorporate dynamic, computational information into the space.
To consider this is to realize that computation not only affects our cognitive faculties, but the ontology of our environment as well. The information layers of a given space with digital augmentation allows for a layer of dynamic information, of variability (Manovich 234). This is important, because our spaces and our environment often contains information that is not readily extractable, but rather it exists in a network of knowledge. Without some familiarity of the network within which a given space or artifact exists, the knowledge within the space or artifact is inaccessible. By adding an additional layer of information on top of an artifact or space, we are engaging in a project of augmenting reality, and thereby augmenting human cognition.
Central to this notion is that our environment plays an active role in human cognition; given this, we can think of computer mediated human action as a coupled system (Clark 8). Consider the backwards brain bicycle:
While a bicycle is a locomotive technology, not a cognitive technology, what this video shows us is how integral the form of our technologies are to the cognitive process of using it. Just one component of the bicycle is changed in the backwards brain bicycle, and yet the entire cognitive process of operating the bicycle is stunted. Changing the handling does not make this bike any less of a bike, and all of the components that compose a bike are there. Furthermore, it takes little effort to imagine and conceptualize the new task of riding a backwards brain bicycle – you steer left to turn right and vice versa. However, the cognitive process itself is still stunted. That is because the bicycle itself, and its individual component parts are all part of the cognitive process of riding the bicycle.
For some researchers, this principle of cognition begs for cognitive ethnography to play a central role in HCI research (Hollins 181). This is an important point for my project because this idea of augmenting space and human cognition must incorporate the features and properties of specific cognitive processes if we are to design appropriate cognitive technologies for the tasks they are designed for. I can think of no one better to source in a transition to a discussion of interface than Douglas Engelbart, credited with the invention of the Graphical User Interface, the desktop metaphor, and the mouse/pointer. His theoretical idea of augmenting human intellect is as follows: “By ‘augmenting human intellect’ we mean increasing the capability of a man to approach a complex problem situation, to gain comprehension to suit his particular needs, and to derive solutions to problems. Increased capability in this respect is taken to mean a mixture of the following: more-rapid comprehension, better comprehension, the possibility of gaining a useful degree of comprehension in a situation that previously was too complex, speedier solutions, better solutions, and the possibility of finding solutions to problems that before seemed insoluble” (Engelbart 95).
The Role of Interface in the New Paradigm
Direct manipulation as a paradigm for HCI was extremely important, as we needed an interface that was universally accessible. However, as computation grows in ubiquity in our everyday meaning making and cognitive tasks, there is a growing need to rethink interface and employ more complex cognitive faculties in interface design. The primary role interface plays in the navigation of information spaces is that the interfaces characterize our relationship with those spaces and systems. Interfaces mediate cognitive symbolic faculties with the symbolic representations in external sign systems. Sign systems themselves, a relationship with any external sign system is dependent on interfaces.
More generally, we can use interface to think about meaning systems not only in terms of technological processes, but other cognitive meaning making processes such as language. Ray Jackendoff uses the concept of interface to revise Chomsky’s claim about syntax’s role in language to say that phonological, syntactic, and semantic structures all interface with each other to produce language and generativity within that language (125). The role of interface as a construct in meaning making is about how it networks components of two distinct and related sign systems.
This is crucial to our understanding of interface, because even when we talk about sign systems, we are using a sign system to interface to that given system: language. In Peircean semiotics, “A Sign is a Representamen of which some Interpretant is a cognition of a mind” (Peirce 291). Furthermore:
Signs are divisible by three trichotomies: first, according as the sign in itself is a mere quality, is an actual existent, or is a general law; secondly, according as the relation of the sign to its Object consists in the sign’s having some character in itself, or in some existential relation to that Object, or in its relation to an Interpretant; thirdly, according as its Interpretant represents it as a sign of possibility, or as a sign of fact, or a sign of reason (291).
Even in talking about the components of our symbolic thinking, reduced to a triadic model of sign components, Peirce generated a lexicon for talking about the sign system of symbolic cognition: a sign system within the greater sign system of language. We use logic and language to navigate between these sign components and reason with them.
Symbolic thinking affords an endless chain of associations made to create larger systems of meaning. The more sign systems are created, and the more they grow in complexity, the more crucial a theoretical understanding of interface, especially on the topic of designing interface for cognitive technologies. If our cognitive technologies automate abstractions at various levels, then our interfaces are responsible for assuring these automations as well as our abilities to interact with them are efficient enough to reduce noise or elements that inhibit meaning making.
While the origins of language and the interfaces between phonology, syntax, and semantics continue to be a mystery, the origins of interfaces are not always a mystery. In the case of our symbolic cognitive faculties and their relationships with language and technologies, and especially their origins, there are still many matters of debate. However, semiotics lends us an important understanding of HCI, namely that “research should focus on the interface — considered as a sense production device — and should analyze the ambiguous game between signification and interpretation played by designers and users. For semioticians this process is not a linear transmission of information (interface–>user) but a cooperative one (designers <–>interface <–> user): both designers and users, mediated by the interface, participate in this contractual game of sense production” (Scolari 5). In other words, to execute actions on a personal computer is largely a communicative process between the user and software designers through the medium of interface. This is crucial because this important facet of our relationship with our technologies through interface is hidden in our direct interactions with interfaces. Researchers need to focus on the specific artifacts of interface to really discuss the relationship between users and designers. The Scolari article focuses on this for research in improving interface design, which is important and while I will discuss it briefly, I think the same research project should be employed for understanding the various consequences and implications of human designed interface, which I will also discuss later.
Interface as a design problem finds its solution in the same place the problem comes from in a semiotic framework – the cognitive affordances and restraints within a given computational event. For example, systems designers of flight systems need to implement recognizable textual and pictorial symbols in their interfaces for users, which are already characterized by the event of flight (Andersen 6). In other words, systems designers must navigate their interface design between intuitive inferences of users’ cognitive faculties and the technical systems’ symbol processing. Designers must utilize the affordances of the event within the restraints of the event – the problem is the restraints; the solution – affordances. And balancing between the affordances and restraints are exactly why interface is so important. Because “interfaces do not only present a content and a set of instructions for interaction: they also offer information about users possible movements and represent the relationships of the communicational exchange” (Scolari 9).
The Importance of Interface Design
Before discussing the features of how interface design should be approached, I first want to handle why this is an important problem by discussing representations of interface. First of all, though we’re used to thinking about interface as the GUI of personal computers, I once again want to invoke the idea that interface is more so a sign system that mediates two different sign systems. This is important, because new types of interfaces are constantly emerging in our technologies, such that utilize “gesture recognition, voice commands, and eye-tracking” which “present themselves as lower level inputs that do not tire out the user, but offer a good cognitive control-to-task fit” (Mentzelopoulos 65). This is to say that our interface affordances are increasing, such that utilize different cognitive processes and physiological processes that fit different technological tasks better than the traditional graphical WIMP interface.
Consider Xbox’s gesture recognition and voice command interface, Kinect.
The camera and microphone recognize gestures and voice commands for navigating the Xbox graphical interface, selecting/executing Xbox games/applications, and for controlling gameplay. In terms of an entertainment atmosphere, this ability to use hand gestures and voice commands increase the cognitive affordances possible than are possible with just a game controller, and thereby adds layers to the information space of the room the Xbox is in. Returning to Manovich’s point of augmented space, the camera affords a translation of hand gestures to a digital layer of data. In other words, the human body itself becomes a layer of digital information and part of the Xbox interface in the form of gestures recognizable by both Kinect and the user.
Important to note from the advertisement is the representation of this physical interface. The setting is the home; the whole family participates; children are smiling as they operate virtual steering wheels; and the tone of the whole advertisement fits the image of the product. The whole idea is to have fun with a new interface for an entertainment system.
What if the same interface action is applied to a different setting and tone altogether? Consider the following clips: one from the Xbox advertisement, and one from an episode of the science fiction thriller series Black Mirror.
The interface affordance of a handwave to scroll through content horizontally is the same in both clips. However, note the differences between colors and facial expressions in the two clips. In the Xbox advertisement, a warm tone is created for the effect of marketing an image of family entertainment and wholesome fun. In the clip from Black Mirror, darker colors are used and a somber facial expression are used to generate a somber tone. This is because the scene is set in a workspace. Bingham, the character in the scene, is on an exercise bike, and as he pedals, he earns credits which he can use to purchase more content features on the entertainment system – in the world of this episode, riding the exercise bike is a form of labor. In this sense, the exercise bike interfaces to a technically mediated economic system, which interfaces to the entertainment system he scrolls through. Thematically, this is in line with Neo-Marxist frameworks of thinking about technologically mediated capital systems. Bingham feels oppressed by this system, and this is understood by his obsession with beauty and authenticity and his expressions of frustration with the popular American Idol-esque show within the show that turns his romantic interest into an adult film actress.
All this is to say that the interface characterizes our relationship with the sign systems it mediates. In the Kinect commercial, the hand wave gesture is characterized by a warm relationship with the entertainment the user is scrolling through. In the Black Mirror clip, the same gesture is characterized by feelings of hostility toward the industry of entertainment and oppression by the economic system. In one, the hand gesture is a mechanism of play and choice; in the other, one of indoctrination. Interface plays a crucial role in our relationships with our sign systems, and what the above clips show us is that interface is a dynamic sign system itself, characterized by the status of the sign systems it mediates, as well as the users relationship with those sign systems. Therefore, we have to be careful about how we design interface, think about interface design, especially as information spaces and computationally mediated systems grow more ubiquitous.
A New Theoretical Framework for Interface
With an eye toward augmented space and information spaces, with a new paradigm in computing affordances must come a rethinking of interface. The desktop metaphor and GUI have perhaps become inadequate for computationally mediated spaces that utilize more of our own cognitive affordances. To return back to Mentzelopoulos’s point of “control-to-task fit” in interface, it becomes more and more important to offload the right cognitive tasks to the right type of interface, be it through direct manipulation in a traditional GUI, voice command or gesture in a perceptual interface, or remediation of space in an augmented reality interface. We need a synthesis of our computational affordances to redesign the way we think of computer interface.
Namely, I am calling for a paradigm shift from the way we think about interface in our current personal computer environments. While culture-centered design is not quite what I’m calling for, the abandonment of the desktop interface metaphor is something in line with culture-centered design researchers: “…the desktop, which in theory should empower users to customise and personalise, according to their cultural context as manufacturers promise in their marketing slogans, has been restricted by existing operating systems, which only give the user a certain level of autonomy, such as freely chosen multiple languages, character sets and national formats” (Shen et al. 822). The desktop interface and metaphor are inadequate for a computer system as ubiquitous as the modern OS. At risk is cultural variance between textual formats, color schemes, object layouts,
The problem with our current desktop environment is that it instantiates the desktop metaphor as a principle of computing, whereas the desktop metaphor and WIMP interface are not principles of computing, but rather they are one of the many possible ways computing can be expressed in a GUI computer system. The universality of the desktop metaphor comes from the role interface plays: “a user-interface metaphor is a device for explaining some system functionality or structure (the tenor) by asserting its similarity to another concept or thing already familiar to the user (the vehicle)” (Barr et al. 191). Designers need to choose an interface metaphor that is recognizable to consumers, and perhaps the most obvious choice for a project of augmenting human intellect starts with an augmentation of our work environments. However, as ethnography has been suggested as part of interface design research above, we are entering an era where different designs can be utilized for various cultural and computing environments. This opens up freedom for variance in designs and metaphors, across a range of computing events and cognitive processes. While there is certainly diversity in software design and interface, there needs to be whole new sets of metaphors for computing systems. In the direct words of CCD researchers:
“There seems to be a gap between notions of technology and culture, and a lack of appropriate and valid approaches to their synchronisation. More positively, researchers have been encouraged recently to establish more empirical and practised-based studies within the field of culture and usability. It is likely that a deeper understanding of culture, human cognition and perception followed by the evolution of technology, may help to bridge the gap” (Shen et al. 826).
Interface metaphors are also closely linked with the tasks they are designed for. For example, in a Swedish office setting, the word “kort” (which translates to card) is used to refer to electronic cards the employees use to organize information in an online file system. The same word is used to refer to the paper cards in their physical filing systems. This is because work language itself evolves with the tasks of the work environment (Andersen 25). So too should our interface designs and metaphors.
It doesn’t take much past looking to the semiotic properties of interface.
(Barr 200). Considering the computer icon as the representamen, the conceptual component of the sign process is the potential for the action of printing. The Interpretant is that clicking the icon leads to a printed document. However this idea of printing a document that’s on the computer already utilizes a metaphor that the file is similar to a document. Consider this graphic:
(Barr 202). The metaphorical entailments are the cognitive associations between the metaphor itself which functions as a representamen and the affordances the metaphor offer. In the communication between designer and user, an essential semiotic component of interface are the purposeful associations made by use of metaphor. So if we call a text file on a computer interface a document, we make all the associations that come with the metaphor. The researchers refer to this set of associations as the UI metaphorical entailments (Barr et al. 207).
The entire discussion of interface is crucial to how we think about our cognitive technologies in a process of unlimited semiosis (Barr et al. 201). As our interfaces network us to our cognitive technologies, it is important to design our interfaces in a way that properly represents their role in the continuum of our symbolic thinking. As our cognitive technologies, by their artefactualization of sign processes, exist as instantiations of sign systems, we need to be careful about how we design these technologies. Left alone, the risk of improper metaphorical associations as well as an erasure of computational diversity are too high. Interface plays too crucial of a role to allow a single interface metaphor to be the basis of how computation is culturally constructed. If computation is a desktop in our computers, wouldn’t that imply the computational logic we employ in our own minds have the same associations? We are far more than how our workspaces define us, and therefore we need a more engaging and diverse series of interface metaphors and designs.
Andersen, Peter B. “Computer Semiotics.” Scandinavian Journal of Information Systems, vol. 4, no. 1, 1992, pp. 3-30.
Barr, Pippin, Robert Biddle, and James Noble. “A Semiotic Model of User-Interface Metaphor.” Virtual, Distributed, and Flexible Organisation: Studies in Organisational Semiotics, edited by Liu Kecheng, Kluwer Academic Publishers, pp. 189-215.
Barrett, John C. “The Archaeology of Mind: It’s Not What You Think.” Cambridge Archaeological Journal, vol. 23, no. 1, Feb 2013, pp 1-17.
Benyon, David. “The New HCI? Navigation of Information Spaces.” Knowledge-Based Systems, vol. 14, no. 8, 2001, pp. 425-430.
Clark, Andy and David Chalmers. “The Extended Mind.” Analysis 58, no. 1 (January 1, 1998): 7–19.
de Sousa, Clarisse S. “Semiotic Approaches to User Interface Design.” Knowledge-Based Systems, vol. 14, no. 8, 2001, pp. 415-418.
Destinws2. “The Backwards Brain Bicycle – Smarter Every Day 133.”YouTube. YouTube, 24 Apr. 2015. Web. 19 Oct. 2016.
“Fifteen Million Merits.” Black Mirror, season 1, episode 2, Channel 4, 11 Feb. 2013. Netflix, https://www.netflix.com/watch/70264858?trackId=13752289&tctx=0%2C1%2C7de57de5-13bb-47dc-8d56-862526c8977b-132115175.
Gearlive. “E3 2009: Project Natal Xbox 360 Announcement.” YouTube. YouTube, 02 June 2009.
Hollan, James, Edwin Hutchins, and David Kirsh. “Distributed Cognition: Toward a New Foundation for Human-computer Interaction Research.” ACM Transactions, Computer-Human Interaction 7, no. 2 (June 2000): 174-196.
Irvine, Martin. “The Grammar of Meaning Making: Signs, Symbolic Cognition, and Semiotics.”
Mentzelopoulos, Markos, Jeffrey Ferguson, and Aristidis Protopsaltis. “Perceptual User Interface Framework For Immersive Information Retrieval Environments.” International Journal Of Interactive Mobile Technologies 10.2 (2016): 64-71.
Peirce, Charles S. From “Semiotics, Symbolic Cognition, and Technology: A Reader of Key Texts,” collected and edited by Martin Irvine.
—. Peirce’s Lecture on Triadic Relations and Classes of Signs, Lowell Institute, 1903.
Ray Jackendoff, Foundations of Language: Brain, Meaning, Grammar, Evolution. New York, NY: Oxford University Press, USA, 2003
Jeannette Wing, “Computational Thinking.” Communications of the ACM 49, no. 3 (March 2006): 33–35.
Manovich, Lev. “The Poetics of Augmented Space.” Visual Communication, vol. 5, no. 2, 2006, pp 219-240.
Renfrew, Colin. “Mind and Matter: Cognitive Archaeology and External Symbolic Storage.” In Cognition and Material Culture: The Archaeology of Symbolic Storage, edited by Colin Renfrew, 1-6. Cambridge, UK: McDonald Institute for Archaeological Research, 1999.
Scolari, Carlos. “The Sense of the Interface: Applying Semiotics to HCI Research.” Semiotica, vol. 177, no. 1, 2009, pp. 1-27.
Shen, Siu-Tsen, Martin Woolley, and Stephen Prior. “Towards Culture-Centred Design.” Interacting with Computers, vol. 18, no. 4, 2006, pp. 820-852.