Author Archives: Ojas Patel

Towards a Revised Theoretical Framework for Interface Design – Ojas Patel


There have been several attempts to theorize computation and cognitive technologies as we approach an age of ubiquitous computing. While there is a large body of literature that focuses on how to maximize the efficiency of interface design as we approach such an era, there is a lack of theoretical discourse. As interface is the mechanism by which we engage with computers, a revision to theoretical frameworks of interface is necessary in an age of ubiquitous computing. Semiotics lends much in the way of theorizing generative meaning making as well as recursive symbol processing. Applying the study of semiotics to a discussion of interface leads to a move away from the desktop metaphor and the prominence of the GUI interface, and towards more integrated, multi-faculty interfaces that map to distinct layers of information in space. Furthermore, it lends to a process of unlimited semiosis in interface design and diversity in interface metaphors.


Semiotics has offered much in the way of how we theorize about our cognitive technologies. From symbolic logic, to cultural institutions, semiotics has provided conceptual frameworks for analysis as well as for conducting research, such as communication in computer mediated environments; the mapping of signal processing in machines and symbol processing in humans; and the role of Semiotic Engineering in issues revolving around HCI (de Souza 415-418). In what follows, I will use semiotic approaches to interface design to contribute to a discussion of interface in a paradigm shift of computation from a single user with a single computer system to models of computation that take growing ubiquity into account.

Cognitive Technologies as Artefactualization of Conceptual Sign Systems

The triadic model in Peircean semiotics breaks down sign processes as such: “A sign, or representamen [the material-perceptible component], is something which stands to somebody for something in some respect or capacity. It addresses somebody, that is, creates in the mind of that person an equivalent sign, or perhaps a more developed sign. That sign which it creates I call the Interpretant of the first sign” (Peirce 16). We can think of the three components of sign processes (Representamen, Object, and Interpretant) as such: sense data, concept, and percept. Each component of a sign process is a sign system in itself, in much the same way Ray Jackendoff treats the different components of spoken language, to be discussed later. As much semiotic literature has found, the fundamental components of human cognition break down to this system of sign processes.

Of interest to a discussion of interface in digital technologies is the instantiation of sign systems through sign processes. To interpret a sign process is to reinforce the sign systems that form the foundation of that sign process. A single painting reinforces the medium (or sign system) of painting, which reinforces visual art, which reinforces artistic expression, which reinforces and preserves human culture. All our cognitive technologies and symbolic expressions exist in an ongoing continuum: a historically constructed network of artifacts, ideas, languages, cultures, media, technologies – anything constructed by humans (Irvine 43). This idea is critical in a discussion of interface in cognitive technologies, because the interface mediates the sign systems of our cognitive faculties with system architecture sign systems, to be elaborated on further later.

Most importantly, cognitive technologies are artefactual instantiations of sign processes. Unlike spoken language, computer mediated sign processes are tokenized instantiations of meaning making. This ties in with the concept of “vital materiality,” or the concept that our artifacts are not a product of our symbolic meaning systems but rather mediate our relationship with the world and are indicative of the human species’ shared experiences (Barrett 10). In other words, artifacts are codified with meaningful and interpretable human data, and furthermore are a necessary component in meaning making in the continuum of symbolic cognition. Cognitive technologies afford artefactualization of conceptual sign systems, and thereby an artefactual network of meaning systems. The more this concept is realized in computational technologies, the more pervasive our artefactual representations of sign systems; the more we computationally remediate sign processes and sign systems, the closer we get to a major paradigm shift in the role of cognitive technologies.

The New Paradigm

One perspective on dealing with this new paradigm in HCI is a transition from thinking about direct manipulation and object-oriented computing to navigation of information spaces (Benyon 426). Thinking about computing as navigation of information spaces frees up the study of HCI from a single user and single system to a larger system of information spaces. In the words of Benyon, “As computing devices become increasingly pervasive, adaptive, embedded in other systems and able to communicate autonomously, the human moves from outside to inside an information space” (426). What this refers to is how computation is mediating more of our sign processes, from the workplace, to museums, to politics, to entertainment, and far beyond. Furthermore, the information in each individual system is unique to that system, which is how we move between various information systems. While the topic of ubiquitous computing is a bit out of the scope of this paper, it will point its eye toward the idea of ubiquitous computing as a motivator for this paradigm shift in HCI.

This idea of technical systems as information spaces ties directly with the notion that we offload symbolic information into artifacts that become part of our cognitive processes, in a process known as “external symbolic storage,” (Renfrew 4). The idea is that our artifacts are part of the cognitive process of preserving culture and information, both in terms of writing as well as architecture, art, pottery, etc. This idea is applicable to the idea of technical systems as information spaces as technical systems are cognitive artifacts. As Peter Bogh Andersen writes, computer systems are layers of sign systems: “If we continue this descent through the different layers of the system, passing through the operating system and the assembly code, down to the actual machine code, we will encounter signs most of the way” (Andersen 6). Even down to the machine code, we can treat sequences of electrical signals as a sign system. Because of this, when we encounter an information system, we are effectively navigating into an information space and its network of information spaces. This is true whether we are navigating the World Wide Web from a personal computer, accessing client information through an employer’s intranet, or ordering a sandwich at a deli kiosk, just as much as the spaces themselves flood us with information about whether it’s a home environment, professional environment, or commercial environment through architecture, décor, and social contracts within those spaces.

The globality of sign systems, especially as it permeates technical computer systems, is indicative of how we offload and automate those sign systems. In the words of Jeannette Wing, “Computational thinking is using abstraction and decomposition when attacking a large complex task or designing a large complex system” (Wing 33). If we think of our artifacts, including technical systems, as organizing complex computational tasks in the everyday generation and preservation of culture, the emergence of computer systems playing central roles in human culture is an extremely confluent event (not to fall into any mystic determinism about emerging technologies). And it is no mistake that computers are layered with sign systems much like our own cognition – computation is a type of human logic. We use computer systems to more efficiently compute, store, and organize the processes and products of this logic.

Further speculation on this perspective of HCI leads to the realm of augmented space or reality. In a ubiquitous computing environment, augmented space refers to the dynamic information spaces layered over physical spaces – more a cultural and aesthetic practice than technical (Manovich 220). This refers to more than just the virtual data in a given space, but the layers of abstraction in a given space. As Manovich writes, “If previously we thought of an architect, a fresco painter, or a display designer working to combine architecture and images, or architecture and text, or to incorporate different symbolic systems into one spatial construction, we can now say that all of them were working on the problem of augmented space – the problem, that is, of how to overlay physical space with layers of data” (Manovich 226). We can think of each layer of data as a layer of symbolic abstraction. For example, in an art exhibit with four walls a floor and a ceiling, one layer of abstraction may be the paintings on the wall; another layer may be the construction of the room itself; still another may be the barriers, walkways, plaques, and other informational behavior modifying objects in a room; yet another abstraction are the people in the room – be they employees or other patrons. To add digital layers of information is to incorporate dynamic, computational information into the space.

To consider this is to realize that computation not only affects our cognitive faculties, but the ontology of our environment as well. The information layers of a given space with digital augmentation allows for a layer of dynamic information, of variability (Manovich 234). This is important, because our spaces and our environment often contains information that is not readily extractable, but rather it exists in a network of knowledge. Without some familiarity of the network within which a given space or artifact exists, the knowledge within the space or artifact is inaccessible. By adding an additional layer of information on top of an artifact or space, we are engaging in a project of augmenting reality, and thereby augmenting human cognition.

Central to this notion is that our environment plays an active role in human cognition; given this, we can think of computer mediated human action as a coupled system (Clark 8). Consider the backwards brain bicycle:

While a bicycle is a locomotive technology, not a cognitive technology, what this video shows us is how integral the form of our technologies are to the cognitive process of using it. Just one component of the bicycle is changed in the backwards brain bicycle, and yet the entire cognitive process of operating the bicycle is stunted. Changing the handling does not make this bike any less of a bike, and all of the components that compose a bike are there. Furthermore, it takes little effort to imagine and conceptualize the new task of riding a backwards brain bicycle – you steer left to turn right and vice versa. However, the cognitive process itself is still stunted. That is because the bicycle itself, and its individual component parts are all part of the cognitive process of riding the bicycle.

For some researchers, this principle of cognition begs for cognitive ethnography to play a central role in HCI research (Hollins 181). This is an important point for my project because this idea of augmenting space and human cognition must incorporate the features and properties of specific cognitive processes if we are to design appropriate cognitive technologies for the tasks they are designed for. I can think of no one better to source in a transition to a discussion of interface than Douglas Engelbart, credited with the invention of the Graphical User Interface, the desktop metaphor, and the mouse/pointer. His theoretical idea of augmenting human intellect is as follows: “By ‘augmenting human intellect’ we mean increasing the capability of a man to approach a complex problem situation, to gain comprehension to suit his particular needs, and to derive solutions to problems. Increased capability in this respect is taken to mean a mixture of the following: more-rapid comprehension, better comprehension, the possibility of gaining a useful degree of comprehension in a situation that previously was too complex, speedier solutions, better solutions, and the possibility of finding solutions to problems that before seemed insoluble” (Engelbart 95).

The Role of Interface in the New Paradigm

Direct manipulation as a paradigm for HCI was extremely important, as we needed an interface that was universally accessible. However, as computation grows in ubiquity in our everyday meaning making and cognitive tasks, there is a growing need to rethink interface and employ more complex cognitive faculties in interface design. The primary role interface plays in the navigation of information spaces is that the interfaces characterize our relationship with those spaces and systems. Interfaces mediate cognitive symbolic faculties with the symbolic representations in external sign systems. Sign systems themselves, a relationship with any external sign system is dependent on interfaces.

More generally, we can use interface to think about meaning systems not only in terms of technological processes, but other cognitive meaning making processes such as language. Ray Jackendoff uses the concept of interface to revise Chomsky’s claim about syntax’s role in language to say that phonological, syntactic, and semantic structures all interface with each other to produce language and generativity within that language (125). The role of interface as a construct in meaning making is about how it networks components of two distinct and related sign systems.

This is crucial to our understanding of interface, because even when we talk about sign systems, we are using a sign system to interface to that given system: language. In Peircean semiotics, “A Sign is a Representamen of which some Interpretant is a cognition of a mind” (Peirce 291). Furthermore:

Signs are divisible by three trichotomies: first, according as the sign in itself is a mere quality, is an actual existent, or is a general law; secondly, according as the relation of the sign to its Object consists in the sign’s having some character in itself, or in some existential relation to that Object, or in its relation to an Interpretant; thirdly, according as its Interpretant represents it as a sign of possibility, or as a sign of fact, or a sign of reason (291).

Even in talking about the components of our symbolic thinking, reduced to a triadic model of sign components, Peirce generated a lexicon for talking about the sign system of symbolic cognition: a sign system within the greater sign system of language. We use logic and language to navigate between these sign components and reason with them.

Symbolic thinking affords an endless chain of associations made to create larger systems of meaning. The more sign systems are created, and the more they grow in complexity, the more crucial a theoretical understanding of interface, especially on the topic of designing interface for cognitive technologies. If our cognitive technologies automate abstractions at various levels, then our interfaces are responsible for assuring these automations as well as our abilities to interact with them are efficient enough to reduce noise or elements that inhibit meaning making.

While the origins of language and the interfaces between phonology, syntax, and semantics continue to be a mystery, the origins of interfaces are not always a mystery. In the case of our symbolic cognitive faculties and their relationships with language and technologies, and especially their origins, there are still many matters of debate. However, semiotics lends us an important understanding of HCI, namely that “research should focus on the interface — considered as a sense production device — and should analyze the ambiguous game between signification and interpretation played by designers and users. For semioticians this process is not a linear transmission of information (interface–>user) but a cooperative one (designers <–>interface <–> user): both designers and users, mediated by the interface, participate in this contractual game of sense production” (Scolari 5). In other words, to execute actions on a personal computer is largely a communicative process between the user and software designers through the medium of interface. This is crucial because this important facet of our relationship with our technologies through interface is hidden in our direct interactions with interfaces. Researchers need to focus on the specific artifacts of interface to really discuss the relationship between users and designers. The Scolari article focuses on this for research in improving interface design, which is important and while I will discuss it briefly, I think the same research project should be employed for understanding the various consequences and implications of human designed interface, which I will also discuss later.

Interface as a design problem finds its solution in the same place the problem comes from in a semiotic framework – the cognitive affordances and restraints within a given computational event. For example, systems designers of flight systems need to implement recognizable textual and pictorial symbols in their interfaces for users, which are already characterized by the event of flight (Andersen 6). In other words, systems designers must navigate their interface design between intuitive inferences of users’ cognitive faculties and the technical systems’ symbol processing. Designers must utilize the affordances of the event within the restraints of the event – the problem is the restraints; the solution – affordances. And balancing between the affordances and restraints are exactly why interface is so important. Because “interfaces do not only present a content and a set of instructions for interaction: they also offer information about users possible movements and represent the relationships of the communicational exchange” (Scolari 9).

The Importance of Interface Design

Before discussing the features of how interface design should be approached, I first want to handle why this is an important problem by discussing representations of interface. First of all, though we’re used to thinking about interface as the GUI of personal computers, I once again want to invoke the idea that interface is more so a sign system that mediates two different sign systems. This is important, because new types of interfaces are constantly emerging in our technologies, such that utilize “gesture recognition, voice commands, and eye-tracking” which “present themselves as lower level inputs that do not tire out the user, but offer a good cognitive control-to-task fit” (Mentzelopoulos 65). This is to say that our interface affordances are increasing, such that utilize different cognitive processes and physiological processes that fit different technological tasks better than the traditional graphical WIMP interface.

Consider Xbox’s gesture recognition and voice command interface, Kinect.

The camera and microphone recognize gestures and voice commands for navigating the Xbox graphical interface, selecting/executing Xbox games/applications, and for controlling gameplay. In terms of an entertainment atmosphere, this ability to use hand gestures and voice commands increase the cognitive affordances possible than are possible with just a game controller, and thereby adds layers to the information space of the room the Xbox is in. Returning to Manovich’s point of augmented space, the camera affords a translation of hand gestures to a digital layer of data. In other words, the human body itself becomes a layer of digital information and part of the Xbox interface in the form of gestures recognizable by both Kinect and the user.

Important to note from the advertisement is the representation of this physical interface. The setting is the home; the whole family participates; children are smiling as they operate virtual steering wheels; and the tone of the whole advertisement fits the image of the product. The whole idea is to have fun with a new interface for an entertainment system.

What if the same interface action is applied to a different setting and tone altogether? Consider the following clips: one from the Xbox advertisement, and one from an episode of the science fiction thriller series Black Mirror.

View post on


View post on

The interface affordance of a handwave to scroll through content horizontally is the same in both clips. However, note the differences between colors and facial expressions in the two clips. In the Xbox advertisement, a warm tone is created for the effect of marketing an image of family entertainment and wholesome fun. In the clip from Black Mirror, darker colors are used and a somber facial expression are used to generate a somber tone. This is because the scene is set in a workspace. Bingham, the character in the scene, is on an exercise bike, and as he pedals, he earns credits which he can use to purchase more content features on the entertainment system – in the world of this episode, riding the exercise bike is a form of labor. In this sense, the exercise bike interfaces to a technically mediated economic system, which interfaces to the entertainment system he scrolls through. Thematically, this is in line with Neo-Marxist frameworks of thinking about technologically mediated capital systems. Bingham feels oppressed by this system, and this is understood by his obsession with beauty and authenticity and his expressions of frustration with the popular American Idol-esque show within the show that turns his romantic interest into an adult film actress.

All this is to say that the interface characterizes our relationship with the sign systems it mediates. In the Kinect commercial, the hand wave gesture is characterized by a warm relationship with the entertainment the user is scrolling through. In the Black Mirror clip, the same gesture is characterized by feelings of hostility toward the industry of entertainment and oppression by the economic system. In one, the hand gesture is a mechanism of play and choice; in the other, one of indoctrination. Interface plays a crucial role in our relationships with our sign systems, and what the above clips show us is that interface is a dynamic sign system itself, characterized by the status of the sign systems it mediates, as well as the users relationship with those sign systems. Therefore, we have to be careful about how we design interface, think about interface design, especially as information spaces and computationally mediated systems grow more ubiquitous.

A New Theoretical Framework for Interface

With an eye toward augmented space and information spaces, with a new paradigm in computing affordances must come a rethinking of interface. The desktop metaphor and GUI have perhaps become inadequate for computationally mediated spaces that utilize more of our own cognitive affordances. To return back to Mentzelopoulos’s point of “control-to-task fit” in interface, it becomes more and more important to offload the right cognitive tasks to the right type of interface, be it through direct manipulation in a traditional GUI, voice command or gesture in a perceptual interface, or remediation of space in an augmented reality interface. We need a synthesis of our computational affordances to redesign the way we think of computer interface.

Namely, I am calling for a paradigm shift from the way we think about interface in our current personal computer environments. While culture-centered design is not quite what I’m calling for, the abandonment of the desktop interface metaphor is something in line with culture-centered design researchers: “…the desktop, which in theory should empower users to customise and personalise, according to their cultural context as manufacturers promise in their marketing slogans, has been restricted by existing operating systems, which only give the user a certain level of autonomy, such as freely chosen multiple languages, character sets and national formats” (Shen et al. 822). The desktop interface and metaphor are inadequate for a computer system as ubiquitous as the modern OS. At risk is cultural variance between textual formats, color schemes, object layouts,

The problem with our current desktop environment is that it instantiates the desktop metaphor as a principle of computing, whereas the desktop metaphor and WIMP interface are not principles of computing, but rather they are one of the many possible ways computing can be expressed in a GUI computer system. The universality of the desktop metaphor comes from the role interface plays: “a user-interface metaphor is a device for explaining some system functionality or structure (the tenor) by asserting its similarity to another concept or thing already familiar to the user (the vehicle)” (Barr et al. 191). Designers need to choose an interface metaphor that is recognizable to consumers, and perhaps the most obvious choice for a project of augmenting human intellect starts with an augmentation of our work environments. However, as ethnography has been suggested as part of interface design research above, we are entering an era where different designs can be utilized for various cultural and computing environments. This opens up freedom for variance in designs and metaphors, across a range of computing events and cognitive processes. While there is certainly diversity in software design and interface, there needs to be whole new sets of metaphors for computing systems. In the direct words of CCD researchers:

“There seems to be a gap between notions of technology and culture, and a lack of appropriate and valid approaches to their synchronisation. More positively, researchers have been encouraged recently to establish more empirical and practised-based studies within the field of culture and usability. It is likely that a deeper understanding of culture, human cognition and perception followed by the evolution of technology, may help to bridge the gap” (Shen et al. 826).

Interface metaphors are also closely linked with the tasks they are designed for. For example, in a Swedish office setting, the word “kort” (which translates to card) is used to refer to electronic cards the employees use to organize information in an online file system. The same word is used to refer to the paper cards in their physical filing systems. This is because work language itself evolves with the tasks of the work environment (Andersen 25). So too should our interface designs and metaphors.

It doesn’t take much past looking to the semiotic properties of interface.

View post on

(Barr 200). Considering the computer icon as the representamen, the conceptual component of the sign process is the potential for the action of printing. The Interpretant is that clicking the icon leads to a printed document. However this idea of printing a document that’s on the computer already utilizes a metaphor that the file is similar to a document. Consider this graphic:

View post on

(Barr 202). The metaphorical entailments are the cognitive associations between the metaphor itself which functions as a representamen and the affordances the metaphor offer. In the communication between designer and user, an essential semiotic component of interface are the purposeful associations made by use of metaphor. So if we call a text file on a computer interface a document, we make all the associations that come with the metaphor. The researchers refer to this set of associations as the UI metaphorical entailments (Barr et al. 207).


The entire discussion of interface is crucial to how we think about our cognitive technologies in a process of unlimited semiosis (Barr et al. 201). As our interfaces network us to our cognitive technologies, it is important to design our interfaces in a way that properly represents their role in the continuum of our symbolic thinking. As our cognitive technologies, by their artefactualization of sign processes, exist as instantiations of sign systems, we need to be careful about how we design these technologies. Left alone, the risk of improper metaphorical associations as well as an erasure of computational diversity are too high. Interface plays too crucial of a role to allow a single interface metaphor to be the basis of how computation is culturally constructed. If computation is a desktop in our computers, wouldn’t that imply the computational logic we employ in our own minds have the same associations? We are far more than how our workspaces define us, and therefore we need a more engaging and diverse series of interface metaphors and designs.

Works Cited

Andersen, Peter B. “Computer Semiotics.” Scandinavian Journal of Information Systems, vol. 4, no. 1, 1992, pp. 3-30.

Barr, Pippin, Robert Biddle, and James Noble. “A Semiotic Model of User-Interface Metaphor.” Virtual, Distributed, and Flexible Organisation: Studies in Organisational Semiotics, edited by Liu Kecheng, Kluwer Academic Publishers, pp. 189-215.

Barrett, John C. “The Archaeology of Mind: It’s Not What You Think.” Cambridge Archaeological Journal, vol. 23, no. 1, Feb 2013, pp 1-17.

Benyon, David. “The New HCI? Navigation of Information Spaces.” Knowledge-Based Systems, vol. 14, no. 8, 2001, pp. 425-430.

Clark, Andy and David Chalmers. “The Extended Mind.” Analysis 58, no. 1 (January 1, 1998): 7–19.

de Sousa, Clarisse S. “Semiotic Approaches to User Interface Design.” Knowledge-Based Systems, vol. 14, no. 8, 2001, pp. 415-418.

Destinws2. “The Backwards Brain Bicycle – Smarter Every Day 133.”YouTube. YouTube, 24 Apr. 2015. Web. 19 Oct. 2016.

“Fifteen Million Merits.” Black Mirror, season 1, episode 2, Channel 4, 11 Feb. 2013. Netflix,

Gearlive. “E3 2009: Project Natal Xbox 360 Announcement.” YouTube. YouTube, 02 June 2009.

Hollan, James, Edwin Hutchins, and David Kirsh. “Distributed Cognition: Toward a New Foundation for Human-computer Interaction Research.” ACM Transactions, Computer-Human Interaction 7, no. 2 (June 2000): 174-196.

Irvine, Martin. “The Grammar of Meaning Making: Signs, Symbolic Cognition, and Semiotics.”

Mentzelopoulos, Markos, Jeffrey Ferguson, and Aristidis Protopsaltis. “Perceptual User Interface Framework For Immersive Information Retrieval Environments.” International Journal Of Interactive Mobile Technologies 10.2 (2016): 64-71.

Peirce, Charles S. From “Semiotics, Symbolic Cognition, and Technology: A Reader of Key Texts,” collected and edited by Martin Irvine.

—. Peirce’s Lecture on Triadic Relations and Classes of Signs, Lowell Institute, 1903.

Ray Jackendoff, Foundations of Language: Brain, Meaning, Grammar, Evolution. New York, NY: Oxford University Press, USA, 2003

Jeannette Wing, “Computational Thinking.” Communications of the ACM 49, no. 3 (March 2006): 33–35.

Manovich, Lev. “The Poetics of Augmented Space.” Visual Communication, vol. 5, no. 2, 2006, pp 219-240.

Renfrew, Colin. “Mind and Matter: Cognitive Archaeology and External Symbolic Storage.” In Cognition and Material Culture: The Archaeology of Symbolic Storage, edited by Colin Renfrew, 1-6. Cambridge, UK: McDonald Institute for Archaeological Research, 1999.

Scolari, Carlos. “The Sense of the Interface: Applying Semiotics to HCI Research.” Semiotica, vol. 177, no. 1, 2009, pp. 1-27.

Shen, Siu-Tsen, Martin Woolley, and Stephen Prior. “Towards Culture-Centred Design.” Interacting with Computers, vol. 18, no. 4, 2006, pp. 820-852.

omg it’s week 12 when did that happen

Oof, where do I even start? Perhaps the most important conceptual leap I made throughout our readings is the demystified understanding of computers. I used to look at the ubiquity of computers and software as an abstraction of culture, a creation of new cultural space, and that the singularity was imminent (I know, I’m the coolest guy there is). But now I realize that computers are utilizing our abstractions to maximize the efficiency of the different layers of abstraction. That these layers of abstractions are interfaces in themselves for how we design computers to mediate them.

I’m now looking at computers as an augmenting human technology. If the singularity ever occurs, it will not be because of advances in our computation technology exclusively, because computation is not the only thing that defines our humanness or makes consciousness possible. Rather, it encodes our inputs into a form that is computable by hardware in which we can do all sorts of fancy stuff.

It’s not surprising how pervasive dystopian narratives of computers are. When we offload so many important processes to machines (banking is a big one), skepticism is bound to arise. This idea highlights one of the biggest problems we face now with the role that computers play in culture – illiteracy with the technology (funny, because I could’ve learned this stuff at any point using a darn computer). The literacy to understand and operate the machines is the missing element in the vision of designers who conceptualized and engineered interactive computing. And furthermore, without the literacy, computers as abstract human-interaction-destroying monsters is a self-fulfilling prophecy. When our software is computing open-ended processes, it kind of seems like it’s alive or thinking. In actuality, it’s just waiting for our inputs (metaphorically speaking… right?).

Perhaps my favorite idea I’ve had is the idea that computing is a process that allows us to translate signs into meta-artifacts. The units of these artifacts are bits, which are themselves symbolic representations of our inputs, which are symbolic in a semiotic sense. So when we have open-source communities or teams collaborating on cloud software, we have the distributed mind manifest. It’s translated into bits and then represented in a human-perceptible way by our software. I think this idea reflects Alan Kay’s idea of symmetric authoring and consuming.

I’m also interested in exploring with the idea of phenomenological illusion in GUI and computer interface. It’s an important question to explore because we have to balance designing intuitive interfaces (as complex a problem that is in itself) with engineering a technologically literate user base. If we offload all computing processes onto our software, computers functionally are monoliths of sci-fi abstraction. Is there any way we can design computers so that interacting with them puts us in a place to reason with the computations themselves?

Fancy Dancy Phenomenological Trickery

The biggest surprises in this week’s readings came from Mahoney for me – how difficult a task it is to talk about the history of computing; how difficult a task to figure out where even to start. I barely have a grasp on how to define computing, let alone to truly contextualize it in human history. But the biggest takeaways for me were two ideas: 1) that computing history is the history of computing market demands and the people who were in charge of meeting these demands, and 2) the idea of designing computing processes as creating “operative representations.” The prior worked out an understanding that the design of computer hardware and software is not a magical black box – it is a constant reimagining of computing capabilities to meet the changing demands of consumers, as well as an ongoing process of human innovation in computing capabilities. The latter, applying the process of semiosis, shows what computing adds to meaning making. It is to add to our meaning making process a “meta layer,” as stated in Dr. Irvine’s introduction. It is to reason with our meanings in a hyper automatized way, and in this way, we can think of advances in computing capabilities as advancements of cognitive capabilities.

I love all the neat stuff we’ve gotten to look at throughout the course of this class. This week, the original design of the Memex and Engelbart’s patent for the computer mouse were especially neat-o. The idea of computing history as a history of the designers and engineers in history who made computers especially got my mind going when looking at Engelbart’s technological realizations of Bush’s post-war computing challenges and visions. Today’s computers carry all of the functionality that Bush’s Memex envisioned, but designed so we can even carry our devices with us. Vannevar would lose his mind if he could fiddle with an iPad.

Looking at the patent for the mouse helped me make the connection between design, interface, and the idea of a human-computer symbiosis. Up until this week, I always thought of the mouse pointer as an object that I was moving across the computer screen. That the laser at the bottom of my mouse and the movement it tracked was actually moving the pointer across the screen. What this course is showing me is that the mouse pointer is not necessarily an object, but an array of pixels, and the manipulation of these pixels gives the effect that I am moving an object across the computer screen as opposed to the messages from the mouse changing the position of the array of pixels that look like a mouse pointer (the shape of the mouse pointer itself being a semiotic sign). This phenomenological perceptual trickery is by design – the computer screen should feel like an extension of space, and the mouse pointer should be a metaphorical limb with which I navigate the digital interface, especially if the intention mirrors Vannevar Bush’s vision of how computing would integrate into human cognition. Another instance of this trickery can be seen from an older computer mouse design, the one with a ball in it. Though the effect could show the mouse pointer traveling across the screen diagonally, this doesn’t change that the pointer can only move along an x or y axis (I’m not sure if this is the case with laser mouses).


Bush, Vannevar. “As We May Think.” Atlantic, July, 1945.

Engelbart, Douglas. “Augmenting Human Intellect: A Conceptual Framework.” The New Media Reader. Ed. Noah Wardrip-Fruin. Ed. Nick Montfort. Cambridge: The MIT Press, 2003. Pp. 93-108.

Irvine, Martin. “Introduction to Affordances and Interfaces: Semiotic Foundations.”

Licklider, J.C.R. “Man-Computer Symbiosis.” The New Media Reader. Ed. Noah Wardrip-Fruin. Ed. Nick Montfort. Cambridge: MIT Press, 2003. pp. 73-82

Mahoney, Michael S. “The Histories of Computing(s).” Interdisciplinary Science Reviews 30, no. 2 (June 2005): 119–35.


Extended Cognition and the Automoton

This week’s reading and python exercises really helped me to see the major distinction between computational thinking and thinking more generally. I had used Python before for manipulating large sets of Twitter feeds in Social Media Analytics, but the readings along with doing the course in Code Academy helped me connect a few dots I hadn’t seen before.

First and foremost was this idea of layers of abstractions (Wing). That computational processes shed layers of any one human abstraction to only the abstractions necessary to carry out a function really helped me de-blackbox this idea of computational thinking. That human abstractions can be reasoned with by separating them into layers of abstractions really tossed me around this week, because this idea differs from the more traditional semiotic process. It seems like in computational thinking, we start with the meaning, and we break it into its components and reason with them that way – a reverse semiosis. The treatment of artifacts was interesting as well. That in computer science, the scientific method observes artifacts are created is so fundamentally different from how we scientifically study other things, especially because computers are so intertwined with our cognition (Dasgupta). That the intervention of a computer’s purpose must be considered in it’s scientific study effectively makes the study of computers the study of humans.

However, the utility of computers is this idea that they automate abstractions (Wing). This was reminiscent of cognitive offloading and the extended mind for me. We use computers to offload complex processing and automate it, making it more efficient. And then, these automated processes become part of our cognitive process. Even though our brains aren’t the ones doing the automating work, computers allow us to behave as if they are. So the question still stands… what is and isn’t computable? What is the range of possibility with an extended cognition that uses this advanced computational thinking? This question is further complicated if we accept that computers are not the origin of computational thinking (Campbell-Kelly).


  • Campbell-Kelly, Martin. “Origin of Computing.” Scientific American 301, no. 3 (September 2009): 62–69.
  • Dasgupta, Subrata. It Began with Babbage: The Genesis of Computer Science. Oxford, UK: Oxford University Press, 2014.
  • TheIHMC. “Jeannette M. Wing – Computational Thinking and Thinking About Computing.” YouTube. YouTube, 30 Oct. 2009. Web. 27 Oct. 2016.

Thinking Out of the Boxcar (Alex + Ojas)

We decided to analyze this week’s readings through the lens of automotive technology. Navigation, commute, and travel are  fundamental processes both for people and society, and the cognitive tasks we employ while operating vehicles serve critical roles.

Anyone who has ridden a bicycle or driven a car can attest to the fact that at some point, the operation of a vehicle becomes second nature. You stop consciously thinking about depressing the pedal or turning the handlebar, and the individual movements and actions, whether accelerating, braking, or turning, become autonomous. To use Clark’s parlance, a person and vehicle become a “coupled system” (Clark and Chalmers 8). You will often hear motorcycle and bicycle riders refer to being in “the zone”, a state in which they and their bike become one. The vehicle becomes an extension of your mind. The cognitive process of moving forward on a bicycle effectively makes the vehicle a part of the cognitive process, which reinforces this process as “extended mind.”
A vehicle’s active externalism is exemplified by this video:

In this video, Destin makes a distinction between knowledge and understanding and applies it to the backward brain bicycle. What happens if we engineer a bicycle’s handlebars to inversely change direction, or if turning right on the handlebars leads to turning left? Conceptually, it’s a simple idea and the ease with which we ride a normal bike might lead us to assume it to be easy to ride a backward brain bicycle. As the video shows, the knowledge of how the bike works doesn’t necessarily lead to the understanding required to operate it. This highlights how the handlebars’ design in a normal bike is important to our own cognitive processes of balance and steering. When the functionality of that is altered, the cognitive process of riding a bike is disrupted.

According to Jiajie Zhang and Vimla Patel, distributed  cognition refers to “cognitive systems whose structures and processes are distributed between internal minds and external environment, across a group of individual minds, and across space and time” (340). We discussed how we can fit their arguments regarding affordances, and the interplay of internal and external representations, to vehicular travel by thinking about it in terms of traffic flows and lane changing. Lanes of traffic are demarcated by lines painted on the road, and it is punishable by law to violate this order. For roads with high traffic, there may be multiple lanes going back and forth. The way vehicles organize themselves in multiple lanes is by speed – keep right except to pass. This results in lane changes while driving to reorganize to maximize the efficiency of this traffic flow system. To safely change lanes, a driver goes through several cognitive processes, demonstrated by the following diagram:

Zhong, et al

The speed of vehicles in front of you, speed limit signs, and the location of vehicles in the next lane are all external factors, which together with the internal representations of law and rushing create the affordances that allow for lane changing. The process is further facilitated by turn signals, which correlates to the distributed meaning of lane changes because of how it may influence drivers around the vehicle changing lanes, whether the driver slows down and allows for the lane change, speeds up to pass the car and make room, or simply recognizes the desire for a lane change.

We want to conclude our post by talking about some of the processes that have been offloaded to technology. One perfect example is the popularity of vehicles that have a GPS installed into the center console. By integrating a GPS into the vehicle, the cognitive task of navigation and memorizing directions is offloaded. The flows of traffic are offloaded to streetlights. We don’t have to get out of our cars and communicate at popular intersections. When a left signal turns green, I trust the signal facing oncoming traffic is red. The heavy yielding traffic during rush hour on the beltway is managed by intermittent red/green light signals. Anti-lock brakes allow for the driver to more efficiently consider space, road conditions, and timing when stopping a vehicle, instead of pumping brakes. And automatic gear shifting frees up our left foot and right hand. Accelerating to the flow of traffic is so easy when we don’t have to worry about manually shifting gears, and it frees up our hand to fiddle with the music volume and sip on coffee.


Clark, Andy and David Chalmers. “The Extended Mind.” Analysis 58, no. 1 (January 1, 1998): 7–19.

Destinws2. “The Backwards Brain Bicycle – Smarter Every Day 133.”YouTube. YouTube, 24 Apr. 2015. Web. 19 Oct. 2016.

Dror, Itiel D. and Stevan Harnad. “Offloading Cognition Onto Cognitive Technology.” In Cognition Distributed: How Cognitive Technology Extends Our Minds, edited by Itiel E. Dror and Stevan Harnad, 1-23. Amsterdam and Philadelphia: John Benjamins Publishing, 2008.

-Zhang, Jiajie and Vimla L. Patel. “Distributed Cognition, Representation, and Affordance.” Pragmatics & Cognition 14, no. 2 (July 2006): 333-341.

Zhong, Yaofeng, Yunyi Zhang and Xiao Zhao.”Keep Right To Keep “Right.” UMAP Journal 35.2/3 (2014): 111-137. OmniFile Full Text Mega (H.W. Wilson). Web. 19 Oct. 2016

Stored Media – A Different Sign in Every Context

So the meaning context of a text message is a desire to communicate without the limitation of space. So already, there’s an immediacy to the meaning context of a text message. When someone encodes a message into their phone to be sent, they are tapping pixels that are recognizing the designated spaces that are being tapped as electronic signals to be represented as typographic characters on the screen. After the message has been encoded into these characters, when the sender sends the message, the entire message that has been represented as electronic signals is sent via networks of radio waves to the recipient’s device, which is always idly working to receive signals encoded specifically for that device (through a phone number). The receipt of this message is often represented by an audible signal (sometimes a popular Beach Boys chorus). The recipient will use the phone’s software to locate the message that has already been decoded by the device into the form of typographic characters – however it won’t necessarily be exactly the same as the encoded message. If it’s a different phone, the font, interface of the software, and colors can be completely different than what the message looked like in the encoded message. However, because the meaning process is independent of the encoding and decoding of the actual signals, this does not necessarily affect the transmission of meaning.

While there is no meaning embedded into the actual electrical signals or radio waves that are carrying meanings, they are still part of the meaning making process. This is what it means to be in the digital age of information. There is information, in the technical sense, travelling around us always along highways for information through radio waves, copper wires, fiber optic cables, and so on. Even take this blog post; the purpose is to respond to the readings and will be used in class, and will be on the website far before I make it to the classroom. And when the website is accessed in class, the text I inputted into this text box will be viewable as characters in a string of posts by everyone else in the class. Thus the encoding of information is complete; the decoding will happen in the classroom, later in the semester when I want to track what I’ve learned and review all my blog posts, even next semester when I want to review what I learned in this course when I re-access the website. And the information saved in the website stored in a server somewhere will travel those information highways to reach my computer, my phone, my iPad and the text and my interpretation of it will function as a sign – a different sign in every context.

Shark Attack – Ojas Patel

via GIPHY (idk why I’m having so much trouble embedding this GIF, sorry y’all, you’ll have to click the link)

The GIF above comes from a famous scene in Jaws when the protagonist, chief of police Martin Brody, witnesses the event he’s anxiously and begrudgingly anticipating: a shark attack (check out this song if you’re into weird, experimental, abrasive hardcore with tenor sax, or if you just wanna hear someone scream “shark attack”). The scene is famous for its thematically befitting use of the dolly zoom, a technique in which the camera simultaneously pulls away from and zooms in on the subject. The technique creates the effect of the background moving farther away and more of the peripheral scenery coming into view, while the subject’s distance stays the same, or as in this case, pulls closer. A technique first popularized in Hitchcock’s Vertigo, this distortion in perspective has a disorienting effect. In this scene, it intensifies the panic spreading through the tourists and Amity Island townies witnessing the event. I’m going to attempt a reading of this scene through the lens of semiotics and parallel architecture.

Applying Peirce’s triadic model of semiosis to film, we can think of the sensory data, the audio and video, as representamen. This includes the score, the image of the beach, and people in the foreground, etc. Brody’s physical features, and all the representamen are representative of the object, or the ideas the video and audio are supposed to represent. The scene’s role as a narrative device and what it adds to the tone and catharsis of tension can be considered interpretants. All of these components together give the shot interpretable meaning that functions like a sign. Furthermore, we can distinguish the different types of signs. In the symbolic realm, the distortion of space through time is symbolic of the scene’s tone. Ellen’s hands and how they’re placed on Brody’s shoulders is indexical of affection. Finally, the image of the beach is iconic of a beach. And let’s not forget, our interpretations of these signs are signs in themselves. Turtles, turtles, turtles, it’s just turtles all the way down.

I’ll try to discuss the shot in terms of parallel architecture. The phonological components of film would be the raw sensory data – the audio and video. The screaming of the beach patrons, the colors and shapes, and the film score all function as minute, sensory components of meaning. The semantic components would be the themes, logical structure, emotional response, etc. The syntactical components would be the sequences of images/audio/scenes. In the shot above, the syntax of frames maps us from the first image where the background is close up to an image in which it is much farther. However, the syntax of frames, as in parallel architecture, is not solely responsible for generating the meaning of the shot; the syntax is its own generative process that interfaces with the phonological and semantic structures of the film.


Chandler, Daniel. Semiotics: The Basics. Routledge, 2002.

Jackendoff, Ray. Foundations of Language: Brain, Meaning, Grammar, Evolution. Oxford University Press, 2002.

Mary had a Little Lamb with Mint Sauce

IS OS ALPHA SYMBOLIC OR LINGUISTIC?? The problem of deducing whether language precedes symbolic thought or vice versa is an important one because which one is true draws an important distinction in studying Universal Grammar. If language leads to symbolic thought/cognition, we can say all symbolic representation is a language and we can identify all the rules of those languages, much like we have with language in the study of linguistics. In essence, symbolic thought is the realization of language, and the features of Universal Grammar in language, as well as other linguistic principles, can be applied to symbolic thought. However, if the opposite is true, if symbolic cognition is what leads to the capacity for language, then language is just one extension of symbolic thought. The Rosetta Stone for Universal Grammar and language is in discovering the Universal Grammar and principles of symbolic thought. What do other manifestations of symbolic cognition look like? What would our language be like if we did not develop the physiological adaptations for verbal language, or even physical/visual patterns of language? Is there something other than language that symbolic cognition can produce?

These questions grow in complexity with an interesting distinction between two word classes that can be extracted from the Radford reading. Content categories represent ideas, objects, actions – things with a direct correlation with signs. Functional categories are a way of shaping those signs – a meta-analysis of what is possible in the content categories. If language precedes symbolic cognition, then these features are a clue to discovering the abstraction of stored knowledge, a “gist” of information we collect. However, if symbolic cognition precedes language, these features are inferences of the abstract ideas stored in our brains. The answer, as with Deacon’s argument of co-evolutionary adaptations, could be somewhere in the middle.

From Stone Tools to the Cloud

Barrett’s article offers a hypothesis for the origin of human language that fills some of the gaps in Deacon’s text. Deacon circles around an evolutionary event in the human timeline when we developed the capacity for symbolic representation, as opposed to thought mediated by indexical or iconic representations. This involves a correlative relationship between neurological and physiological adaptations to support the internal ability to represent symbolically and the external ability to gesture and speak. Barrett adds social structure and the idea of the “social brain” to this, arguing that patterns of social organization and meanings created and shared between our ancestors also had an impact on the evolution of the human brain. Given that these shared meanings can precede symbolic thought, combining these two approaches sharpens the image of language origin, as much of a mystery it may be.

It furthermore gives us a new and important reading of the external portion of dual inheritance. Barrett’s mention of “vital materiality,” the concept that our artifacts are not a product of our symbolic meaning systems but rather mediate our relationship with the world and are indicative of the human species shared experiences means our artifacts are codified with meaningful and interpretable data. It is a way of preserving and transferring information in the external world, liken to the way DNA passes on our genetic information. Language and culture are the ultimate manifestations of that, but looking to the origins of language, artifacts and technologies are ways of preserving our meaning systems.

Unlike our genetic coding, we have to manually do the work of transferring our external meaning systems to posterior. As Barrett argues, even the earliest tools can certainly convey meaning because of our relationship with our artifacts. It’s really weird thinking about cloud storage in this capacity. Thinking about Donald’s heavy reliance on rehearsal and refinement as indicators of symbolic capacities, we could perhaps view the history of technology as the evolution of our capabilities in reinforcing our meaning systems. If the complexity of our meaning systems grows with our technological capacity to mediate them, cloud storage certainly achieves a level of reinforcing our meaning systems way beyond oral and written traditions. We are literally digitally storing millennia of collected data and meanings, and our interpretation capabilities drastically improve when they are stored and observable in this way. Exciting times, exciting times.



Barrett, John C. “The Archaeology of Mind: It’s Not What You Think.” Cambridge Archaeological Journal, vol. 23, no. 1, Feb 2013, pp 1-17.

Deacon, Terrence W. The Symbolic Species: The Co-Evolution of Language and the Brain. New York: W. W. Norton & Company, 1997.

Donald, Merlin. “Evolutionary Origins of the Social Brain.” Social Brain Matters: Stances on the Neurobiology of Social Cognition, ed. Oscar Vilarroya, et al. Amsterdam: Rodophi, 2007.

Memes XD lol haha

Peirce’s triadic model of semiosis as a generative process highlights the open-endedness of any instance of human meaning, and this pertains to all three parts of the model. This means that throughout time, any instance of human meaning is going to have a differently shaped representamen, object, or interpretant, even if one or all three of these are relatively the same. Drawing from the second reading, Peirce writes, “The object of representation can be nothing but a representation of which the first representation is the interpretant. But an endless series of representations, each representing the one behind it, may be conceived to have an absolute object at its limit. The meaning of a representation can be nothing but a representation… So there is an infinite regression here. Finally, the interpretant is nothing but another representation to which the torth of truth is handed along; as as representation, it has its interpretant again. Lo, another infinite series.” So from any interpretant, there is an open-endedness to how this refers back to object and representamen, as well as an open-endedness to how an interpretant behaves as an object or representamen itself.

As it pertains to cultural artifacts, this is an important concept to keep in mind because of the user manipulation or interpretation of any object. It’s of particular interest to me in video media and parody. Take for instance the following parody of Eminem’s “Lose Yourself” song and music video:

(a link to the video in case the video doesn’t embed properly: Mom’s Spaghetti)

Other than editing the audio to add the line “mom’s spaghetti” and adding a few clips of spaghetti, the audio and video are the same as the original. The parody draws its humor from subverting the more serious messages of bold individualism and grasping opportunity (over a dramatic riff and beat) by repetition of the line “mom’s spaghetti” which is kind of goofy by comparison. Because the line that is repeated is a feature of the original song, the parody doesn’t so much create a new meaning but magnifies one already embedded in the old one. However it is the open endedness of meaning both to the original content (through user manipulation – like physically, like my mans used audio/video editing software) and interpretation that makes creating a parody, or even envisioning one, possible.

I guess my biggest question with what I’m talking about above pertains to the actual process of manipulating cultural artifacts digitally. I mean, first of all is an audio/video clip an artifact? If it’s edited like the parody above, is it part of the same artifact or is it a completely new artifact?