Author Archives: Becky White

Big Ideas and Small Revolutions: Learning, Meaning, and Interface Design

By Rebecca N. White



Alan Kay helped revolutionize computing, but it was not quite the revolution he wanted. With his Dynabook in the 1970s, he aimed to teach children to program so they could experiment with and learn from personal computers, and to help humans’ thought processes adapt to the digital medium. What is the legacy of this interactive, educational vision?

I seek to answer this question by looking at Kay’s ideas in the context of C. S. Peirce’s meaning-making models and Janet Murray’s work on twenty-first-century interaction design, which is rooted in semiotic principles. Exploring Kay’s vision in this way sheds light on how technological developments interact with natural human meaning-making processes, revealing principles that make good digital design to augment humans’ cognitive processes and that help technologies become societal conventions. For this project, I conducted a textual analysis of primary-source material from Alan Kay Janet Murray in the framework of C. S. Peirce.

While Kay’s educational ideas are evident in many of today’s technologies, a semiotic analysis reveals that Kay was perhaps pushing humans to be too computer-like too quickly. Interactions with computing systems must satisfy human expectations and meaning-making processes.



Using computing technology today is both a social and a personal experience. Video games driven by powerful graphics cards play on large, flat-screen monitors without lag, allowing users to spend a Friday night at home alone navigating complex simulations or connecting with gamers across the globe in massive online worlds. Touchscreen devices that fit in purses and pockets line up icons that are gateways to telephonic capabilities, web browsers, music streaming applications, and more in conventional grid layouts. A range of portable computers with full keyboards weigh under 3 pounds but can still deliver users to more externalized memories than they would ever need in their lifetimes. And with their personal computing innovations in the 1970s, Alan Kay and the Learning Research Group at Xerox PARC helped bring this world into being, providing the inspiration for technical developments for decades to come.

Kay’s[1] widely implemented technical vision for the graphical user interface and more was driven by media, communication, and learning theories. Yet, a simple-sounding, non-technical idea that Kay put forward has not broadly caught on (alankay1 2016). An educational goal was at the center of the vision: to create a “metamedium” that could “involve the learner in a two-way conversation” (Kay and Goldberg). The aim was for users to be able to write their own programs, not just use prepackaged ones, so they could experiment and learn. The proposed device, called the Dynabook, was intended for “children of all ages” (Kay 1972), but Kay focused heavily on the potential for youth to learn by doing (Kay and Goldberg).

From Kay 1972

From Kay 1972

This was a truly bold vision. Kay sought to launch what he calls today the real computer revolution—the one in which humans’ thought processes adapt to the possibilities presented by the digital medium (alankay1 2016).

These ideas are part of a broader process of meaning making and knowledge building that C. S. Peirce has described. And Kay was building on a long legacy of innovations and ideas about augmenting human intelligence with computing systems, from Charles Babbage to Samuel Morse to Claude Shannon to Douglas Engelbart and J. C. R. Licklider.

Interaction designer Janet Murray is operating in the environment that this history made possible. It is an environment in which humans interact with computers, not just use them as tools, and in which digital design is focused on that interaction. It is a space in which people dabble with new frontiers of technology, such as virtual reality. It is an age in which ideas proliferate, and some exceed technical capabilities. Murray strives to add some design structure to this at times chaotic interactive environment, with the aim of giving humans agency in interaction and amplifying their meaning-meaning processes.

To begin exploring these topics and the ways in which Kay’s educational ideas have developed over time, I asked the question: What is the legacy of Alan Kay’s interactive, educational vision for personal computing? I sought to answer this question by looking at Kay’s ideas in the context of C. S. Peirce’s meaning-making models and Janet Murray’s work on twenty-first-century interaction design, which is rooted in semiotic principles. This analysis has implications beyond tracing the Dynabook’s legacy. Exploring Kay’s ideas through semiotic models sheds light on how technological developments interact with natural human meaning-making processes. It reveals general principles that make good digital design to augment humans’ cognitive processes and that help technologies become societal conventions. For this project, I drew on a textual analysis of primary-source material from Alan Kay and Janet Murray conducted with a Peircean framework. To understand Kay’s influences, I also turned to the media theories of Marshall McLuhan and the work of Kay’s colleagues and predecessors, including Engelbart, Licklider, Vannevar Bush, and others, supplemented by my analysis of current technological developments.

Kay and Murray are united around the idea that digital devices should be designed in a way that helps humans build knowledge. Yet, the two diverge in approach. Murray’s focus is on humans and computing systems meeting in the middle—what the computer can do procedurally must match the user’s expectations, not the other way around. Kay too wanted devices and interfaces that matched the way humans make meaning, but he sought to make human thinking more procedural, to quickly adapt thought to the way computers process information. Additionally, Murray’s theories are firmly rooted in society’s communal process of making meaning, while Kay is focused on individual learning, often seeming to overlook the significance of collective processes.

The educational vision Kay put forward remains relevant, and his ideas are evident in many of today’s technologies. However, a semiotic analysis reveals that Kay was perhaps pushing humans to be too computer-like too quickly.


A Vision for Learning

The plans for personal computing developed at Xerox PARC required rethinking hardware and programming to build a computing system that was not just a passive recipient of information but an active participant in a creative process. Much of the technical vision was widely implemented. The graphical user interface and overlapping windows, object-oriented programming, and the use of icons in interfaces that we know today were all born at Xerox PARC (Kay and Goldberg 2003). Yet, Kay’s precise educational vision, which built on Seymour Papert’s and other’s work (Kay 2001), did not catch on as intended (Manovich 2013). Over forty years since it was first introduced, the plan to teach every child how to program and set him or her up with a digital teacher with which to experiment has not been widely adopted by industry or educational systems (alankay1, Greelish). Yet, when these ideas are viewed in broader terms of human meaning making and knowledge building, aspects of Kay’s learning vision are apparent in many areas.

Augmentation and Communication

The idea that the human mind needs to be understood before designing interfaces motivated Kay’s ideas about computers and human interaction with them. He was there at the birth of user-interface and human-centered design, if not its only father. According to this way of thinking, humans are active users not passive consumers. A computer isn’t just a tool, but rather a “metamedium” that combines other media and is an extension of thought (Kay 2001).

Humans have long used tools to extend their abilities and help them navigate the world, as well as for more symbolic purposes (Donald 2007, Renfrew 1999). And these processes were firmly rooted in an external and networked process of making and sharing meaning. Language, writing, and literacy allowed humans to store memories externally and transmit them to future generations, aiding knowledge building and cultural progress (Donald 2007). Humans extend their cognition to these systems, which Clark and Chalmers describe as “a coupling of biological organism and external resources.” Language is one tool that extends cognition in this way (Clark and Chalmers 1998, 18).

Kay operates in this spirit and in many cases has pointed out the importance of language, but he is also firmly situated in the realm of digital thinking—using computing systems that deal in abstract symbols that humans can understand and machines can execute to aid thinking. As he wrote, “language does not seem to be the mistress of thought but rather the handmaiden” (Kay 2001). For Peirce, like Kay, the linguistic and other symbolic systems are just one part of a broader system of logic and meaning-making processes.

With his focus on computing technologies to extend cognition, Kay was also building on the work of Douglas Engelbart, Ivan Sutherland, J. C. R. Licklider, and others (Kay 2004). Licklider termed this “man-computer symbiosis” (Licklider 1990). Sutherland developed the Sketchpad and light pen, which made graphical manipulation possible via an interface, creating a computing device with which humans could begin to be visually creative (Sutherland 2003). Engelbart developed foundational interface ideas and technology, such as the mouse, to change human behavior and augment human intelligence (Engelbart 2003). Often, Kay nods to these and other innovators. In one recent conversation he described his aims in the broader context, “We were mostly thinking of ‘human advancement’ or as Engelbart’s group termed it ‘Human Augmentation’ — this includes education along with lots of other things” (alankay1 2016).

The individual was Kay’s focus. He wanted to build a personal computer with which the user shared a degree of “intimacy.” In his conception, achieving that intimacy required users to be able to both read and write with the computer, to make it truly theirs. He sought to adapt computing devices to the way humans think while also changing the way humans think (Kay 2001).

At the center of Kay’s ideas were principles of communication and meaning making. He has often described a revelation he had when reading Marshall McLuhan’s work on media: the computer itself is a medium. It is a means for communicating information to a receiver that the receiver can then recover and understand. Kay took this further and interpreted McLuhan’s work as saying the receiver must become the medium in order to understand a message—an idea that would drive his conception of human-computer interaction as an intimate connection (Kay 2001). Referring to Claude Shannon who pioneered a theory for conveying information in bits without noise, Kay recently put his general thoughts on process and meaning this way:

The central idea is “meaning”, and “data” has no meaning without “process” (you can’t even distinguish a fly spec from an intentional mark without a process.
One of many perspectives here is to think of “anything” as a “message” and then ask what does it take to “receive the message”?
People are used to doing (a rather flawed version of) this without being self-aware, so they tend to focus on the ostensive “message” rather than the processes needed to “find the actual message and ‘understand’ it”.
Both Shannon and McLuhan in very different both tremendously useful ways were able to home in on what is really important here. (alankay1 2016)

In the same discussion, Kay elaborated on the Shannon ideas: “What is “data” without an interpreter (and when we send “data” somewhere, how can we send it so its meaning is preserved?). . . . Bundling an interpreter for messages doesn’t prevent the message from being submitted for other possible interpretations, but there simply has to be a process that can extract signal from noise” (alankay1 2016). He is tackling this idea of meaning from both a technical—as in, extracting signals from noise—and a human perspective.

Beyond Shannon and McLuhan, this sounds much like Peirce’s triadic conception of the process of meaning making. In this model, a human makes meaning (an interpretant) by correlating an object (a learned concept) and a sign (a material and perceptible representation). Reception is key in this model as well. What are signs without an interpretant? There is no meaning without the human process of recognition and correlation (Irvine 2016a). Peirce also came to call his signs mediums—that is, interfaces to meaning systems, or instances of broader types (Irvine 2016b)—an interesting parallel to Kay’s revelation that the computer is a metamedium. Moreover, Peirce was very focused on process, but in a way slightly different from Kay. The process of meaning making with symbolic systems is dynamic and is always done in a broader societal and communal context. Communicated information can only be understood if the sender and the receiver are drawing from the same conventional understandings (Irvine 2016a).  Kay does not seem to fully account for the communal aspect of these processes.

The Building Blocks of Learning

Working with this internal meaning-making framework, Kay drew heavily on ideas about the nature of children’s thought processes. He wanted a personal device that could match the meaning-making processes of children at their individual developmental levels (Kay 1972). The design of the computer interface needed to be tied to natural learning functions (Kay 2001). As Kay put it recently, “For children especially — and most humans — a main way of thinking and learning and knowing is via stories. On the other hand most worthwhile ideas in science, systems, etc. are not in story form (and shouldn’t be). So most modern learning should be about how to help the learner build parallel and alternate ways of knowing and learning — bootstrapping from what our genetics starts us with” (alankay1 2016). He demonstrated some of these learning techniques using modern technology in a 2007 TED Talk (if the player is not working properly, the video can be viewed on TED’s website; jump to 12:15 for the clip):

Kay’s primary influence when it came to cognitive development was Jerome Bruner, though he was also inspired by other developmental psychologists and Seymour Papert’s educational work with LOGO. Most influential in Kay’s interface efforts were Bruner’s descriptions of children’s stages of development and three mentalities—enactive, iconic, and symbolic. The enactive mentality involves manipulation of tangible objects; iconic or figurative involves making connections; and symbolic involves abstract reasoning. Additionally, Kay recognized that a human’s visual, symbolic, and other systems operate in parallel (Kay 2001).

According to Kay, a computing device needed to be designed to serve and activate all of these areas if it was to encourage learning and creativity, one of his overarching goals for children and adults alike. He sought to combine the concrete and the abstract. Based on these principles, he developed the motto “doing with images makes symbols” (Kay 2001).

Although Peirce’s conception of sign systems did not involve clear stages of development and he was not modeling internal cognitive processes, his ideas roughly correspond to Kay’s. The enactive mentality is about how humans interact with the material signs in their worlds. It is a tactile and action-oriented experience with the outside world that provides input to humans with which they then make meaning. The iconic mentality could map to two of Peirce’s signs: the iconic and the indexical, which represent and point to objects, respectively. The symbolic realm appears to be the same in both conceptions—abstractions and generalizations made from other signs. Kay’s motto, “doing with images makes symbols,” could thus be broadened to “doing with signs makes meaning.”

The computing device Kay envisioned would help users make their own generalizations and abstractions from digital symbols (Kay and Goldberg), which are themselves abstractions. This is the process of meaning making and knowledge building with signs that Peirce describes (Irvine 2016a). And in the Peircean sense, the computers and their parts are signs as well, the material-perceptible parts of humans’ symbolic thought processes (Irvine 2016b).

Central to Peirce’s conception is the dialogic nature of sign systems. That is, the individual process of meaning making is based on conventions shared with other humans (Irvine 2016a). In contrast, Kay focuses on the “interactive nature of the dialogue” between the human and the computing system, another symbolic actor in a sense (Kay and Goldberg). It is almost as if Kay views computers as on the same level as humans in terms of the symbolic dialogue. This thought process emerged clearly in a recent Q&A. A questioner asks specifically about tools to encourage the communal process of knowledge building, and Kay brings the conversation back around to individual human and human-computer processes (in addition to criticizing the interface he has to use to respond to the questioner) (alankay1 2016).

Alan Kay responds to a question in a Hacker News thread

Alan Kay responds to a question in a Hacker News thread

In this case, Kay’s conception is both in line and in conflict with Peirce’s. Particularly in more recent writings after the spread of the internet, it appears that Kay recognizes the communal network of human meaning making and extends it to computers. It is not just about augmenting human intellect and increasing creativity on a personal, internal level. Rather, those interactive processes stretch far beyond the coupling of one human with a personal computer. However, in the Peircean conception, a computer is not the same kind of symbolic actor as a human. Computing systems cannot make meaning. They can convey information with which humans can then make meaning due to their capacity for abstraction and generalization, but they do not make correlations in the same way.

The Dynabook

The technical computing revolution all began with these ideas. Focused on the human-computer dialogue, Kay set out to translate these principles into a personal computing vision. Kay and Goldberg envisioned a metamedium that could simulate and represent other media. The vision took the form of the Dynabook, though many of its components were incorporated into other computing devices as well.

Part of this development process involved conceptualizing the foundational concepts in terms of the affordances of the digital space, but it also entailed shifting the way users approached computing systems. As Kay put it, “The special quality of computers is their ability to rapidly simulate arbitrary descriptions, and the real [computer] revolution won’t happen until children learn to read, write, argue and think in this powerful new way” (Kay 2001). He wanted to alter the way people approach digital technologies, and still sees that as the aim. In a recent Q&A, he warned: “children need to learn how to use the 21st century, or there’s a good chance they will lose the 21st century” (alankay1 2016). Kay at times calls his educational vision the “service” conception of the personal computer.

Regardless of terms applied, the idea was bold and far-reaching. As Kay wrote, “man is much more than a tool builder . . . he is an inventor of universes” (Kay 1972), and he sought use computers to make the most of that potential. The intention was for humans, particularly children, to be able to program the system themselves and learn concepts by experimenting in the digital space. Kay described the idea and experiments with computing technology in detail in “A Personal Computer for Children of All Ages” and “Personal Dynamic Media,” co-written with Adele Goldberg.

From Kay and Goldberg

From Kay and Goldberg

The underlying, meaning-making principles on which Kay drew translated into physical design choices (Greelish 2016). To encourage creativity and based on his understanding of the iconic mentality, Kay envisioned an interface that presented as many resources on the same screen as was feasible. To meet this need, Kay created overlapping windows using a bitmap display presenting graphic representations that users could manipulate with a pointing device—a mouse. The Smalltalk object-oriented programming language was an outgrowth of Kay’s understanding of how people process messages, and it unified the concrete and abstract worlds “in a highly satisfying way” (Kay 2001). The language was intended to be easy to use so even children could create tools for themselves and build whatever they wanted to in the metamedium. Users could personalize a text editor to process linguistic symbols as they saw fit, or create music and drawing programs. The Dynabook itself was intended to be lightweight, portable, able to access digital libraries of collective knowledge, and able to store, retrieve, and manipulate data (Kay 2001, Kay 2004, Kay and Goldberg).


Where These Ideas Took Us

Although many of his technical conceptions are ubiquitous, Kay’s somewhat utopian vision of world in which each child had an individualized computer tutor with which to experiment through programming did not take off. This was at least in part because Kay was bound by the technical capabilities of the day, not to mention the magnitude of the task of shifting bureaucracies and ingrained human processes built up over centuries.

The devices Kay described in his early papers had to first be created before they could be used widely to enact his service vision. This was, after all, a new medium. The seeds of his ideas grew out of existing conventions like editing, filing systems, drawing, and writing (Kay and Goldberg), so they were somewhat familiar to users and activated existing human knowledge. But the possibilities afforded by the digital space were just being probed when Kay was first writing (Kay 1972). Technologies that we today think of as commonplace and some that have not yet come to fruition were being invented back then. For example, Kay hypothesized that the technology could “provide us with a better ‘book,’ one which is active (like the child) rather than passive.”

And Kay always intended for the initial ideas to grow and evolve, summed up in the “finessing style of design” employed at Xerox PARC (Kay 2004). These ideas were not meant as the end-all-be-all. Kay and his cohort imagined that others would not just improve upon them but also produce new innovations.

Yet, Kay hinted that there could be problems with the metamedium conception as well. He and Goldberg doubted that a single device could be pre-programmed to accommodate all user expectations. It was better to allow the user to customize the device as they saw fit. Kay and Goldberg explained: “The total range of possible users is so great that any attempt to specifically anticipate their needs in the design of the Dynabook would end in a disastrous feature-laden hodgepodge which would not be really suitable for anyone” (Kay and Goldberg).

That is, in many ways, what happened. Today, computing devices are frequently used for passive consumption of other forms of media. Users do create with current computing systems, but that creativity is constrained by software that has been programmed by someone else. The ability to program machines remains the purview of those with specialized skills (Manovich 2013).

Photo by Walden Kirsch, licensed under the Creative Commons Attribution-Share Alike 2.0 Generic license

Photo by Walden Kirsch, licensed under the Creative Commons Attribution-Share Alike 2.0 Generic license

Kay, for one, is not satisfied with the way in which computing technology has evolved, and has bemoaned the lack of innovation and the current state of computing. To him, people are mostly tinkering around the edges of existing conventions and not thinking about inventing for the future. Overall, there is not enough emphasis on the services side of his original ideas. Current programming languages remain too abstract and are not user-friendly enough. He would like to see languages that are scalable and easier for humans to use. Kay criticizes tablets and other systems that do not use pointing devices, which are necessary for the enactive mentality. He still thinks simulations are an important part of the computer revolution but is not satisfied with any, although he describes NetLogo as interesting. And he widely criticizes user-interface designs, particularly those from Apple, as not being easy enough for their users to manipulate and personalize (alankay1 2016, Kay 2004, Oshima et al 2006). One recent example comes from a Q&A (alankay 1 2016):


Alan Kay responds to a question in a Hacker News thread

Despite his specific criticisms, many of the general principles Kay put forward, which were based on his conception of thought and learning processes, can be seen today. For instance, today design fields exist that are focused on user interfaces and human-computer interaction, which is a significant change in and of itself (Kay 2001). And interactive computation that can better predict emergent behaviors and better respond to humans’ mental models is an active area of research that could perhaps make it unnecessary to teach children programming in order to achieve Kay’s aims (Wegner 1997, Goldin et al 2006).

Although software is not open to programming by children, many educational tools have been and are being developed that draw on interactive principles pioneered by Kay and that can react to a learner’s needs. Duolingo, a language-learning app built using collective intelligence and that adapts to users’ learning levels, is just one example. Broader initiatives to incorporate computation in early childhood exist as well., backed by Google, Microsoft, Facebook, and others, seeks to make computer science accessible to all children. Active learning practices incorporate many of the principles Kay sought to foster using computing technologies (Center 2016). Jeanette Wing describes computing as the automation of abstractions and seeks to teach children how to think in this way (Wing 2006, Wing 2009). Ian Bogost argues that procedural literacy, based on computing processes, should be applied outside the realm of programming to teach people how to solve problems (Bogost 2005). The One Laptop Per Child initiative, which Kay mentions in the video above, seeks to give children the metamedia with which to experiment.

The list goes on, but perhaps most true to Kay’s vision is MIT Lifelong Kindergarten’s Scratch project. This is no surprise given that the Media Lab of which this project is a part was co-founded by Nicholas Negroponte, who was also influenced by Papert and worked with Kay and Papert on the One Laptop Per Child project (MIT Media Lab). Kay’s Squeak Smalltalk language formed the backbone of Scratch (Chen, Lifelong 2016b), which seeks to help “young people learn to think creatively, reason systematically, and work collaboratively — essential skills for life in the 21st century.” And it allows all users to program, create, and share their creations. Although anyone can use the platform, educators are encouraged to use it as a learning tool, and resources are provided to help teachers on that front (Lifelong 2016a). Thanks to the internet, this project can go directly to educators and students, rather than proponents having to navigate educational systems as would have been necessary in the 1970s.


Designing for the Networked, Metamedia World

Janet Murray is operating in this context, and encouraging others to think more like Kay did in the 1960s and ’70s. She takes the new form of representation Kay helped create, the metamedium, and lays out principles of design to maximize meaning and user (“interactor”) agency or interactivity. To do this, she says, designers should deconstruct projects into components and then rethink them in terms of the affordances provided by the digital space.

Murray draws on a range of different fields and acknowledges deep historical context, generally operating from a semiotic, Peircean perspective. Like Peirce, Murray is thinking abstractly and trying to build out a general model in a sense. Her model is of the digital design process, and she seeks to extract common, general principles that can be applied regardless of project. The model is a component of the broader meaning-making system described by Peirce.

Like Kay, Murray approaches computing systems as media and not tools. Media, she says, “are aimed at complex cultural communication, in contrast to the instrumental view of computational artifacts as tools for accomplishing a task” (Murray, 8). Stemming from this, she discourages use of the word “user” and the phrase “interface design,” as they are too closely related to tools (Murray, 10–11).

How she would prefer to describe these human-computer processes sounds much like Kay’s vision: “an interactor is engaged in a prolonged give and take with the machine which may be useful, exploratory, enlightening, emotionally moving, entertaining, personal, or impersonal” (Murray 2011, 11). This idea is, in a sense, broader than Kay’s, which was intended to be somewhat narrowly focused on children’s learning processes.

But at base, both attempt to tap into broader human processes of making meaning—the process described by Peirce in which a human forms an interpretant from an object and a sign/medium. Human beings, as members of the symbolic species, are unique in that they operate in the realm of abstractions and generalizations. They can provide computing systems with symbols that those systems can then execute—because humans have drawn on their symbolic capacities to build them that way. And humans can make meaning of the symbols that the systems return. Each new abstraction creates new meaning, building knowledge—which is the process of learning in a general, non-psychological sense.

The job of designers, according to Murray, is to use code to design digital artifacts that meet interactors’ needs and expectations, allowing them to form those new correlations—as Kay sought to do with his original designs. This involves using existing conventions in new ways, to signal certain meaning correlations to users (Murray, 16). Conventions allow humans to recognize patterns amid complexity and noise. Those patterns, or schema in the cognitive science sense, are built from experience (Murray, 17). Users must be able to make meaning and connections out of what they have in front of them; in Peirce’s terms, a system should not be so foreign that it prohibits users from extracting features and forming interpretants based on existing knowledge (Irvine 2016b).

The affordances of the digital medium help designers achieve these aims. An interactive system that is successful will create “a satisfying experience of agency” for the user by matching the digital medium’s procedural and participatory affordances—that is, the programmed behaviors of the system and the expectations of the users (Murray, 12). Kay developed one of the types of languages used to encode those behaviors—object-oriented programming.

Kay’s work also laid the groundwork for participatory affordances. Murray’s description of this topic takes those foundations for granted: “Because the computer is a participatory medium, interactors have an expectation that they will be able to manipulate digital artifacts and make things happen in response to their actions” (Murray, 55). That expectation is possible in part because of Kay’s original vision; this is essentially his “doing with images makes symbols.” Kay, however, sought to take this further, and to transfer more agency to users by allowing them to design their own programs to meet their knowledge-building needs.

There are also spatial and encyclopedic affordances of the digital medium. The former is about visual organization, and it builds off of what Kay initially created with the graphical user interface. This graphical organization involves the abstractions made up of bits of information that have come to signal particular meanings to users of computing systems, such as file folders, icons, and menus. Here too, as when Kay was designing the Dynabook, the focus is on meaning making and tapping into human thought processes: “Good graphic design matches visual elements with the meaning they are meant to convey, avoiding distraction and maximizing meaning” (Murray, 76). In Peirce’s terms, the perceptible signs (designs) correspond to objects, and humans correlate the two to make meaning. Murray argues, harking back to Shannon’s information theory, that designs should minimize noise so the interpreters can make maximum meaning.

The encyclopedic affordance, meanwhile, stems from the vast capabilities of computing technology to store information that humans can retrieve and process. This enables cultural progress and collective knowledge building because these technologies can store vast amounts of information for use over time, allowing many humans now and in the future to form interpretants from the same information. Kay thought of this as well in his Dynabook conception, discussing the use of personal computers to access digital instances of books or libraries full of information through the LIBLINK (Kay 1972). In 2015, he even wrote about the challenges of ensuring this wealth of externalized memory can be accessed by future generations (Nguyen and Kay). And he was reared in the culture of the Advanced Research Projects Agency (ARPA), which focused on “interactive computing” in a “networked world” (Kay 2004). Yet, one area Kay does not spend much time commenting on is the dialogic, communal nature of meaning making, remaining focused on the individual experience.

This nature factors centrally into Murray’s thinking. She focuses on meaning making as not just an individual but also a social activity; humans interpret digital media based on both personal and collective experiences. Interaction with digital media, she says, necessarily involves interpretation of artifacts within broader cultural and social systems (Murray, 62). Interactors also use computing technology to access and interact with other people and broader cultural systems (Murray, 11). Drawing on this dialogic nature of the symbolic computing system, Murray calls for using existing media conventions to actively contribute to and develop the collective, or as she puts it, “to expand the scope of human expression” (Murray, 19).

This meshes with Peirce in many ways. According to his semiotic model, meaning is always communal, intersubjective, collective, and dialogic (Irvine 2016a, 2016b). Signs are the ways in which we communicate meanings to others, and those meanings are always made in the context of collective understanding, drawing on existing conventions so others may make their own correlations. Humans can communicate in ways members of their society understand because they can communicate in mutually agreed-upon symbols (Irvine 2016a). Digital technologies offer ways to externalize and share the meanings interactors make from these collective systems.

Still, intersubjectivity does not mean that the same signs lead all humans to make the same correlations. Interpretant formation is necessarily based on context, and each individual interprets a perceptible sign based on their individual experiences and perspectives on conventions, which can lead to the making of various meanings. In this sense, meaning is personal and dynamic. And Murray acknowledges that inventors of digital technologies cannot control the ways in which those artifacts will be interpreted or used:

The invention of a new form of external media augments our capacity for shared attention, potentially increasing knowledge, but also increasing the possibilities for imposing fixed ideas and behavior and for proselytizing for disruptive causes. Media can augment human powers for good or for evil; and they often serve cultural goals that are at cross purposes. (Murray 40)

On this topic, one point of direct comparison between Murray and Kay relates to music. Murray references the pirating of music that took place over the internet starting in the 1990s, which resulted in decreased sales of CDs among other outcomes. In contrast, Kay wrote in 1972 that “most people are not interested in acting as a source or bootlegger; rather, they like to permute and play with what they own” (Kay 1972). Kay expected individual users would want a flexible computing device with which they could make their own meanings, but he underestimated the impact of networking and communal meaning processes. These computational artifacts have the power to alter the way humans behave, for bad and not just good.

Often the deciding factors in this development are out of any individual’s control. Murray puts a fine point on this: “Cultural values and economic imperatives drive the direction of design innovation in ways that we usually take for granted, making some objects the focus of intense design attention while others are ignored altogether” (Murray, 28).

Today, Kay acknowledges this power to a degree. He consistently and fondly remembers his time at Xerox PARC as a somewhat utopian experience of all researchers working together toward a common vision and in the absence of market drivers (Kay 2004). He has struggled to find another place like that. With respect to current artificial intelligence, for instance, he commented that “the market is not demanding something great — and neither are academia or most funders” (alankay1 2016). Still, he persists in trying to control the outcomes and change the way people think.


Small Revolutions

It is clear that Murray and Kay are moving toward similar ends. Both are attempting to create digital technologies that tap into the human processes of making meaning and building knowledge, and to augment those processes. They argue for meeting users where they are—delivering on expectations and helping with the process of extraction and abstraction. Both also recognize that the new digital space provides new affordances, not least the opportunity to give users greater agency over devices, and requires rethinking how information is presented.

When it comes to broad brush strokes, Murray’s general design process sounds much like the process Kay undertook when thinking up the Dynabook. Murray’s basic recipe for digital design includes: “framing and reframing of design questions in terms of the core human needs served by any new artifact, the assembling of a palette of existing media conventions, and the search for ways to more fully serve the core needs that may lie beyond these existing conventions” (Murray, 19).

Similarly, Murray and Kay are both firmly oriented toward the future. The only time Murray mentions Kay directly in her book, in fact, is on this subject. She quotes him saying, “The best way to predict the future is to invent it” (Murray 25). And the title of her book, Inventing the Medium, is essentially what Kay did in the 1970s.

Yet, Murray in some ways has a much broader scope than Kay. This is perhaps a counterintuitive thought given Kay’s truly revolutionary vision. Still, she is working in a much more complicated, networked computing environment than Kay was, and her goal is to fit anything that could be designed in the new digital space under the same set of umbrella principles. Her ideas are firmly rooted in broad, societal processes of meaning making, not just in the individual learning process. And she is exceeded in ambition by C. S. Peirce, who sought to produce a model of all meaning processes.

The implications of this are difficult to discern. But a more holistic view such as that taken by Murray could indeed help designers better meet human needs than one focused on individual goals, even if that approach does not impact the flow of history as spectacularly as Kay did. After all, humans are the symbolic species. Making new meaning is inborn and collective. The power of conventions should not be underestimated. And as Murray writes, the computer is a large and complicated “cultural loom, one that can contain the complex, interconnected patterns of a global society, and one that can help us to see multiple alternate interpretations and perspectives on the same information” (Murray, 21). Designing digital technologies today requires keeping the communal possibilities in mind.

Another difference has to do with agency. Murray frequently stresses the need for digital designs to match human meaning-making processes. Kay also stressed the need for computing technology to operate on the user’s level. Yet, Kay was actually trying to drastically change the way humans processed information as part of that symbiosis. With his vision to teach children programming so they could experiment with computer tutors, he was attempting to start a revolution in which humans made meanings with an entirely new set of abstractions that did not evolve organically from human processes but that was created by a small subset of experts. The new sign system did not emerge from the existing behaviors of a collective society but was rather imposed on broader society by a small culture. In a sense, his device was not meeting children on their playing field but was moving them to an entirely new country.

Murray speaks to this point when discussing situated action theory. She writes, referencing anthropologist Lucy Suchman, “Instead of asking the user to conform more fully to the machine model of a generic process, Suchman argues for an embodied and relational model, in which humans and machines are constantly reconstructing their shared understanding of the task at hand” (Murray, 62).

Society, broadly speaking, may now be approaching a point at which computing technologies can meet humans in this way, closer to their natural processes. That is, humans are becoming more accustomed to this new symbolic system—and its power—and technological developments are allowing computing systems to be more adaptable to human processes. This is not the disruptive, futuristic thinking both Kay and Murray call for, but evolution. Perhaps that is what it takes for long-term, deep changes in human behavior and meaning-making processes to happen.

In this vein, Kay has adapted his educational model to reflect developments, achieved and projected, in artificial agent technology. He, along with co-authors, outlined a new plan for “making a computer tutor for children of all ages” in 2016. The team wants to leverage innovations in artificial intelligence technology to develop an interactive tutor that can observe and respond to students’ behaviors, without the student having to program the device’s activities (Oshima et al 2016).

Although all meaning making is intersubjective according to Peirce, there is also something to be said for the stress Kay puts on the individual experience. Members who share symbolic systems draw from the same conventions, but their experiences are personal, and the interpretants they form are individualized to some extent. Humans operating in the digital space also now expect to make use of its participatory affordances. In colloquial terms, they want control, and to be able to do their own thing.

Many technologies are now more customizable and reactive to individual desires to create and learn, although most have not reached the point Kay wanted to with his service vision. The aforementioned Scratch and NetLogo are examples. Amazon has opened up its Alexa system and others to developers, so users can develop functionality for these devices to serve their own needs, as well as commercial ones. These apps can be shared with other users. Google also allows developers to create add-ons that bring new functionalities to its apps. To amplify individual and collective meaning-making processes, more flexibility is perhaps needed on this front.

Virtual and augmented reality technology, meanwhile, could completely change user interfaces once again. Although the technology is today often just used to play games and have fun, as the interactor below is doing, it could eventually revolutionize the way in which humans interact and make meaning with computing technologies.

When it comes to these technologies, Kay is both the base and the tip of the iceberg in many ways. His and others’ ideas drove and support the development of what we know today to be personal computing, and in formulating that vision, he helped unlock endless possibilities. But as Murray hints and Peirce demonstrates, there is a broader logic at play. Murray tries to tap into those broader meaning-making processes to push digital design forward one step—or giant leap—at a time.

In large part due to the technological access brought about by cheaper and smaller hardware components and the internet, there has, perhaps, not been one computer revolution of the sort Kay outlined but a multitude of smaller revolutions as humans have tried to catch up to the technological advancements. At least the two symbolic processors—the human and the computer—seem to be moving closer together in many ways, even if that change is slower than Kay might like.



[1] In many forums, Kay has given credit to his colleagues at Xerox PARC for their roles in bringing this vision to life. However, for ease of reading, unless he was a co-author on a publication, I have only used Kay’s name throughout this text.



alankay1. 2016. “Alan Kay Has Agreed to Do an AMA Today.” Hacker News. Accessed December 7.

“Amazon Developer Services.” 2016. Accessed December 18.

Bogost, Ian. 2005. “Procedural Literacy: Problem Solving with Programming, Systems, & Play.” Telemedium (Winter/Spring): 32–36.

Bolter, Jay David, and Richard Grusin. 2000. Remediation: Understanding New Media. Cambridge, MA: The MIT Press.

Bush, Vannevar. 2003. “As We May Think.” In The New Media Reader, edited by Noah Wardrip-Fruin and Nick Montfort, 35–47. Cambridge, MA: MIT Press.

Center for New Designs in Learning and Scholarship. 2016. “Active Learning.” Accessed December 18.

Chen, Brian X. 2010. “Apple Rejects Kid-Friendly Programming App.” WIRED. April 20.

Clark, Andy, and David Chalmers. 1998. “The Extended Mind.” Analysis 58, no. 1: 7–19.

“ Anybody Can Learn.” 2016. Accessed December 18.

Deacon, Terrence W. 1998. The Symbolic Species: The Co-evolution of Language and the Brain. New York: W. W. Norton & Company.

“Develop Add-Ons for Google Sheets, Docs, and Forms | Apps Script.” 2016. Google Developers. Accessed December 18.

Donald, Merlin. 2007 “Evolutionary Origins of the Social Brain.” In Social Brain Matters: Stances on the Neurobiology of Social Cognition, edited by Oscar Vilarroya and Francesc Forn i Argimon, 215-222. Amsterdam: Rodophi.

Engelbart, Douglas. 2003. “Augmenting Human Intellect: A Conceptual Framework.” In The New Media Reader, edited by Noah Wardrip-Fruin and Nick Montfort, 93–108. Cambridge, MA: MIT Press. Originally published in Summary Report AFOSR-3223 under Contract AF 49(638)-1024, SRI Project 3578 for Air Force Office of Scientific Research, Menlo Park, CA: Stanford Research Institute, October 1962.

Gleick, James. 2011. The Information: A History, a Theory, a Flood. New York: Pantheon.

Goldin, Dina, Scott A. Smolka, and Peter Wegner, eds. 2006. Interactive Computation: The New Paradigm. New York: Springer.

Greelish, David. 2016. “An Interview with Computing Pioneer Alan Kay.” Time. Accessed December 5.

Irvine, Martin. 2016a. “The Grammar of Meaning Systems: Sign Systems, Symbolic Cognition, and Semiotics.” Unpublished manuscript, accessed December 17. Google Docs file.

———. 2016b. “A Student’s Introduction to Peirce’s Semiotics with Applications to Media and Computation.” Unpublished manuscript, accessed December 17. Google Docs file.

Kay, Alan. 2001. “User Interface: A Personal View.” In Multimedia: From Wagner to Virtual Reality, edited by Randall Packer and Ken Jordan, 121–131. New York: W. W. Norton. Originally published in 1989. Available at

———. 2003. “Background on How Children Learn.” VPRI Research Note RN-2003-002. Available at

———. 2004. “The Power of Context.” Remarks upon being awarded the Charles Stark Draper Prize of the National Academy of Engineering, February 24. Available at

———. 2007. “A Powerful Idea about Ideas.” Filmed March 2007. TED video, 20:37. Accessed December 7, 2016.

Kay, Alan C. 1972. “A Personal Computer for Children of all Ages.” Palo Alto, CA: Xerox Palo Alto Research Center.

———. 1977. “Microelectronics and the Personal Computer.” Scientific American 237, no. 3: 230-44.

Kay, Alan, and Adele Goldberg. 2003. “Personal Dynamic Media.” In The New Media Reader, edited by Noah Wardrip-Fruin and Nick Montfort, 393–404. Cambridge, MA: MIT Press. Originally published in Computer 10, no. 3 (March 1977): 31–41.

“Learn a Language for Free.” 2016. Duolingo. Accessed December 18.

Licklider, J. C. R. 1990. “The Computer as Communication Device.” In Systems Research Center, In Memoriam: J. C. R. Licklider, 21–41. Palo Alto, CA: Digital Equipment Corporation. Originally published in IRE Transactions on Human Factors in Electronics HFE-1: 4–11, March

Lifelong Kindergarten Group at the MIT Media Lab. 2016a. “Scratch – Imagine, Program, Share.” Accessed December 18.

———. 2016b. “Smalltalk – Scratch Wiki.” Last modified December 13.

Manovich, Lev. 2013. Software Takes Command. New York: Bloomsbury Academic.

Maxwell, John W. 2006. “Tracing the Dynabook: A Study of Technocultural Transformations.” PhD diss., University of British Columbia.

McLuhan, Marshall. 1964. “The Medium Is the Message.” In Understanding Media: The Extensions of Man, 7–21. Cambridge, MA: MIT Press. Available at

MIT Media Lab. “In Memory: Seymour Papert.” Accessed December 18.

Murray, Janet H. 2011. Inventing the Medium: Principles of Interaction Design as a Cultural Practice. Cambridge, Massachusetts: The MIT Press.

Nguyen, Long Tien, and Alan Kay. 2015. “The Cuneiform Tablets of 2015.” Paper presented at the Onward! Essays track at SPLASH 2015, Pittsburgh, PA, October 25. Available at

“One Laptop per Child.” 2016. Accessed December 18.

Oshima, Yoshiki, Alessandro Wart, Bert Freudenber, Aran Lunzer, and Alan Kay. 2006. “Towards Making a Computer Tutor for Children of All Ages: A Memo.” In Proceedings of the Programming Experience Workshop (PX) 2016, 2125. New York: ACM.

Renfrew, Colin. 1999. “Mind and Matter: Cognitive Archaeology and External Symbolic Storage.” In Cognition and Material Culture: The Archaeology of Symbolic Storage, edited by Colin Renfrew, 1–6. Cambridge, UK: McDonald Institute for Archaeological Research.

Sutherland, Ivan. 2003. “Sketchpad: A Man-Machine Graphical Communication System.” In The New Media Reader, edited by Noah Wardrip-Fruin and Nick Montfort, 109–126. Cambridge, MA: MIT Press. Originally published in American Federation of Information Processing Societies Conference Proceedings 23:329-246, Spring Joint Computer Conference, 1963.

Wegner, Peter. 1997. “Why Interaction Is More Powerful Than Algorithms.” Communications of the ACM 40, no. 5: 80–91.

Wing, Jeannette. 2006. “Computational Thinking.” Communications of the ACM 49, no. 3: 33–35.

———.  2009. “Jeannette M. Wing – Computational Thinking and Thinking About Computing.” YouTube video, 1:04:58. Posted by ThelHMC. October 30.


Becky’s Work in Progress

Ideas Without Research Questions

I’m in the process of making and remaking my way through a good amount of content trying to pinpoint a specific research question and focus.

I took Alan Kay’s early ideas about interface design and using personal computers as collaborative learning tools as my starting point. And I began with a general question about what the legacy of this approach is. So much of development has been driven by technological advancements and the market, but was the more utopian learning component completely overlooked or does it exist in any form? Are there any areas into which it could be incorporated now? What were/are the obstacles?

After collecting what I thought were relevant readings we’ve already covered, I moved into new territory. When rereading Manovich, I noticed that he cites Kay’s article “User Interface: A Personal View.” That piece has been particularly illuminating when it comes to the theory behind GUI development. The GUI itself was based on child development theory and built as a learning tool. His “doing with images makes symbols” slogan caught my attention in particular (see image).


Kay also did an AMA over the summer in which he commented on current programming languages, simulations, education, and more. And he put out a recent memo on the subject of education. He’s shifted approach a bit and lays out a plan for creating “automated ‘intelligent’ tutor systems” using developing tech. The memo includes a number of applications to potentially explore.

I’ve started on McLuhan (I’d never heard the phrase “the medium is the message” before this class, so I feel like have to read it) and Wegner’s work, trying to learn a bit more about one of the theories that inspired Kay’s model and about more recent work on interactive computing models as extensions of Kay’s work.

And underlying Kay’s theories is a lot of child development research, specifically by Seymour Papert, Jean Piaget, and Jerome Bruner. I’ve read some Piaget. And I’d like to read more of this if I have the time before the semester’s over. But I don’t think I will.

One other thing I have not been able to wrap my head around is the role of language and reading and writing in all this. Kay stresses these things often, in a number of ways. They seem integral to his theories. He doesn’t seem happy about some of the impact computers have had on those skills: “it is not a huge exaggeration to point out that electronic media over the last 100+ years have actually removed some of day to day needs for reading and writing, and have allowed much of the civilized world to lapse back into oral societal forms (and this is not a good thing at all for systems that require most of the citizenry to think in modern forms)” (Time article, my emphasis). But he also makes sure to clarify that some things can’t be taught in stories. Additionally, the way he describes computers-as-a-medium in the context of McLuhan’s medium-is-the-message thesis sounds a lot like the way language has been described as unlocking a vast new world.

So that’s a lot. Based on all of it, I’ve come up with a few potential paths:

  • trying to map Kay’s ideas for the learning interface onto Peirce’s models, and comparing that to Murray’s take on design and Manovich’s take on software
  • something with Wegner, which I haven’t figured out yet because I haven’t finished those readings as of the time I posted this
  • looking at specific applications of tech today that seems in line with Kay’s vision and trying to determine to what degree it traces back to/is in line with that

Some thoughts about potential cases for that last idea:

Working Bibliography

Class Readings

Bolter, Jay David, and Richard Grusin. Remediation: Understanding New Media. Cambridge, MA: The MIT Press, 2000.

Deacon, Terrence W. The Symbolic Species: The Co-evolution of Language and the Brain. New York: W. W. Norton & Company, 1998.

Donald, Merlin. “Evolutionary Origins of the Social Brain.” In Social Brain Matters: Stances on the Neurobiology of Social Cognition, edited by Oscar Vilarroya and Francesc Forn i Argimon, 215-222. Amsterdam: Rodophi, 2007.

Engelbart, Douglas. “Augmenting Human Intellect: A Conceptual Framework.” In The New Media Reader, edited by Noah Wardrip-Fruin and Nick Montfort, 93–108. Cambridge, MA: MIT Press, 2003.

Greelish, David. “An Interview with Computing Pioneer Alan Kay.” Time. Accessed December 5, 2016.

Irvine, Martin. “Introduction to Affordances and Interfaces: The Semiotic Foundations of Meanings and Actions with Cognitive Artefacts.” Unpublished manuscript, accessed November 2, 2016. Google Docs file.

Kay, Alan C. “A Personal Computer for Children of all Ages.” Palo Alto, CA: Xerox Palo Alto Research Center, 1972.

Kay, Alan C. “Microelectronics and the Personal Computer.” Scientific American 237, no. 3 (September 1977): 230-44.

Kay, Alan, and Adele Goldberg. “Personal Dynamic Media.” In The New Media Reader, edited by Noah Wardrip-Fruin and Nick Montfort, 393–404. Cambridge, MA: MIT Press, 2003.

Manovich, Lev. Software Takes Command. New York: Bloomsbury Academic, 2013.

Murray, Janet H. 2011. Inventing the Medium: Principles of Interaction Design as a Cultural Practice. Cambridge, Massachusetts: The MIT Press.

Sutherland, Ivan. “Sketchpad: A Man-Machine Graphical Communication System.” In The New Media Reader, edited by Noah Wardrip-Fruin and Nick Montfort, 109–126. Cambridge, MA: MIT Press, 2003.

Wegner, Peter. “Why Interaction Is More Powerful Than Algorithms.” Communications of the ACM 40, no. 5 (May 1, 1997): 80–91.

Wing, Jeannette. “Computational Thinking.” Communications of the ACM 49, no. 3 (March 2006): 33–35.

———.  “Jeannette M. Wing – Computational Thinking and Thinking About Computing.” YouTube video, 1:04:58. Posted by ThelHMC. October 30, 2009.

New Readings

“Alan Kay Has Agreed to Do an AMA Today.” Hacker News. Accessed December 7, 2016.

Goldin, Dina, Scott A. Smolka, and Peter Wegner, eds. Interactive Computation: The New Paradigm. New York: Springer, 2006.

Kay, Alan. A Powerful Idea about Ideas. Accessed December 7, 2016.

Kay, Alan. “User Interface: A Personal View.” In Multimedia: From Wagner to Virtual Reality, edited by Randall Packer and Ken Jordan, 121–131. New York: W. W. Norton, 2001. Available at

Maxwell, John W. “Tracing the Dynabook: A Study of Technocultural Transformations.” PhD diss., University of British Columbia, 2006.

McLuhan, Marshall. “The Medium Is the Message.” In Understanding Media: The Extensions of Man, 7–21. Cambridge, MA: MIT Press, 1994. Available at

Oshima, Yoshiki, Alessandro Wart, Bert Freudenber, Aran Lunzer, and Alan Kay. “Towards Making a Computer Tutor for Children of All Ages: A Memo.” In Proceedings of the Programming Experience Workshop (PX) 2016, 2125. New York: ACM, 2016.

For Reference/If I Have Time

Selected works of Seymour Papert, Jean Piaget, and Jerome Bruner

Selected works of Nicholas Negroponte

Family History (Becky)


My grandmother in 1941. She was also a spot welder in the war.

My grandmother Lillian was born in 1918, a handful of years after C. S. Peirce’s death and Alan Turing’s birth. She lived to see the internet, and taught herself HTML code so she could embed midis of her favorite old songs in the body of emails she’d send me. Because she couldn’t see very well, she worked from a WebTV attached to the large screen of her television set. Needless to say, she was amazing.

Before I was born, she worked at Bell and AT&T as a switchboard operator, establishing connections between people by manually moving electrical cords and switches. Claude Shannon’s information theory with its bits, Henry Nyquist’s ideas about digitization and bandwidth, and much more grew from the telephone, which itself built on telegraphy and other inventions before it. The switchboards operated much like the early computers, which required people to manually move parts of room-sized machines to make calculations. Eventually, human-written binary code, electrical signals, and other innovations would come to replace the mechanical actions, paving the way for input-output machines modeled by Turing to become interactive computing systems built on software better modeled by something else.

Of course, my grandmother and I first started communicating before I knew language, let alone software. I had no concept of abstractions or the alphabet or other signs and symbols. But as a member of the symbolic species, I had in me a hidden capacity to map meaning, and gradually the syntax and semantics fell into place. I moved from primitive reactions to hunger, cold, and the like to using tools to play and eat. My understanding of icons, indexes, and symbols built up into an understanding and verbalization of the symbolic conventions that English speakers apply. My acquisition of language, or potentially just the ability to create artifacts, unlocked a capacity to store memory externally and build knowledge.

In the late 1980s, I extended those cognitive processes to computing systems thanks to my dad, who worked for Hewlett Packard. He was an electrical engineer by trade, trained in the navy, and went from working on radar oscilloscopes to computer scopes, from punch cards to PalmPilots, from huge pre-network machines to Oracle databases. He brought new HP equipment home to learn how it worked so he could fix it, which meant I got to explore computers in the living room and not just at school as a kid. I got lost in the DOS prompt, traveled the Oregon trail, and played my favorite game.

If Alan Kay’s vision had been fully implemented, I might’ve been learning code along with natural languages in elementary school. I might’ve been programming and learning by doing—taking my expanding symbolic capabilities and using them to conduct experiments with my computer as my teacher. Instead, I played Math Blaster and memorized multiplication tables.

But I shouldn’t be greedy. I have inherited a great deal. I’ve moved from holding multiplication tables in my head, to offloading my memories with a pen in notebooks, to exclusively using software on a laptop to store what I want to remember from class. And that software does more than just represent the written word; it is an interface to other symbolic systems as well. I can embed videos and audio into the files, or draw with a tool that still looks like a paintbrush but behaves in an entirely different, digital way. If I need the internet, I simply move to another layer thanks to Kay’s graphical user interfaces, windows, and more.

The concepts we’ve learned are helping me not just better understand the human condition but better understand my own family’s experience. I’ve come to learn that the cathode ray tubes used in old televisions were integral to the creation of technology that would lead to my grandmother’s WebTV, and many other more successful computing systems. That the HTML code my grandmother wrote consisted of symbols that both meant something to her thanks to a complex meaning-making process and could be read by computing devices that execute actions.

And there’s so much more in store. We’ve seen human cognition coupled with cars, but not the cognitive offloading that would accompany ubiquitous driverless vehicles. And we’ve seen HTML and hyperlinks and mice, but not widespread use of augmented reality lenses, wearable technology, and other versions of Douglas Engelbart’s vision of extending human intellect.

The curtain is slowly being pulled back on the meaning and complexity of this legacy and possibility. And the whole way, individual humans have been at the center, building on things that came before and finding new ways to expand their symbolic-cognitive processes.

Living Engelbart’s Dream (Rebecca and Becky)

There is a lot to say about how computing got to the metamedia stage, and about where it could be going next. Lev Manovich puts the focus on software as unlocking computing’s potential as metamedia. But humans, of course, wouldn’t be able to build computing systems or interact with them if they were not members of a symbolic species who can make new meaning from abstractions. Not only can we make meaning with symbols on a screen, but we can in parallel make meaning out of other symbols as well, from sounds to videos to facial expressions. The hybridization of multiple mediums would be, well, meaningless without those capabilities.

Working with that foundation, electrical engineers, mathematicians, computer scientists, and more moved from massive mechanical artifacts that take inputs and produce outputs to electricity-powered interactive computing systems that automatically feed outputs back into the system to produce new meanings. With the advent of high-level programming languages that humans can relatively easily read and understand, the process of writing programs that computers can then execute became more efficient. Software for computing systems proliferated, allowing humans to offload some of their cognitive burden onto the machines.

Most notably, Bush, Sutherland, Licklider, Engelbart, and Kay advanced computer design by putting forth plans for interfaces and human-computer interaction that would support and augment human intellectual capabilities. In particular, Kay sought to establish the PC as a tool for learning. His vision was significant because it gave users, children even, the ability to manipulate programs to solve unexpected problems and develop new ideas and processes.

While Kay’s vision seemed clear, it is interesting to think that our two mainstream commercial options for operating systems (Mac and Windows) are closed to normal-user manipulation. Some software can be modified, but doing so requires programming knowledge that isn’t universally taught. Apps, and the relative ease with which they can be developed, are potentially current manifestations of Kay’s vision.

Though Kay’s learning concept was not standardized, as we read each new kernel of information about the DynaBook and his other ideas, it became clearer that he in many ways wrote the blueprint that developers would follow for decades. Many of the concepts have been attempted in real life or already standardized: text editing, the mouse, graphical user interfaces, screen windowing, pressure-sensitive keyboards, synthesizers that can hook up to PCs to make music.

A particularly transformative concept was Kay’s vision of personal dynamic media, which was designed to “hold all the user’s information, simulate all types of media within a single machine, and ‘involve the learner in a two-way conversation'” (Manovich 61). This could be viewed as an early description of various AI technologies available today, such as Amazon Echo or IBM’s Watson. Yet, as Manovich explains, it also generally applies to the interactions with software that would come to transform the way we understand media.

Meanwhile, Sutherland with his Sketchpad prototype emphasized the need to interact with data in different dimensions. The division of his screen into four separate quadrants could be viewed as an early predecessor to the concept of hypermediacy. Engelbart’s concept of view control, which allowed users to switch between different views of data, shows the importance that he placed on the concept of user perspective and indicates his understanding of how to “layer” mediums.

However, Kay’s development of the graphical user interface, which provided a “desktop” on which different programs could be displayed and layered, is something that we truly take for granted when using modern computing devices. For instance, both Rebecca and Becky have many programs running simultaneously to process text, listen to music, send texts, manage emails, navigate multiple webpages, and more. We can toggle between the various windows and tabs with easy keyboard shortcuts and little thought, thanks to Kay’s design concepts.

Yet, both Rs independently ended up at the same ideas about tweaking this concept: flattening the layered interface system. In a sense, Microsoft’s OneNote and Google Docs are headed in this direction. However, this could go further by first, for instance, including as part of the word processing interface a web browser so that users no longer have to switch between windows but rather have everything contained in the workspace in which they are operating. (Word has some internet search functionality, but the integration doesn’t go as far as we have in mind.) Eventually, all media software could be combined into one layer. This might be awkward to do, given current hardware limitations and given the drive to make devices smaller, but not impossible. It could work well with larger fields of view, such as with a virtual or augmented reality computing system that is not limited by display size. The goal would not be to simply play music in iTunes or edit movie clips in iMovie or draft documents in Word and then put them all together. Rather, the point would be to allow someone to use these various forms of media in one software platform in order to access them in a more integrated way.

These readings brought up a number of additional ideas for both of us, but for ease of reading, we’ll keep them brief and discuss in person. A common theme among Kay’s and others’ ideas seemed to be the concept of developing interfaces that are more adapted to the human body: the chair with the keyboard in it, for instance. This category also includes the idea of eliminating the keyboard altogether and just using voice or graphical input to interact with the computing system. This is an area that has not been explored fully, but that would potentially be of great benefit to all those with stiff necks and more. Innovation, and history in general, also seems to flow in cycles to a degree, with innovation, consolidation and refinement, and innovation again, roughly speaking. It seems as if we might be ready for that next age of innovation in the computational world.


Works Referenced

“Alan Kay — Doing with Images Makes Symbols.” Filmed 1987. YouTube video, 48:20. Posted by John DeNero, November 12, 2013.

Bolter, Jay David, and Richard Grusin. Remediation: Understanding New Media. Cambridge, MA: The MIT Press, 2000.

Engelbart, Douglas. “Augmenting Human Intellect: A Conceptual Framework.” In The New Media Reader, edited by Noah Wardrip-Fruin and Nick Montfort, 93–108. Cambridge, MA: MIT Press, 2003.

Kay, Alan C. “A Personal Computer for Children of all Ages.” Palo Alto, CA: Xerox Palo Alto Research Center, 1972.

Kay, Alan, and Adele Goldberg. “Personal Dynamic Media.” In The New Media Reader, edited by Noah Wardrip-Fruin and Nick Montfort, 393–404. Cambridge, MA: MIT Press, 2003.

Licklider, J. C. R. “The Computer as Communication Device.” In Systems Research Center, In Memoriam: J. C. R. Licklider, 21–41. Palo Alto, CA: Digital Equipment Corporation, 1990.

Manovich, Lev. Software Takes Command. New Yor: Bloomsbury Academic, 2013.

Sutherland, Ivan. “Sketchpad: A Man-Machine Graphical Communication System.” In The New Media Reader, edited by Noah Wardrip-Fruin and Nick Montfort, 109–126. Cambridge, MA: MIT Press, 2003.


Memory Supplements and Cylons (Becky)

Bush, Sutherland, and Engelbart are all discussing ways that humans can interact with existing and future technology. And each is discussing ways to bridge the gap between humans and computing systems—that is, developing interfaces. But the differences in and progression of approaches is fascinating.

Particularly interesting to look at is the way in which the authors propose to extend cognition. Bush’s Memex device seems to be all about offloading information, organizing memories in a receiving device. These memories can be linked and stored, but the ideas can’t be manipulated. In Sutherland’s Sketchpad, graphical manipulation is possible via the interface—the device is a more active participant, so to speak, in the process of meaning making. Engelbart, meanwhile, extends these ideas even further and wants to change human behavior to build a sort of symbiotic system of human-computer interaction and augmented human intelligence.

Bush describes Memex as a “memory supplement.” A piece of furniture, the technology is meant to be integrated as seamlessly as possible into humans’ surroundings. The human user does all the thinking and processing and then the products are stored in the Memex  as images or sounds (in miniature!) that can be recalled. They can be sent to others and loaded into other Memex devices to share the information wealth. This is essentially the process of saving and sharing files today.

The Memex user is responsible for establishing trails and remembering codes, which seem to be somewhat unnatural even if they’re mnemonic. Perhaps that is why we ended up with icons that look like folders and software that can establish the trails for us (and that is more organized around “goals” as Licklider describes). But the general concept of linking information has withstood the test of time. (As has, perhaps, Bush’s “roomful of girls”/secretaries in the form of Romney’s “binders full of women” and others’ colorful phrases.)

Wearable tech is a not-fully-realized projection of Bush’s ideas. Technology is becoming more seamlessly integrated into human behavior and the environment as devices become smaller. The GoPro looks close to the “little lump larger than a walnut” that Bush describes a “camera hound” wearing in the future. And thanks to ubiquitous smartphones with cameras on their backs, the capability to record life in still photos as Bush imagines is standard these days. But there is much more to do along this path.

Bush's scientist of the future and a GoPro with a head mount

Bush’s scientist of the future and a GoPro with a head mount

With the Sketchpad, or the computerized Etch A Sketch, we start to see the beginnings of interfaces that are more a part of the cognitive process as opposed to memory storage devices. Many of the descriptions seem as if they could be explaining interactions today, with Photoshop, for instance. The ideas of recursive functions and creating instances of master versions have certainly spanned the decades and been deeply ingrained in today’s software. And tablets seem to be descendants of the Sketchpad and light pen ideas.

Etch A Sketch from CC BY-SA 3.0

Etch A Sketch from CC BY-SA 3.0

Meanwhile, if I’m understanding correctly, Engelbart builds on Bush’s linking trails and describes technology that performs in a way that is similar to the way humans make meaning and manipulate symbols—a non-serial conceptual structure. Yes, the technology he describes offloads information and stores it. But it also helps humans in the process of making meaning, if humans can make little changes to their MOs.

Many of the ideas that Engelbart describes sound eerily familiar. Parts of his process of working with statements could easily be explaining today’s word processing. His clerk seems like it could be describing today’s software or hardware, I can’t quite tell which.  And I wonder if Engelbart (and Licklider) could’ve imagined where the networking concepts have led. Yet, the way the augmented architect operates, by using a pointer and then moving his hand over the keyboard, is a bit cryptic—is this is describing what we do today with a mouse and a keyboard or something else that hasn’t been adopted (97)?

Taking a broad view, Engelbart appears to be making the case for even closer integration between humans and computers. Bush too wondered about a more direct path for transferring information, though in a different context. What is the natural extension of these trains of thought? Augmented reality? Neural implants? Battlestar Galactica–style cylons?

Beautiful Is Better Than Ugly (Becky)

I knew the process of making meaning as part of the symbolic species was complex before I started on this week’s adventure. But as I tried to wrap my human brain around all of the processes needed to translate human speak into electrical signals, my mind was blown.

At each stage of the transition from symbols that mean things to symbols that do things, an astounding amount of human symbolic power was needed to create these technologies in the first place. And that it all runs efficiently and quickly and mostly without errors, no doubt in large part thanks to Claude Shannon, is hard to believe.

Using the efficient software and apps are humans, with our apparently complex, ambiguous, irregular, uneconomic, and limited natural language (thanks for the adjectives, Evans). We make meaning out of our natural language despite and because of its imperfections, but computers can’t make sense of it like we do. They need precise and unambiguous instructions to do their jobs.

One of the interfaces that helps us communicate with the machines is programming language. Python is one. We can read and write Python, make meaning of it; computers can execute it with the help of some other code. Interestingly, I’ve always used the word “understand” rather than “execute” in that last part, but I stopped myself this time because while the machines are processing symbols, they aren’t understanding meaning. They’re executing.

Python is a relatively high-level programming language that was developed to be accessible and readable to humans versed in natural language—the principles “beautiful is better than ugly” and “readability counts” are part of the canon. Yet, I find trying to learn Python a bit difficult simply because it is so close to natural language. I assume that if or for-in statements should do certain things based on my knowledge of English, but as CodeAcademy and Coursera have taught me, my assumptions are not always correct. I wonder if a more abstract language would be better for me. But I digress.

A compiler (or an interpreter for other programming languages) translates the code I’ve written and that I can understand into code the machine can do something with, or at least starts that process. This has usually been the boundary of the black box for me, but I think it’s been pried open this week.

The compilers map our code on to Python’s underlying grammar, a more abstract symbol system that some very smart people created. That grammar translates the information into machine code, which directs instructions to the computer’s hardware in more symbols—binary 1s and 0s—and ends up in electrical signals (I think). The machine, through symbols, is acting on the commands I gave it using another symbol system. And the symbol system I made meaning of translates into a physical action in the form of electrical pulses. The results of my little program are stored in memory until the program wraps up and the results are sent back to me so I can interpret and make meaning of them. (Although, I think there is another layer before machine code for Python so it can work with lots of different operating systems, kind of like Java, but I’m really on shaky ground here.)

With all this complexity and all the work that went into developing these processes, let alone the complex pieces of software and tiny pieces of hardware involved, I probably shouldn’t get too grumpy when Microsoft Word freezes up every once in a while.

I saw the fruits of compilers when using Python, but I think I’m finally starting to grasp how they work thanks to Evans, Denning, and Martell. The P=NP problem and the concept of stacks are also much clearer than they’ve ever been. Recursion in general makes a lot of sense, and Python training has helped to clarify that more, but the idea as described by Evans is still a little fuzzy. And I find myself wondering about the concept of cognitive scaffolding—does the concept have parallels in computing? Both the process of using heuristics to get answers to problems that can’t be logically computed (described by Denning and Martel) and regular expressions in programming language reminded me of the concept of cognitive scaffolding, but I imagine this might be an incorrect comparison.

I leave this week wondering if computation is the universal language. And I certainly see the value of teaching computational thinking. But there is beauty and adventure in the imprecision and inefficiency of life that would certainly be a shame to lose.


Works Referenced

Denning, Peter J., and Craig H. Martell. Great Principles of Computing (Cambridge, MA: MIT Press, 2015).

Evans, David. Introduction to Computing: Explorations in Language, Logic, and Machines. August 19, 2011 edition. CreateSpace Independent Publishing Platform, Creative Commons Open Access:

“PEP 20 — The Zen of Python.” Accessed October 27, 2016.

Wing, Jeannette. “Computational Thinking.” Communications of the ACM 49, no. 3 (March 2006): 33–35.

———.  “Jeannette M. Wing – Computational Thinking and Thinking About Computing.” YouTube video, 1:04:58. Posted by ThelHMC. October 30, 2009.


The Distributed Cognition of R^2 (Becky and Rebecca)

Humans use tools and technologies, no matter how primitive, to facilitate and increase their cognitive brain function. This capability is particularly important because the process of storing and recalling information in our memories, either long-term or short-term, can at times be problematic. For example, the case of Otto demonstrates someone whose memory does not allow him to properly retain information, and thus he relies on his notebook to function as his memory (Clark and Chalmers).

Microsoft’s OneNote offers another example of how technology can potentially extend cognition in this way. The fundamental behavior of taking notes with this software, which Rebecca and Becky both use, represents a number of different symbolic and cognitive-offloading processes.

Before we can take notes, we have to process the information presented in whatever lecture we’re attending. The professor communicates information to us—by using, say, the English language or images in a PowerPoint. We have to translate the information communicated into another sign system—that of the written English language. This requires knowledge of the English alphabet, grammar, lexicon, context, and so on. When taking these notes, we might draw some tentative conclusions of our own, creating new meanings by making connections between the ideas presented.

We write these ideas down using an artifact that is a token of a type—a particular instance of a program installed on our laptops, which are made up of a number of different technologies, from the keyboard to the graphics card to the RAM. We previously learned the association between the purple N icon and the OneNote software. Even though the icon is only tangentially related to the process of note-taking because it includes stylized lined paper (see image), we still know from conventional use that clicking the icon will open the software we need. Of course, this entire process functions on our understanding of the various sign systems involved, such as the English language and the keyboard interface.



These artifacts help us offload our cognitive burden in a number of ways. We are in effect storing our memories in these digital files. If we didn’t have these tools, each of us would have to store all of the information in our mind, based on the constraints of time and the sequence in which the ideas were presented. But thanks to these artifacts, we don’t have to mentally store and recall as much. We can instead store these lecture memories externally and access them whenever we want to by opening OneNote and drawing on our symbolic talents to read the material. This facilitates new and abstract thought, freeing up our brains to use those ideas in more complex ways than simple recall, such as to solve problems.

This file can also be used to distribute cognition and communicate with others. We can email the notes to another person who can read them and learn from them. That person will need the proper tools—a computer and the software, for example—to access that information. And only a person with the proper contextual knowledge will be able to understand the full meaning. The note-reader has to be part of the English-speaking community, for example. Additionally, the notes may, for instance, make shorthand references to earlier content, so the note-reader must understand that material already covered in the class.

But is this process truly cognition? Can the mind actually be extended to external technologies? As discussed in “Cognitive Offloading,” terms such as the “extended mind” and “distributed cognition” are somewhat misleading (Dror, and Harnad). In some ways, it seems contradictory to most of what we’ve learned thus far to suggest that the act of cognition occurs outside of the brain. Depending on the nature of the sign, whether icon, index, or symbol, there may be no apparent connection between the sign and its object. According to Dror et al, it seems like a cognizer is needed to make that connection—to perform the cognitive exercise of interpretation and understanding.

External artifacts, individuals, and so on can both trigger and impact acts of cognition. But the ultimate source of interpretation appears to be someone who can participate in generative, recursive, and at times unpredictable processes of thinking and creating meaning.

The “extended mind” theory seems to be a nuanced explanation of how cultural associations are stored, recalled, and utilized, whether through technology or human organisms. In this sense, what are the main distinctions between this theory and something like cultural progress?


Works Referenced

Clark, Andy. Supersizing the Mind: Embodiment, Action, and Cognitive Extension (New York, NY: Oxford University Press, USA, 2008).

Clark, Andy, and David Chalmers. “The Extended Mind.” Analysis 58, no. 1 (January 1, 1998): 7–19.

Dror, Itiel E., and Stevan Harnad. “Offloading Cognition Onto Cognitive Technology.” In Cognition Distributed: How Cognitive Technology Extends Our Minds, edited by Itiel E. Dror and Stevan Harnad, 1–23. Amsterdam and Philadelphia: John Benjamins Publishing, 2008.

Currents and Cryptography (Becky)

I tried to modify the information theory diagram to account for meaning-making in a nice, concise way. But it quickly became very crowded. Narrative prose will have to do, and I’ll start with this essay as an example.

The first word of this sentence was “The,” and it has meaning to me. I understand the meaning of the capital T, for instance; in this case it means the beginning of a sentence, a new thought, given the period and space before it. I know all this and more because I’m part of a community of symbolic beings that understands English-language conventions.

So I tell the computer to make a capital T. Thanks to a helpful keyboard a very smart person invented and software and memory and Boolean algebra and electricity and more, I can do that without speaking computer speak (though ASCII tables are nice windows). My key press is translated into my computer’s “language” (to steal a particular linguistic term) of 1s and 0s. The representation that I understand as the capital letter T appears on the screen.

An English speaker could look over my shoulder and understand what that letter and all these letters mean, a process which Peirce explains in more depth.

And I could send these words or images in an email to someone else. Thanks to machinations I do not yet understand (but hope to!), what I do would be translated into electrical pulses that correspond to binary values that are transmitted through ethernet cables to another mechanical device some distance away that can decode them. Or they’re sent over radio waves. Or in light pulses. Or something else.

In any of these cases, the goal in terms of the information theory model is to replicate a “message,” as Shannon put it, as completely as possible. The model does not describe the transmission of meaning in the Peircean sense but rather the transmission of information in the form of bits. (Of course, someone capable of making meaning out of abstract ideas had to create that model in the first place.)

For my email message to have meaning at its destination, some member of a symbolic species on that end must be capable of making that meaning. A sign doesn’t exist until it is interpreted as such. This means the actor at the destination must be operating in the same context as the source. He or she must understand English or have a good translator. The medium matters for meaning as well. Most users of email know that all caps mean SHOUTING and should be used sparingly. Terseness is OK in texts, but could be rude in email. And so on.

I don’t want to venture too far into book-report territory, but I found the readings helpful illustrations of the meaning-making process—the stories about cryptography that Gleick retells, for example. There are also a few scenes in Imitation Game, the movie about Turing and the Enigma, that might help and are conveniently insertable into this essay. Take this one (a dramatized version of what actually happened, of course).

Based on the meaning of already-decoded messages and their knowledge of language conventions, code breakers understood that certain words—greetings, the weather—always came up in German messages. They built a machine that could focus on words that they already understood would be there. (Has technology advanced to a point where computers using algorithms can identify these seed words?) Floridi hints at something similar with his Normandy discussion (44).

On the source rather than destination end, it seems that Day is illustrating how the development of information theory’s conduit metaphor and its application to nontechnical areas were influenced by a specific meaning community—a Cold War environment. He says information studies should be rethought for today’s context.

This week—even more than others—I’m thinking about AI. When all of these factors and more are considered, it is no wonder the task of building a new kind of human machine is so difficult.

Not the Gumdrop Buttons (Becky)

Shrek is one of my favorite animated movies. I think I still love it after ruminating on it as much as I have over the past few days. I looked at the scene in which Lord Farquaad interrogates the Gingerbread Man (Gingy), which involves conventional visual and linguistic sign systems.

In this scene, Farquaad and Gingy are talking to each other, and they use words, phrases, and sentences from a lexicon that is common to those who speak English. Most of these utterances follow syntactical rules. But the characters also employ defective lexical items, such as Gingy’s “pthuh” when he spits, which has semantics and phonology but doesn’t have syntax—although I’m not sure this example has meaning without the spitting visual. (I also wonder what a visual “defective lexical item” would be. Abstractions, maybe? It has semantics and “phonology,” but perhaps it does not have syntax.)

These lexical items “serve as interface rules,” as Jackendoff writes, that “correlate the parallel structures.” They are the bridges between the three major parts of the architecture of language—phonological, syntactic, and conceptual structures—which works in a nonlinear way. Interfaces seem very important, but how they actually work is unclear to me.

The interfaces and the parallel architecture as a whole help us make sense of what Gingy and Farquaad say. Our phonological structures somehow decipher the uttered lexical items according to learned rules; because of these rules, we know how the word “monster” is pronounced, for instance. Meanwhile, thanks to syntax, we understand the lexical items in the phrase “you’re a monster” to be instances of more general categories, such as contractions and nouns. That allows us to understand, for instance, more about what is being said because we know how these types are supposed to behave in English. And because of the semantic formation rules we know, we understand the negative implications of the word monster. In terms of sign functions, there’s more to this equation, but it seems that it boils down to the nature of a monster (the object, to use Peirce’s term) + the person of Farquaad (the representamen/sign vehicle) = Farquaad is being negatively described (the interpretant). I believe the resulting association of Farquaad with a monster is highly conventional, while Farquaad is less conventional in comparison (though I think both are symbolic signs).

Intersubjectivity and pragmatic context come into play quite a bit as well. Beyond the general Western fairy tale context, Gingy’s “eat me!” comment assumes the viewer understands that, though the directive could be taken quite literally in this case because he’s supposed to be made of gingerbread, he’s using the phrase as an expletive.

Something like a parallel architecture is potentially making sense of the visual side too. Our visual “phonogical” structures are perhaps decoding minimal visual units (pixels?) into something our brain understands. These units come together to form different patterns—of borders of shapes, light variation, textures, and so on. Something in our brains is capable of recognizing the patterns—the syntactical structures or the “phonological” structures, or some combination of both? The animators created these visual patterns according to syntactic rules that we can decipher thanks to shared understanding; something that is supposed to be like a human, for instance, shouldn’t have lips on its forehead. And semantic structures decode these images in different ways, such as Gingy’s eyebrow and mouth movements being understood to convey concern.

And then there are the previous experiences that influence our visual understanding of the scene. To use one example, anyone familiar with interrogation and torture visuals understands that this is a menacing situation without the characters even saying a word thanks to various (analogical?) signs, such as the lighting, Gingy’s placement on the table, the evidence of “milk boarding,” and other items. I can imagine a child unfamiliar with waterboarding asking why there was milk around Gingy’s head.

The visual interfaces here are a bit mysterious. I wonder if the patterns themselves could be interfaces; they seem as though they could correlate the various structures.

While much of the scene can be understood using just one of the sign systems, there are parts whose full meaning seems to depend on both visual and linguistic information. I wonder what kind of interfaces might bridge the divide between these two sign systems to bring all of the meaning together. Is it the same parallel architecture making sense of it all? Is there some interface that encompasses all sign systems? Now, I’m probably just be grasping for straws, or gumdrop buttons.


Chandler, Daniel. Semiotics: The Basics. 2nd ed. New York: Routledge, 2007.

Irvine, Martin. “The Grammar of Meaning Systems: Sign Systems, Symbolic Cognition, and Semiotics.” Unpublished manuscript, accessed September 28, 2016. Google Docs file.

Jackendoff, Ray. Foundations of Language: Brain, Meaning, Grammar, Evolution. New York: Oxford University Press, 2003.

The Bee and the Rosetta Stone (Becky)

I tried to come up with a good lede that tied all of my thoughts together. But I think it’s better for me to just leave this as a stream of consciousness. So diving right in…

Language is a way to communicate your thoughts to others. From that basic description, it would seem as though a visual symbolic system might function in a way similar to a spoken language. Does it when you look at the details?

Take photography in particular, and the linguistic concepts of phonology, syntax, and semantics. In a spoken language, sounds (with which phonology is concerned) and meaning/concepts (semantics) have to be encoded in a certain way, and there needs to be a system to organize these ideas (syntax), to make sure a speaker’s brain and a receiver’s brain can understand the intended meaning. Like the ears take in sounds and meaning, the eyes process images. Those images and concepts then have to be encoded in a way that brains can understand them. Photography has a grammar, too. You can play with light using combinations of shutter speed and aperture to convey different meanings, as you can use various words to convey meaning, for instance. Similar to sentence structure, the rule of thirds in photography tells you where to place points of interest for maximum effect. (But of course, photographic license, like poetic license, is also allowed.) Ok. All of that seems relatively comparable between systems.

The rule of thirds in photography

The rule of thirds in action (from the Digital Photography School)

Photography also has a lexicon made up of units, like words in language, that are combined to form an infinite number of images (phrases and sentences). I think those minimal units would be pixels in digital photography, and photographers can use cameras and the ophthalmological (stealing from the eye docs) and syntactic tools at their disposal to manipulate and combine those pixels, conveying layers upon layers of meaning, much like in a spoken language.

But the bar seems to be lower for visual media, somehow. Photography, for instance, is capable of conveying meaning like a spoken/written language, but not in a way that is always as precise as a language (assuming that that language is spoken to others who understand it). So, if the photographer’s goal is just to reproduce something in a cut-and-dry iconic way, then it will be easy for the viewer to understand what is meant. Yet if the photographer instead seeks to convey some deeper commentary on society or the idea of love or some other sentiment through his work, the viewer enters murkier water. Those meanings can’t be precisely conveyed in a photograph. The meaning can easily be misinterpreted or interpreted differently by the viewer because the tools used aren’t as fine. To throw in some Kate Wong—today, we can uncover and view old cave paintings and other artifacts, but we can’t be sure of their intended meaning. Yet, the Rosetta stone unlocked worlds of precise understanding.

In another way, however, a visual language can be more effective than a spoken language. Photography, for example, can convey meaning across a range of cultures. It may not convey precise meaning, but a visual symbolic system can bridge communities in a way that many spoken languages cannot (English is, potentially, nearing exceptional status?).

Funny, when I started thinking through all this, I was convinced that visual media couldn’t possibly be a language. Now I think the opposite. I keep returning to Saussure’s arbitrary relationship between sound and meaning, for one. There seems to be something below all of this that remains constant regardless of changes in words spoken or aperture selected. And I think that somehow I’ve gotten back around to Merlin Donald’s and other’s ideas about the importance of the spoken/written word in spurring development. The basic idea is the same between the systems, but a spoken/written language is more effective.

(I have questions. Of course. But I’ve already taken up far too many words. I’ll save the rest for class.)

Works Referenced
Donald, Merlin. “Evolutionary Origins of the Social Brain.” In Social Brain Matters: Stances on the Neurobiology of Social Cognition, edited by Oscar Vilarroya and Francesc Forn i Argimon, 215-222. Amsterdam: Rodophi, 2007.

“Rule of Thirds in Photography.” Digital Photography School, May 2, 2006.

Wong, Kate. “The Morning of the Modern Mind: Symbolic Culture.” Scientific American 292, no. 6 (June 2005): 86-95.