Category Archives: Week 9

Draft: New Designs of AAC and TTY from Older Design Softwares

Jalyn Marks

“Why should humanists, social scientists, media scholars, and cultural critics care about software? Because outside of certain cultural areas such as crafts and fine art, software has replaced a diverse array of physical, mechanical, and electronic technologies used before the twenty-first century to create, store, distribute and access cultural artifacts” (Manovich, 2013).

Computing, while initially designed for specialized fields like the military, government, and scientific research communities, was redesigned for non-technical and non-specialist users. Computer scientists like Alan Kay, Tim Mott, and Doug Engelbart drew upon other fields, like cognitive psychology, humanities, and business, which influenced their designs to be more user-friendly, intuitive to learning, and easier to use. The more people who have access to using computers, the more creative and productive society will be as a whole.

“Software has become our interface to the world, to others, to our memory and our imagination—a universal language through which the world speaks, and a universal engine on which the world runs” (Manovich, 2013). An example of a an older supporting system technology that has been adapted into new design is the use of point and click to Augmentative and Alternative Communication (AAC) head pointers.

A mean uses an AAC head pointer with a laser to show his partner, a woman, what letters he is using to spell out what he wants to say. They are both smiling.

A mean uses an AAC head pointer with a laser to show his partner, a woman, what letters he is using to spell out what he wants to say. They are both smiling.








Doug Engelbart invited the mouse and the point-and-click system (Moggridge, 2007).


TTY is like a desktop. “The idea of a desktop came to [Thomas Mott] as part of an ‘office schematic’ that would allow people to manipulate entire documents, grabbing them with a mouse and moving them around a representation of an office on the screen” (Moggridge, 2007).


Other notes from readings (will update post later):

Media software, what I use every day, shared traits, modularity principle: software to application software to media software, “software enables global information society” like knowledge workers, symbol analysts, creative industries, and service industries,

“hypertext” to mean a body of
written or pictorial material interconnected in such a complex
way that it could not be conveniently presented or represented
on paper.21 Theodore H. Nelson described hypertext as more than links and text; instead, hypertext is a symbol within a greater work, referencing anything, not just a link, not just another picture. It can be anything (Manovich, 2013).

Industry supported them more than academia. Nelson said, “a new, readable medium” (Manovich, 2013). Users can choose between “many different views of the same information”.

“We can add new properties or even invent new types of media by simply changing existing or writing new software. Or by adding plug-ins and extensions, as programmers have been doing it with Photoshop and Firefox, respectively. Or by putting existing software together” (Manovich, 2013).

According to Kay, the key step for him and his group was to start thinking about computers as a medium for learning, experimentation, and artistic expression which can be used not just by adults but also by “children of all ages. Cognitive psychologist Jerome Bruner added on to Piaget’s theory of logic. “Mentalities do not replace each other but are added.”

David Canfield Smith referenced Pygmalion in the title of his thesis about a creative programming environment. The book by George Bernard Shaw

Larry Tesler’s license plate said “NO MODES” because he wanted to design user-friendly software, understanding that modes make things more complex.



Interaction Design

Alan Kay views computers as “remediation machine” because he wants to turn them into a “personal dynamic media” and remediation means “the representation of one medium in another” and this tells us that there is a relation that’s connecting the old media and the new and as Manovich has put it – remediation is the new digital media’s defining characteristics. Kay also wants to create a computer as an umbrella, including all sorts of media or as Kay names it a “metamedium”. The next design step is simulation. Kay believes that “simulation is the central notion of Dynabook”.

I enjoyed reading through Moggridge’s book and I actually also quickly read about the fourth chapter as well and I found David Liddle’s three stages of technology use very interesting. Liddle divides the adoption of a technology into three stages: the enthusiast stage, the professional stage and the consumer stage. Not only is the adoption of technology very important but also the classification of users as it is part of interaction design. I personally think that there is no such thing as perfect design suitable for any group or user at any stage. Every design has a certain audience. If the design can satisfy the appetite of the users, then I think it is a successful design. The group of “enthusiasts” can almost be “ignored” for interaction designers because their own preferences are too obvious, and they are often intoxicated by the entertainment brought by the technology itself (there are lots of apple enthusiasts for example and they will buy apple products no matter what), but I think it is still very necessary to understand the opinions of this part of the group in the early stage of the design because these people often very familiar with the core advantages of the technology and they will probably have a more precise understanding of this technology. So for designers, knowing this kind of information will definitely help them consider the impact of this technology in the product and design and therefore make a prominent focus. The more difficult challenge probably lies in finding the good balance between the needs of your professional users and general pubic consumers. For example, FTP, File Transfer Protocol, is a software protocol for exchanging information between computers over a network. We can refer users who are accustomed to or often use FTP as “professional users” and refer those who basically don’t use FTP or only know its name (like myself) as “consumers”. So if you want to design an FTP software interface and conquer these users, interaction designers must consider the general consumers’ lack of professional background, and have a balanced consideration of technical and professional requirements and easy use in the design plan. Otherwise, for consumers, if they can’t make them work, they take them back to the store. This is the real threat.



Bill Moggridge, ed., Designing Interactions. Cambridge, MA: The MIT Press, 2007.

Manovich, Lev. Software Takes Command. New York: Bloomsbury, 2013.

Contemplating the Future of Mixed Reality Technology Through the Lens of Combinatorial Design

Victoria Gomes-Boronat

This week’s reading was especially exciting because it explored the various directions that computer technology development could have gone and also introduced computer science as a world of possibilities and “happy accidents”. As Professor Irvine explains, “the development of personal computers happened by happy accidents of convergence where multiple forces and histories intersected: technical developments, research, and new philosophical contexts for developing computing beyond industrial, business, government, and military applications,”(2018, p. 8).

Tim Mott stumbled upon one of those happy accidents in a bar of all places. He was waiting for a friend and constructed an “office schematic” interface which would later be known as the desktop. And while many other computer scientists were attempting to make similar interfaces using three dimensional, true-to-life simulations, Tom’s colleague, Larry Teslar, realized the genius of the simple two-dimensional icons as representational symbols of everyday office functions (Moggridge, 2007, p. 53). Therefore, the 3D simulation designs were dropped, and icons became king.

However, with the advancement of camera technologies, processing power, Bluetooth capabilities, and high-resolution interfaces, augmented reality has emerged as a disrupter of the status quo. If computer scientists can agree that computing needs to, “be preconceived and redesigned for ‘augmenting human intellect’, that is, for enhancing and expanding human intellectual abilities and creativity through symbolic representation processes,” (Irvine, 2018, p.8), then wouldn’t augmenting reality using computing be the next logical step?

Because of their combinatorial design, modern phones and tablets are able to augment reality for a variety of purposes: entertainment, education, and design to name a few. Apple has been a huge supporter of AR technology and its website demonstrates how AR can enrich the lives of its product users and includes a list of some great augmented reality applications. Even social media applications such as Instagram, Snapchat, and Tik Tok use augmented reality to create various filters and tools to customize their photo and video experience. Games such as Pokemon Go! allow fans to experience a more fantastical version of their world.

But that’s not enough. Augmented reality is still not perfect because of the constraints of physical technology, i.e. small phone screens that limit your immersion in the augmented reality. The following video explains how transparent screens/devices could make augmented reality more realistic, however, in order to truly make a mixed reality world possible, we need to use combinatorial design to create technologies that don’t have the same constraints as a phone, i.e. augmented reality glasses. Soon the beloved simplistic icons of design’s past may become obsolete and replaced by the three-dimensional, real-world representations that used to be considered impossible.


Bill Moggridge, ed., Designing Interactions. Cambridge, MA: The MIT Press, 2007.

Irvine, M. (2018). Computing with symbolic-cognitive interfaces for all media systems: Design concepts that enabled modern “interactive” “metamedia” computers. Unpublished Manuscript.

Augmented Reality. (n.d.). Apple. Retrieved October 21, 2020, from

Why Samsung’s Transparent Phone will fail. (2019, April 14).

A Few Glimpses into the Design Evolution of Computers

I think it was the vision of an open source, accessible computing device to anyone who would want to operate it, that was responsible for beginning of the era of personal computing. Memex, the proto hypertext system that acts as a central universal human augmented knowledge system was conceived as a democratic pool of knowledge (Bush, 1945, p44).

So many design steps that developed in a staggered fashion in the past 70 years are responsible for the way the computer is assembled currently, both physically and digitally. Computers began as numerical and logic processors (Irvine, 2018, 7). But soon in  the 1950’s, we could find the first manipulation of bits to convert these number crunchers into cognitive symbolic mediators. The first stage was alphanumeric symbols printed on scrolls of paper using binary representation. By the 60’s we had CRT screens, instruments used in physics laboratories, that could be repurposed to display inputs and outputs in a computer using bit mapping: a type of memory organisation that enables pix maps. Pix maps are what allow storing of two or more colours in pixels that allow for an image to be displayed on this CRT.

This was the birth of an interface to directly interact with the digital and soon there was a need for smarter “user interfaces”. Douglas Engelbart invented the first mouse using a small blob of wood and two wheels in the bottom, at right angles to each other,  so that the corresponding selection on the screen could move across the X-Y axis (Of course the cursor had to be invented yet) (Moggridge, 2007, 27).

At this point, I think the notion of augmenting human intellect as the defining paradigm of how computing evolves began to shake a little. “It is easy to understand the idea of going for the best, of catering to the expert user, and then providing a path to get there from a simple user interface designed for the beginner. In practice, however, this has proved to be the wrong way round, as it’s not easy to get something right for the beginner when your design is already controlled by something that is difficult to learn.” (Moggridge, 2007, 36). The ideal of reaching maximum augmentation of human intellect by catering to the expert user can be seen in the creation of one of the first proper assemblies of the modern computer. The NLS or the “oN-Line System”  was the first to employ the practical use of hypertext links, the mouse, video monitors, information organized by relevance, and other modern computing concepts. (Moggridege, 2007, 33-37)

It was Larry Tesler who realized the value of participatory design and how it was better to observe people interacting with your interface and make changes accordingly. He started performing what we today call usability tests. He is also respsonsible for the proper integration and invention of the “Double-click”, “Cut”, “Paste”, and Cursors (or improvisation in the case of cut-copy-paste).

All these functions and many other such developments are responsible for the way the computer now presents itself. For example, there may still be a key called “insert” which I last saw in my late 2004 windows enabled computer. This key once was used to insert characters or approve of commands. With the invention of the “Cut & Paste”, the insert key kept falling out of relevance and was repurposed for a few other actions until it became obsolete. It no longer exists on the mac keyboard.



Bush, Vannevar (1945). “As We May Think,” Atlantic Monthly 176 (July 1945)

, Bill. ed., Designing Interactions. Cambridge, 2007. Excerpts from Chapters 1 and 2: The Designs for the “Desktop Computer” and the first PCs. MA: The MIT Press, Pp. 17-68

Irvine, Martin. 2018. Introduction to Symbolic-Cognitive Interfaces for Computer Systems: History of Design Principles (essay).

Irvine Martin. Computer Interface Design Concepts: Major Historical Developments (Original Documents), Compilation.


Conceptual Design Steps

Conceptual Design Steps

Yingxin Lyu

In Lev Manovich ‘s article1, the author describes Kay’s design principles. Alan Kay first defined computers as “remediation machines,” which means that software functions of computers can continuously be remediated by people and become more and more powerful. After trying to transfer various media into software and let them function on computers, computers become “personal dynamic media.” The reason why Kay wanted to turn computers, which were calculation machines, into metamedia functional machines was that he hoped that people could use them for learning, discovery, and artisan creation. It is a huge step from a calculation machine to a “personal dynamic media.” People always found it hard to leap out the established fact, but many Kay and his team created new ideas and ways to remediated computers, which transferred a computer as a tool used by people when inputting orders into an involver that involved in a two-way conversation between the user and computer.

With the remediation of media software in the computer, people become more and more creative and productive, and only people who are specified in some media field, but ordinary people can use these media software to create and produce. Before, people can only view paintings, listening to music, watching videos on computers. Now, they can use Photoshop to create their works or with a pencil and graphics tablet, they can draw in digital way, use Audition and other software to produce their music work, and use Premiere to create their videos. In the nearly future, different media system will combine and generate new media tools and new types of media2. Now, virtual reality technology is an example of combination of video, music, game and so on, and there will be more.

The second design step is Kay’s “simulation” in media3. In order to let media easy to be learned and controlled by people, simulation is an important step, and it bring more possibilities to more media to be invented and developed and made them more universal. That is, making an e-book looks much alike a real book, and let people feel accustomed when seeing it. It makes me recall the design principle affordance, or signifier. Until now, making the functions of how a new designed artifact works transparent to users is still so important. Notability, an application in apple store worked as note-taking software, is still working hard in order to design the interface more and more look like a real notebook paper, discover more functions, and make them more user-friendly. For example, it has a good classification system, so now users can put different kinds of notes, pdfs, or word documents into different folders. When users need to draw or take notes on certain document, they can set your own favorite pens with certain color, size of the brush, and stroke style.

Moreover, Kay did not just imitate paper but also create “magical paper”3. Kay did not forget take advantage of dynamic media in computers, combing simulation, made computers more and more multifunctional. The example mentioned in the article “text becomes a graph”4 is what we are doing in our daily digital life. People text with their friends and the content are not limited to words and sentences but also including photos, videos, music and so on. Since more and more new kinds of media come out these days, people can also add those things on the “graph”. For example, when people chatting with friends through instant chatting applications, they not only text, but also share a lot of other media on the “graph”, the interface of chatting, like music, voice message, photos, videos, and links from other applications. Maybe in the future, the graph will develop into a new media, and more other media can be shared and presented on the graph.



  1. Manovich, Lev. Software Takes Command. New York: Bloomsbury, 2013.
  2. Kay, Alan. “User interface: A personal view.” The art of human-computer interface design (1990): 191-207.
  3. Manovich, Lev., 64.
  4. Manovich, Lev., 71.

Computers as “Personal Dynamic Media”

Mary Margaret Herring

While doing this week’s reading, I was fascinated by Kay’s idea of computers as a “metamedium.” Kay envisioned computers as a medium for displaying other forms of media in a number of different formats. Based on this idea of computers as a metamedium, I would like to think about any unfulfilled or uncompleted design ideas today that Kay and others may have had in that first wave of design concepts.

To answer this question, I would like to start by reflecting on the notion of transparency in interfaces. When completing a task like writing an email, users are typically focused on the action of composing the message rather than navigating through the interface of their computer and email program. In this way, the interface should be transparent in allowing the user to focus on the task at hand. To encourage this transparency, many GUIs are designed to simulate other instances of completing the task in a way that users are already familiar with. To apply this to the email example, an email interface will likely try to have many other elements of word processors that the user is already familiar with to appeal to their past experience and make it easier for them to use. Manovich summarizes the importance of simulation in Kay and Goldberg’s design of the Dynabook by stating, “[i]n short, when we use computers as a general-purpose medium for simulation, we want this medium to be completely ‘transparent’” (2013, p.70). While Manovich was discussing a transparent interface for modeling data, it still seems clear that users will want interfaces to be relatively transparent when they are engaging in any sort of goal oriented behavior.

But, in Manovich’s book, it becomes clear that Kay recognized that computers as a metamedium can afford many things that the original medium could not. Take for example a PDF. When designing an interface for a PDF viewer, it is important to replicate the experience of reading a book or a printed sheet of paper. The experience of reading a book is achieved by allowing the user to highlight or annotate PDFs or jump to a page in the document. However, there are also certain functions that computers can perform that are not possible in books or printed sheets. For instance, users can search PDFs for certain words or phrases. PDFs can also contain hyperlinks that make it easier for users to find related information. Because computers as an interactive medium afford these actions, it would be a shame not to take advantage of these functions. For this reason, the computer as a metamedium should enable users to utilize these functions as much as possible.

In his history of modern computing, Irvine writes that “[m]any design innovators like Kay and Nelson continue to say that the computer revolution hasn’t yet begun” (2018, p. 12). I wholeheartedly agree with this statement. Kay envisioned the computer as “a ‘personal dynamic media’ which could be used for learning, discovery, and artistic creation” (Manovich, 2013, p. 61). Yet, somewhere along the line, it seems that the user became viewed as a passive consumer rather than an active agent. While simulating the design of technologies that users are familiar with in technology is a great way to get users to feel more comfortable using that device it also limits the number of things that users can do with that technology. It seems that there is a delicate balance between acquainting users with their digital environment and allowing them to see and use the additional functions that the technology affords as a metamedium.

A brief personal note on how this reading applies to my research interests:

I’m interested in addressing the problem of disinformation from a user interface standpoint. My main quandary is how we can redesign the way that social media displays news to prompt users to think critically about the source and content before hitting the like or share button. Nine times out of ten when I mention my research interests to someone, they say “well users are lazy and don’t want to think critically” or “social media encourages passive scrolling.” While there is merit to this view, I also think that there are dozens of opportunities to take advantage of the affordances of social media and redesign it in a way that makes users more critical while preserving the good qualities (e.g. interactivity, community building) functions of social media.

As of yet, I’m not sure what these design changes might be. But reading about Kay’s original vision of computers as ‘personal dynamic media’ was extremely exciting for me and made me hopeful that some of these solutions might exist.


Irvine, M. (2018). Computing with symbolic-cognitive interfaces for all media systems: Design concepts that enabled modern “interactive” “metamedia” computers. Unpublished Manuscript. 

Manovich, L. (2013). Software takes command. Bloomsbury.