Category Archives: Week 10

Interfaces: Bringing Man and Machine Closer

Many of the computer designs we take for granted today have had a vivid and imaginative intellectual history. The potential in a computer to do more than just big calculations was noticeable to a significant cohort. Engelbart clearly understood that computation is not just arithmetic calculation, but a machine that can generally work on a symbolic level. In fact, he claimed in his proposal of a hypothetical machine capable of augmenting human intellect, “Every person who does his thinking with symbolized concepts (whether in the form of the English language, pictographs, formal logic, or mathematics) should be able to benefit significantly.” Engelbart acutely drew out the limitations of both the machine and man when they work individually, and demonstrated that when their strengths are combined, it allows the man to work on a new level of efficiency and productivity. Moreover, he took a closer, investigative look at the nature of human cognition to explain the benefits of a computer more suitable to universal human needs. First, he clearly understood the limitation of humans to manage large amounts of information efficiently, which is a prerequisite in solving complex problems. I think that it’s not that humans cannot handle complexity, but rather that since memorizing large data sets are unnatural, it is hard. Secondly, he identified a flaw in the way the conventional pen and paper style of expressing and accumulating human thought constraints the natural symbol-structuring of the human thought process which is not in a serial fashion, but rather a sequential one.

If there were to emerge a “symbiotic association” between man and machine, there needed to be developed an interface between the two different systems so that man can essentially “communicate” or interact with the artifact. This was a significant step forward in computational thinking, moving from the conventional model of the man having a clear understanding about his work and instructing the computer what to do before hand. The Sketchpad was a radically new way of computing, allowing for the user to “talk to the computer graphically”. One of the key features in the “magic” of Sketchpad was a comprehensive memory storage system. The computer essentially translated the graphics as distinct objects to store them and their properties in a designated location. Another was the duplication or replication feature which afforded to make copies of the “masterpicture” (subpictures) which in turn allowed for much greater flexibility in problem solving (mistakes, changes). Since the symbol-structuring in the human mind was commonly understood to work in conceptual structures, visualization of concepts, Engelbart believed, was a great start in building a common ground between the artifact and man. Difference equations built into the program also facilitated dynamic interaction between the user and the computer, because both man and machine could cognitively operate (draw inferences) on the symbolic paradigm of math. “A mathematician is not a man who can readily manipulate figures; often he cannot. He is not even a man who can readily perform the transformations of equations by the use of calculus. He is primarily an individual who is skilled in the use of symbolic logic on a high plane” (Bush)


  1. Vannevar Bush, “As We May Think,” Atlantic, July, 1945.
  2. Engelbart, “Augmenting Human Intellect: A Conceptual Framework.” First published, 1962. As reprinted in The New Media Reader, edited by Noah Wardrip-Fruin and Nick Montfort, 93–108. Cambridge, MA: The MIT Press, 2003
  3. Ivan Sutherland, “Sketchpad: A Man-Machine Graphical Communication System” (1963)
  4. J. C. R. Licklider, “Man-Computer Symbiosis” (1960) | “The Computer as Communication Device” (1968)

What has been achieved and What hasn’t

The readings of this week take us back into the history of the development of computation and show us how where the original ideas about computational technologies we take for granted nowadays come from. In the article As We May Think, which is significant along the way, Vannevar Bush proposed many visions, most of which have been achieved and become common in our daily life.

Bush suggested that the size of substrate that held information would be reduced so that all the information in the Encyclopedia Britannica can be hold in match box size material.  For sure. My portable hard drive which weighs a little more that 100 grams and has a size of a soap case can hold 2 terabytes of information, which is approximately equivalent to all the information that books in the libraries of Georgetown University hold.

Bush also envisioned that a researcher would free his hands by talking to a computer to do recording, which has also came true for the time being when there is programs that recognize voices of human and enable speech input, e.g. Siri in iPhones, though they are far from perfect.

Perspicaciously, Bush predicted that computation would be a manipulation of symbols. This notion was elaborated more than a half century later in Computation is Symbol Manipulation by John Conery. In this article, Conery describes computation as a sequence of state transitions, and he defines states as a set of symbols. Other than that, Bush also alleged that new symbolism must precede the process of problem solving. He happened to be right again. Languages of programming are the prerequisite of development of any programs.

Bush thought the process of selecting information the people needed is difficult because human minds operated in association rather than in alphabetical order by which information is usually stored. To address this problem, he offered the solution of “building a trail” between two piece of information, which is part of what Hyperlinks do nowadays. We go even deeper. Other than alphabet, we tagged data with various metadata so that we can associate them together in different patterns and consult them conveniently.

However, not everything envisioned by Bush has been achieved. For example, he imagined that a direct path between the electric current in computer circuit and the biochemical electric current in human’s brain would be built. It would mean a lot for human, if this vision were to be achieved, for it would totally change the human computer interface, even our cognitive capacities and it would be a huge step toward artificial intelligence.  Other than that, even though we have now developed machine far powerful than Memex, we still cannot say that we fulfill Bush’s intention for such kind of machine–to transmit and review the results of research efficiently, because there are always too many of them. Scholars make painstaking efforts to keep up to date in his own field. That’s quite a paradox that when the problem also upgrades when the solution to it improves.

Week 10

It’s so interesting and thought-provoking to look back upon some original ideas of the technology we are so familiar with and take for granted today.

According to these historic records, computers, at the very beginning were designed to augment people’s abilities to coping with more complex problems in the reality. There two key features I notice that scientists commonly pay attention to in these articles. First, computers are delineated beyond the pre-formulated programming, but an “intuitively guided trial-and-error procedure”. In Lickliner’s article Man-Computer Symbiosis, he claimed that computers would free people from thinking through the problem before programming, with some logical structures and an trigger by human at the beginning.

Second, the concept related to “association” and “trail” is emphasized. In the article As We May Think, Bush realizes that at his time, the way computers using to select items from subclass to its subclass is ineptitude and inefficient — it’s not the way human beings use in their cognitive process. Inspired by this, Bush says rather than indexing, the association of thoughts and some intricate webs of trails should be applied to the design of computers. This kind of associations form a “library”, taking on an enlarged supplement of people’s memory. Moreover, it’s stronger since data stored in computer won’t be faded, as what in people’s memory will usually be if the association hasn’t been recalled for a long time.

These two ideas both indicate an intention to build an — what we have just discussed — extended mind of human beings. Compared to those pre-modern machines, computers in Lickliner’s blueprint are required to do more than pre-formulated programs, which means that what computers process will result in something unknown or even unexpected by its human manipulators. This design enables people to “off-load” not only memories, or to say “mental data” into an extended storage, but the procedure of process, the “intellectual” behavior of “thinking”, to computers. This idea is really a leap in the history of technology.

In that early times, the concept of “interface” in the design of computers is also phenomenal. On the level of computer science and technology, the interface makes computers become a “layer” underlying other interfaces we use, for examples words and images. The screen design of computers creates a meeting point between signs and the network of meaning that our society associates with them.

This kind of associations collected by the whole society also reflects on the affordance of interfaces. The good affordance of a design fulfill users’ expectations, and expectations themselves are results of users’ long-term habits and historic traditions of a society.

This has some similarities with how semiotic system works in people’s communication process. And in the article, all natural languages are covered under the definition of interface — the materials between ideas in mind and objects in external world. This is quite a fresh and fascinating idea and provides new ways to understand semiotic systems.

Manipulate 3D object on 2D interface

From the decrypting Turing Machine, Bush’s storing and consulting Memex, Licklider’s multi-access online interactive community, Sutherland’s sketchpad and light pen, Engelbart’s envisage of modern software, then Alan Kay’s mockup device, to finally, Steve Jobs’ epochal Ipad. This long, long, long list dates back how human work out step by step the logical function and the graphical interaction of the interfaces. This interface has been so long in my life that I’ve been taking for granted. This week’s reading helps me reexamine the magic this familiar black box has been doing so far.

In all those pioneers, Ivan Sutherland’s work is the one that surprises me most. I studied on the history of Disney and Pixar animation studio to notice that, before Pixar, Hollywood animation field is dominated by Disney by its perfect 2D hand-painting system called rotoscoping, which allowed the painter to draw on the base of real scene. But Pixar’s engineer Ed Catmull brought SOMEONE’s work into the film industry, built a cutting-edge software called RenderMan, which allowed painter to draw and design directly on the computer. This technology launched the world first 3D animation film Toy Story and set off the third revolution in the industry. And now, I know who that “SOMEONE” was. I thought it is Sutherland who allowed people to draw with, and on the computer, which is such an important invent that nearly laid the foundation of modern people’s tech habits. In Sutherland’s paper, he used the “light pen” to input semiotic graphic to the computer. Isn’t that light pen now, our fingers?

Talking about 3D, this seems to remain lots of paths to be realized in the future. CAD, Nuke, and the more common Adobe After Effects are three softwares I know who worked on the 3D function. In my own opinion, interfaces involving 3D functions are much more complex than the 2D processing mentioned by the readings so far. To manipulate a three-dimensional box on a two-dimensional board requires not only symbolic forms which constitutes and create interfaces, it also requires more application of human’s extended mind to work its magic. On the interfaces of the 3D display system, there is an “X-Y-Z Position Indicator” used to change your point of view to see the overall display of the subject. This indicator gave user an intuitive view of scene which can’t be seen by the real eyes. Using this indicator, click and move the mouse from left to right means to rotate the object clockwise on the Y-axis, down to up means to rotate on the X-axis, which is quite different from the X-Y Position Indicator. This indicator literally works as the “third eye” of human. But after all, these 3D operations can all be operated by the mouse.

Apart from this indicator which shows the producer’s view, there is another view in this 3D display system called camera. It is what will be rendered out, the spectator’s view. Taking this project for example, the view we now see is the producer’s view, from the left side of the real scene and can be altered by the “X-Y-Z Position Indicator”. However, when rendering this video out, the spectator’s view is one from the camera on the left bottom of the display.

BTW, the blue indicator means to move the camera backwards, in this case, towards the picture, instead of towards the back of the computer. Without proper indices, one can not understand the operation rule of this 3D system like they do in the 2D system.

Operating 3D object in 2D board is quite an interesting yet complex symbolic thing. Because the really operating system lack the Z-axis, we now can’t lift our mouse into the air to simulate 3D operation, hence the complexity. So to speak, some of the affordances are not intuitive enough like 2D system, so in the future, I’m looking forward to a more simplified interface than today’s. (Maybe we can lift our mouse/ fingers/ light pens in the air, who knows?)


Editing on the next day:

This morning I came into an interesting video Tilt Brush, showing how to operate 3D object on the 3D interfaces. In this case the tilt brush invented by google VR upgraded the “light pen” in Sutherland’s mind by endowing it the function to work in real 3D spaces. I was wrong before, we now can literally “lift our mouse into the air to simulate 3D operation.”