Category Archives: Week 12

The Big Picture: Cognitive-Semiotic Theory of Computation

Computers are designed with various limited purposes but almost entirely with the underpinning goal to “extend and distribute human collective cognition and agency”. To my best knowledge now, to do something computationally (thinking, arithmetic calculation, etc.) means that you are sequentially representing information multiple times until a specific goal is achieved. Seeing the big picture requires one to analyze the computational process against Peirce’s semiotic triadic theory of signs, to uncover just how any intelligence that is acquired , exhibited or displayed depends on the symbol systems that it is observably associated with, be it a human being with a physical biological body in synergy or a computer with physical material components organized to carry out operations in tandem.

The analogy drawn between information and light in the introduction of Sarbo’s book reveals a profound insight into the nature of information. Computationally speaking, any input (Peirce’s representant) as a set of signs can be broken down, re-represented as more signs and brought back to its original form to retain its intended meaning original condition (Peirce’s object) for interpretation or whatever use. However, this process also depends on the state (Peirce’s interpretant) of the observer, which in this case may be an internal state of the computer itself. As Sabro discusses in length in the later part of the introduction on knowledge representation in human meaning systems, he says “through interactions, we experience, via learning we know reality”. At first this did not seem too different from the way computers learn through the model of machine learning. However there is a slight modification to be made- “through data the machine experiences, via learning they know reality”. Just as Sabro concludes that the knowledge should come from the interactions that are “forced upon us”, machines acquire knowledge through the data that is forced into them. However that may be again just be a matter of time until technology is developed to afford such human-like interactions from which the physical machine can store data from. This fundamental comparison of knowledge representation between humans and computers proves again the concept of the computer as designed to extend human cognition.

Exposing myself to such an integrated method of thinking about computation through its semiotic-cognitive structures better informs my research interests. First and foremost, although I’ve been impatient in my pursuit to learn more about quantum computing, it does not intimidate or repulse me anymore as I feel better equipped to begin with asking the right (“big picture”) questions. For instance, I think it useful to primarily view quantum computing as an information process and synthesize what functions it has so far already proved to operate in the context of an extension of human collective cognition and agency.


Peter Wegner, “Why Interaction Is More Powerful Than Algorithms.” Communications of the ACM 40, no. 5 (May 1, 1997): 80–91.

Martin Irvine, “Introduction: Toward a Synthesis of Our Studies on Semiotics, Artefacts, and Computing.

Herbert A. Simon,  The Sciences of the Artificial. Cambridge, MA: MIT Press, 1996. Excerpt (11 pp.).

Janos J. Sarbo, Józef I. Farkas, and Auke J. J. van Breeman. Knowledge in Formation: A Computational Theory of Interpretation. Heidelberg; New York: Springer, 2011. Selections. 



Semiotics: a way of inquiry into CCT

From what we have learnt thus far this semester, I think it’s adequate to say that semiotics is a way of inquiry into all three components of our program: communication, culture and technology.

In the area of communication, we can find that people trade their ideas and thoughts by employing shared sign systems within a certain community. Along the course of the development of human beings’ communication, we have made some major inventions that effectively enhance the efficiency of communications–languages, paper, printing to radio, telephone, TV and computer. But the truth is the vibration of particles in air itself does not convey any meanings, nor does inks on a  paper or electricity in a circuit. It is the shared sign systems, e.g. language, gestures, established by conventions among communities of people that convey meanings which envelop people’s ideas and thoughts. That’s what is missed in signal-code-transmission model of information brought by Shannon in terms of meaning transmission. Since we are symbolic species, the sign systems can never be left out of the processes of communication.

Sign systems can only function when a group or community of people have a shared, collective and intersubjecitive cognitive encyclopedia. A distinct culture emerges when enough shared sign systems are employed, like a distinct language, a certain way of living, style of painting, architecture and so on. If we look around, we will find a lot of signs that embody the culture we are in. If one doesn’t possess the collective knowledge necessary for “decoding” the signs, which is the case when a foreigner first come to a country, he will encounter with what is called “culture shock”.

It seems to me that technologies, especially the information and communication technologies, are usually intended to augment the symbolic capacities of human beings. The most significant example of this is undoubtedly computer. Computers allow people to offload some of the symbolic-processes and they have become a part of their users’ extended cognition. For example, people don’t have to memorize everything, they can simply store it into computers, just as they used to take something down on papers or carve it onto the wall of a cave. This example shows that computers also belongs to the symbolic continuum of technologies which can be inferred from the revised system of cognitive phases proposed by Renfrew. The way we design the technological artifacts have everything to do with our symbolic capacity, which can be seen from the the four key affordances of digital interactive interfaces explained by Murray and the concept of “metamedia” brought up by Kay. Moreover, the way we wield the technologies are also symbolic or conventional, e.g. programming language.

In the semiotic point of view, we are surrounded by signs. Using this way of inquiry may help us find something that can’t be found from other point of view.

the Way We Become Smarter and the Future


In Murray’s book Inventing the Medium: Principles of Interaction Design as a Cultural Practice, she delineates the development of human beings with some nodes of significant inventors in history: from language, to writing system, to printing, to the computer technology in our times. People may have different opinions on the most important inventors from different definitions and aspects, but these four, at least from my perspective, are definitely the most important steps in development of human’s cognitive capacity.

When we look back upon the history, now it’s quite clear that what certain inventor at that time — language, writing, and printing — affected and altered the following path of our whole species. But it remains unclear for now that how this time the computer, computation and all the technology related will eventually change our cognitive capacity, since we ourselves are still in this current process of being altered.

However we have some consciousness now, and can make some anticipations as well, since such kind of changes never happen overnight. 

Obviously, all means mentioned before are a kind of extended mind beyond human capacity. Language helps us think about the past and future, writing enables more materials and thoughts to be “tokenized” and passed through generations instead of drifting away after several bardic works, and printing make the copies proliferate, which brings more opportunities for great minds who would disappear physically but to be mentally eternalized. In terms of computer, they were initially designed as an extended minds for doing powerful and complex calculations with pre-designed algorithms.

In Wegner’s and Denning’s articles, both of them are trying to figure out precise definition of computation and the study of it, by reflecting on what has changed and developed during their times after Turing. One concept they both pay great attention to is the “interactivity” of computers.

In his article Interaction More Powerful than Algorithms, Wegner distinguishes the “interaction” from algorithms by emphasizing its ability to learn from and adopt to the environment and response to external unfolding events of time. Similarly, Denning also indicates such features of interactive systems and further emphasizes the “incompleteness” and “uncertainty” of interactive machines. These arguments brings a new understanding of computer’s role of our extended minds. With the interactivity, computers are no longer only unprecedentedly powerful weapons to tackle tough maths calculations, or unprecedentedly huge libraries to store information, they enable human beings also take part in these magical process.

Interactive system makes computers mobile phones “smarter”, so as it did human beings. In our times people can approach data so easily thousand times in amount than people 10,000 years ago. New data are processed by human’s cognition and then become information and meanings among a group or the whole society, so new meanings are tokenized, becoming patterns that people can recognize and use to generate more meanings. In fact this process continues to happen since the moment human beings gain the cognitive capacities, but the availability of data today accelerate this process. On this level, we human beings are “smarter” than ever.

So what’s gonna happen in the future? To which end will computer this “extended mind” extend , or alter our inner-mind? According to some researches in our era, almost everything in the world is about data and interactive “programs”. Some biologists say people are just tons of data caused by DNA, whose activities are restrictedly decided by certain rule and algorithm, as well as some mutations stimulated by natural conditions. Some neuroscientists claim that people have no Free Will at all. Our emotions, behaviors and consciousness are totally the result of data transitions between nerve cells and biologic response-mechanism to the environment. Some economists and sociologists also model the social activities and humanity events as data streams and their effects with each others.

It seems quite a paradox — data are only meaningful when people process them with cognitive capacities, but the genitive capacities themselves are claimed to be nothing more than data and biologic algorithms.

What’s in the future? It’s really hard to say.


Try to Understand Interactive Advertising with Interaction Design

Interactive Advertising: The kinds of advertising that use online or offline interactive media to communicate with consumers and to promote products, brands, services, and public service announcements, corporate or political groups. (Interactive Advertising. Wikipedia, 2017.9.26.

One case for myself to work through: Misereor PlaCard, The Social Swipe Advertising

As a Symbolic Practice

At first, interactive advertisings seem nothing like IIM, Interactive-identity machines, which output their input immediately without transforming it (Wegner. Why Interaction Is More Powerful Than Algorithm): it is interactive, yet the outcome seems not the direct copy of the input, but the programmed respond after algorithms. People “talk” to the machine through all sorts of interfaces and the machine give different feedbacks that are corresponding. Even though different input gets different respond, the respond is far different from the input. Then it occurs to me that this is exactly where the symbolic practice joins the game.

When one interacts with the machine, all they do can be interpreted as symbolic practices. The card-swiping gesture is not just a metaphor of freeing & feeding, in the computer programming language, it refers to money donating (an input). And the images of people get freed & fed is not just a metaphor of someone somewhere is helped by your donation, it refers to money donated (an output). In fact, this algorithm is hiding under the interactive interfaces so the machine can be transparent and intuitive to human. This works to both the main physical symbol systems: to computer, the input (money donating) = the output (money donated); to human mind/ brain, the input (cutting ropes & bread) lead to the immediate respond (rope & bread being cut).

As a Cultural Practice

Continued from the preceding part, if the interactive machine IIM just output their input immediately, why do people think otherwise? Why do people feel like by swiping card, they directly help people in need of that?

Successful interactive advertising must be transparent, which means to provide immediate feedback. However, how to let customers see the feedback as soon as they interact require precise and creative design. All the interactive advertisings are desperate to realize that, either by using extremely simplified affordances, or by conducting it symbolically in the shared culture. This case uses the latter way, by endowed the action of card swiping and the images of rope & bread being cut with symbolic meaning. Because of the shared culture between the designer and the customer, these symbolic references are processed completely in human mind, unconsciously, giving the illusion of transparent.

Successful interactive advertising should also be intuitive. In my own point of view, it is more vital than transparency. In inventing the Medium: Principles of Interaction Design as a Cultural Practice, Murray indicates the intuitive design should draw on our unconscious expectations about how things behave and expectations from experiences and subconscious idea of the world. Which means interactive design will only work if the machine responds as how people expect. In this case, people are unconsciously hoping that by swiping a card in a way of using a knife, there should be some responds to simulate the situation after knife using. And the design meets their expectations.

(What surprised me is that people are always surprised, even though it’s the exact respond they are waiting.)

Janet Murray also mentioned that it should provoke unique response so people can be able to distinguish among many other possible conventions. However, this is what makes me very interested. Cause in the field of advertising, this ambiguity, or pun, is exactly vital. It relies on people to make different references with different possible conventions. Swiping card is a gesture of paying as in everyday experiences, in this case it also is a gesture of cutting in a way of using knife. Both conventions are processed. Seems against the principle of interaction design… so I honestly don’t know how to explain this part.

TO CONCLUDE, consciously and subconsciously, the order of this interactive process which taken place in the extended mind is:

  1. Card swiping—Cutting the rope & bread—Money donating—Money donated—Rope & Bread being cut—People are helped.
  2. human action—the symbolization of this human action—input—output—the symbolization of the respond—the real respond to human action

Both line 1 and line 2 are the progress of how human interact with cognitive-symbolic artifacts. The collective process it is.