This is my first year in CCT. Before this CCT program, if you ask me what is a computer, I will answer that computer is a device that can carry out some arithmetic or logical operations. But at the end of 12 weeks of study of this course, I will answer that computer, is a computing artifact and an affordance of humanity’s cognition.

In these weeks, we go through the history of semiotics from C.S. Peirce’s theory of signs to the Saussure’s semiotic model, the history of meaning systems and symbolic representation from the earliest records of symbolic expression to computational language processing and artificial intelligence, the history of models of computation, the history of computing systems from memex by Vannevar Bush and the Dynabook by Alan Kay to the laptop and other computing devices today. After dipping into these theoretical backgrounds, I can see that my thinking of this world and human beings has been changed.

Humanity Memory V.S. Computer Storage

Nowadays, there have been major breakthroughs in the human brain understanding; some of them can directly change our daily life. However, media on human brain knowledge is far less than common sense. Common people, including me, have a lot of misconceptions about the human brain. The most popular one is the analogy between the human brain and the computing systems. This misunderstanding mainly comes from science fiction. Many science fiction works describe humanity’s memory as a data that can be erased and revised. For example, in the Matrix, people can learn martial arts or how to drive a helicopter by directly receiving codes.

Human’s memory is not recorded in the brain, but grows in the brain. The brain remembers things through neurons. There is an example: a patient had no pain nerves, which means that he can remain awake throughout the craniotomy. The doctor found that one of his nerves jumped when he saw the picture of Jennifer Aniston, and another one jumped when he saw the photo of Bill Clinton. The human brain arranges a specific nerve for each person he knows. In computing, the most semiconductor memory is organized into memory cells. It is easy to save a file on the hard disk, just change the arrangement of bits (0 or 1), but it is impossible to change people’s memory.

Computing Devices & Human Beings

After reading those science fiction works, I could not help but think of the day that the artificial intelligence will destroy the human world. Now, a lot of field people are working on the advantages of human beings relative to artificial intelligence. We have the unique human expectations of the future. We are the only species has the ability to understand the long-term future and make plans for it. In the book “The Future of the Mind”, the Japanese physicist, Kari Daishi, points out that the biggest difference between human beings and other species is that our brains understand the concept of time. So that the long-term interests will drive us to give up some short-term interests, and the collective development will drive us to give up some personal interests.

Of course, no one can predict the extent to which technology can be developed. Maybe one day, the machine can fool the human, and can even affect mankind.

However, after learning knowledge of semiotics, I see them from different angle, I regard these cognitive-symbolic artifacts as the extensions of human cognition. If you look at the history of computing systems, computing systems are getting smaller to smaller. At beginning, its size is about a room. In 1970s, we had the personal computers. Now, I have a laptop in my bag everyday, and we wear Google glass and all the other wearable devices. The next step is that they will be under our skins. So they become closer and more intimate to us, we are absorbing computers into us. In this society, we are bringing technologies to us, and they become parts of us. We are no longer independent from our technology. They change our identity of who we are. In the past, we thought ourselves ending by fingertips. After we are expanding ourselves with the help of those computing systems. Some people may concern about that the artificial intelligence and technologies may replace people from their jobs. It is true that some jobs will disappear, but it will create more new jobs for people. That is what we are doing, using technologies to create new things for people to do and further evolving to new creatures.

Let us go first with the development of human beings. There could be two clues to analyze the history of mankind. The first one is “extension”, or “ affordance”. According to Marshall McLuhan, the development of science and technology depends on the continuous substitution of men’s extension. When people afford moving on the wheels and rockets, we extend our legs; when people afford washing on washing machine, we extend our hands; when people afford watching on TV, we extend our eyes.

The second clue is “interaction”. Although interface, nowadays, “has a narrow range of meaning as a shared boundary across which two separate components of a computer system exchange information, ” as showed in the first Wikipedia’s entry of interface. From an etymological perspective, this word, in 1874, means “ a plane surface regarded as the common boundary of two bodies.” In its broadest sense and also the modern usage, this term, “interface is anything that connects two (or more) different systems across the boundaries of those systems. “

Basically, interface exists since the dawn of mankind. From the moment we use language to substitute the gesture. We can feel the interface between language system and gesture system. The sensory systems of human beings could interact with each other. From the moment we start to draw patterns on the pottery. We can perceive the interface between the aesthetic system and practical system. Outside systems which could fulfill different human needs could interact with each other. We then apply interface to the computational system. We also interact with other species: in the nomadism civilization, people interact with sheep, horses and dogs; in the agriculture civilization, people interact with rice and wheat; in the industrial civilization, people interact with oil and coal.And now, in the information era, with the help of WIMP (windows, icons, menus, and pointer devices), we can detect the interface between the human system and the computational system. Human beings and machine systems could interact. In the future, we will create more interfaces, for example, through recently the heated project “The Reality Editor”, we can read the interface between the computational system and the material system.

“The Reality Editor  is a new kind of tool for empowering you to connect and manipulate the functionality of physical objects. Just point the camera of your smartphone at an object built with the Open Hybrid platform and its invisible capabilities will become visible for you to edit.” Drag a virtual line from one object to another and create a new relationship between these objects. With this simplicity, you are able to master the entire scope of connected objects. This tool could help users to maximize their strength, such as spatial coordination, muscle memory and tool-making. For example, it could help you to turn off the light in your bedroom without letting you to stand up and walk to the switch.

The interactions between human and human, human and object, and human and information will decide how this society operates. In the next step, I think the computing may focus on the small things, the details. The critical details can make the difference between a friendly experience and traumatic anxiety. A problem cannot be solved with the unhandled details.  With the Internet, people are tagged. More specific targets mean more segments of the market, which requires better detail-optimization. 

Question:

What is the difference between the HCI and interaction design?

At least, I learned something connected with semiotics before, when I was an undergraduate student. But computer science, or computation, is a brand-new field to me. Before I touch this field, I felt it is really cool and challenging. It is like one of the greatest inventions and social changes has a great influence on the human beings. So I finished all the free course connected with Python on the codecademy website. After coding with my own hands, I felt computation is not harder than I thought. In Python, all the sentences quoted by “”” and “”” or led by // are symbols that mean things, since the computer will automatically ignore these words. Besides, other sentences without quotation mark or slashes can be treated as the symbols that mean things, but they can only be read in a particular form.

According to Prof. Dasgupta,  computer science is an artifact. Compared with natural science, which devotes itself to figure out the existing functions of the organs, fossils, or oxygen, computer science, as an artifact, whose purpose and functions are all defined by human beings, is decodable. As an outsider, I really hope coding and computation can be as easy as possible, so I cannot help but ask why computers cannot use natural language to code. English may cannot perform this task, but French or German which are more rigorous than English still could not meet the requirements. There is an impassable gulf between natural language and computer language. I think the root of this gulf is that there is a huge difference between how do human beings think and how do computers understand. People use language of thought to communicate, but computers are trying to understand all the descriptive language.

But we can see the shrinking gap between computer language and natural language from the history of coding. People use 0 and 1 as the basic component of coding at first. It is very hard to memorize, to check, and read by human beings. But now we have a java script, python and so many other computer languages. They are really similar to the natural language, except for the fixed form and arrangement.

With the help of computer language, not only we could code, we can also have . The core of computational thinking is the problem-solving skill, which is reformulate a difficult problem into one we know how to solve (Jeannette M. Wing.) For example, IOS system is a relatively closed system, compared with Android. Yes, it has some open-source bits, but the vast majority of the operating system are closed-source. There is no real possibility of changing the settings by an application. So, how do some music apps realize the function that can display the scrolling lyrics on the screen when the phone is locked. Even if the iTunes and Music cannot implement function. Those computers reformulate this hard question into one they know how to solve. They photoshopped every lyric on the same posters slide by slide, and change the posters one by one every several seconds. To a user, it looks like the poster never changes, only the lyrics are rolling. I think this is really a smart practice. If we could apply this computational thinking into daily life, it still could solve some hard problems.

Questions:

Why they cannot define a same framework or arrangement of different computer language?

References:

[1] Denning, Peter J., and Craig H. Martell. Great Principles of Computing (Cambridge, MA: MIT Press, 2015).

[2] Wing, Jeannette. “Computational Thinking.” Communications of the ACM 49, no. 3 (March 2006): 33–35

[3] Subrata Dasgupta, It Began with Babbage: The Genesis of Computer Science. Oxford, UK: Oxford University Press, 2014. Excerpts: Prologue and Chapter 1.

“Sign” , according to C. S. Peirce, is a static unit or individual representational form that can be interpreted as part of a collective sign system. Signs can be signs are because they can magnify the particular feature of a particular object. For instance, in French, a “mother-in-law” is called as “ la belle-mère.” “Belle” means beautiful and pretty. “mère” means mother. It is also true in China, we call a mother-in-law as “qin jia mu”, here, qin means close and related by blood. We can see that we have to emphasize that we are soooo close because we are not that close in reality. Language, here as a symbol, functions in the daily life.

Thanks to the internet, people in this world closely connected to each other as never before. The online version of communication can get rid of the facial expressions, gestures and tones, it is easier to be interpreted. So, how a text message, an email message, or social media message works? What kinds of communication acts understood by communicators are involved?
In The Information Paradox, Shannon’s words were cited to explain the first theoretical model of a mathematical theory of communication. “ A source sends a message. An encoder generates a distinct signal for the message, as prescribed in a code book. The channel is the medium that carries signals from the source to the receiver. a decoder on the receiver end converts the signals back to their original form, using the same code book, and the message has arrived.”

In the first step, the encoder generates a distinct signal for the message. Senders play a decisive role in a communication action. The sign can only be a sign when it is interpreted by a sender in a particular way. So, if the senders’ intention is unknown, this sign cannot be interpreted. A German man has taught his dog Adolf to give a nazi salute when hearing “Adolf sit, give me the salute.” In this case, the situation is much more complicate. The sender of this sign is only a trained dog who cannot understand the meaning of this gesture, so the dog is not violating Germany’s anti-Nazi laws. The sender also decides the way of encoding. A signal can be encoded in various ways, but there exist better options of one idea. A lot of outside factors can matter. For example, If a signal is highly required by an environment, this signal can be sent in many ways. When I want to answer a yes-no question, I can say “yes”, “sure”, “correct”, “of course”, etc. But If I want to mention a term, I can only use “algebra”, or “physics”.

In the last step, a decoder converts the signals back to their original form. The process of decoding is answering a series of yes-no questions. Although the answer of each question can only be 0 or 1, but the probabilities of 1 and 0 are different. I have to distinguish “me” and “not me”, and then the “is a person” -“ is not a person” question. The “is a tiger”-“not a tiger” question. We can see that we have to cut a sign into several questions and then decode them respectively. A receiver has to choose the same code book of interpreting the codes. Although we do have some conventions, they cannot cover every aspect, detail, and trivia of life. That could be the first reason the receiver may not get the sender’s meaning.

Another reason that may distract the receiver is the noise exists in the medium that carries signals from the source to the receiver. Sometimes we can abstract the thing we talk about from the real world. When we sit in a theatre, we can tell the difference between on stage and in reality. That is because there is a intangible wall between you and the play. But, usually, we recognize this world by taking them as an entity.

I can never make sure that the text message I sent can be full understood by you.

Nazi salute. (2016, October 8). In Wikipedia, The Free Encyclopedia. Retrieved 11:29, October 8, 2016, from https://en.wikipedia.org/w/index.php?title=Nazi_salute&oldid=743191847
Irvine, Martin. “Introduction to the Technical Theory of Information.”
Denning, Peter J., and Tim Bell. 2012. “The Information Paradox.” American Scientist 100 (6): 470–77.
Hall, Stuart. “Encoding, Decoding.” In The Cultural Studies Reader, edited by Simon During, 507-17. London; New York: Routledge, 1993.
I have two questions:

1.Do we have some expression in particular languages cannot be translated or deciphered by other languages? I mean, if you can explain a 15-letter English word by 1000 Chinese words, you can still translate it.

2. Why different languages can be unbalance? In Chinese, we use different words to indicate elder cousins, younger cousins, male cousins, female cousins, maternal cousins, and paternal cousins, but in English, there is only one word: cousin. And in French, they use “quatre-vingt-dix-neuf” to indicate 99, quatre means 4, vingt means 20, dix means 10, neuf means 9. It is really like a formula: 99=4*20+10+9. Why they don’t have the words like ninety.

Pop art is an art movement that emerged in the mid-1950s in Britain and the late 1950s in the United States. In pop art, Material is sometimes visually removed from its known context, isolated, and combined with unrelated material. (Wikipedia) Pop art is the abbreviated version of popular art, since one of its aims is using images of popular culture in the art to narrate, parody, or to satirize.

These are examples of famous pop art.

According to C.S. Peirce, human thought is based on signs in symbol systems, each of which have a structure of material and cognitive relations. Language can be the most obvious symbol. But, actually, human beings cannot only rely on the language to express their feelings, since we are facing such a complex world. We can count on other symbols, visual symbols must be one of the popular ones. Compared with language, the components of visual symbols can be more arbitrary, creative, and abundant.

Although some linguist and philosopher, such as Susanne Langer, denied applying the laws of syntax that govern language on the analysis of articulation. I still think there exist a lot of similarities underneath the differences. I, here, use a famous Asian artist: Kusama Yayoi and her arts as an example to interperate.

The parallel constraint-based architecture of interpreting the theory of processing is used here to analyze the art work.Pretty same as the figure of linguistics, the processing of art can also be divided into three parallel structures: the spots structures, the syntactic structures, and the conceptual structures. （Jackendoff）

The Spots Structures
Rather than the lexicon of linguistics, we can see clearly that Kusama Yayoi’s art works are composed by thousands of spots. They are the basic components of her art works. She, herself also mentioned that even earth is one of the millions of spots, when you finished one sport, you have already finished the universe and this world. In her art works, those spots are just like cells, or molecules, which are the origin and basis of the life.

The Syntactic Structures

The combination of the spots also has its own syntax, so that to express. This syntax can also branch outward to color and size of the spots.

We can also see another obvious feature of her work, the infinite. She uses millions of spots to change the original formation of a stuff, and creates a continuing relation between this stuff and the outer space. So when facing this art work, it is hard for the audience to tell the reality and the world she created.

The Conceptual Structures

Pumpkin is one of her representative works. To Kusama Yayoi, pumpkin is an index of food. She had experienced the World War 2, pumpkins are the only food for them to survive through the famine. From then, pumpkin becomes her favorite vegetable. She sees pumpkin as the most humorous vegetables, and the flexible one as well. So in her art works, we can see the pumpkin is soft instead of having hard skin.

[1] Daniel Chandler, Semiotics: The Basics. 2nd ed. New York, NY: Routledge, 2007. Excerpts.

[2 Ray Jackendoff, Foundations of Language, selections on the “Parallel Architecture” model of language as a combinatorial system. Chap. 5.5, pp. 123-128; Chap. 7, pp. 196-200.

What is Language

Before the reading, if someone ask me what is language, I will tell him that language is the ability to use a complex system of communication to express feelings, and languages are totally different from each other and arbitrary. There exist 5000-7000 kinds of languages in the world now. But, after the reading, I will say everything could be language, there must be millions of ways to express thoughts. Not only the languages, spoken or signed, can be language; all those auditory, visual, or tactile stimuli to the brain can also be languages. “It is a abundantly clear that the brain is organized in terms of numerous interacting areas that together determine our experience of the world and our intentions to act.” (Jackendoff, 130)

Brain is like a CPU, the input are all kinds of stimuli, and the output depends on the things you familiar with. All man are born equal, yes,it is also true to a linguist. A 8-month-old baby can have the ability to identify all the different existing pronunciations, there are more than 800 pronunciations! But later, if his brain stop being stimulated by the particular sound, he will lose this ability. A Chinese-Speaker cannot pronunciate /l/ sound and /rrrr/ sound, without training. A Japanese-speaker cannot tell the differences between /r/ and /l/, my english-speaking classmates cannot pronunciate my name……

So long as there is an input, stimuli to the brain, the brain can help us to feel this world.

Here is also another interesting device. BrainPort device, created by Wisconsin-based Wicab, Inc., consists of a pair of sunglasses with an embedded camera, a hand-held CPU, and an electrode-laden paddle which sits on the user’s tongue which is placed on the user’s tongue. The camera will capture the visual information, transfer them into electric impulses and stimulate the brain through the tongue.

Besides, output methods can be various. I think we all had inner voice sometimes, which means we can think in words. Some people may curious about how deaf people think in words. Actually, blind people’s dream may contain those feelings, so deaf people can think in those gestures and signs. So it doesn’t matter which language is your mother tongue, we all think in f-language, or f-mind.

Language can exist in everywhere.

Darwin and Wallace had described natural selection as “ a struggle for existence”, but to human beings, it is better to be described as “ a struggle for control”. Human beings not only have the response to the potential mates, the preys and predators, but we also respond and have the motivation to control other species. So, although brains are the parts of the body, just like animals, they had evolved to store information about an external reality, and were available to be utilized to generate effective behaviors on, and syntactically coherent statement, on the world. (Barrett, 14)

I want to illustrate examples here to demonstrate the theories mentioned in the papers to articulate my understanding. I don’t know whether they are correct……..

To explain the evolutionary process of human cognition, Renfrew proposed four evolutionary stages. The first stage, episodic culture, the culture and capabilities founding our closest living primate relatives, is the transition from hominids to Homo erectus. This is maybe the beginnings of self-awareness. The mirror test is an attempt to determine whether a non-human animal possesses the ability of self-recognition. In this test, an animal is anesthetized and then marked on an area of the body of the animal cannot normally see. When the animal recovers from the anaesthetic, a mirror is provided. If this animal can touch or investigate this mark, it is taken as an indication that this animal perceives the reflected image as itself, has the primitive self-awareness (Wikipedia).

The second stage, mimetic culture, the culture to produce conscious (languages are not excluded), is an evolutionary step beyond episodic culture and a foundation for symbolic representation and language. Mimetic skill requires an individual has the skill to memorize, define and rehearse the body’s movements in a systematic way. There are some examples of Homo erectus’s behaviors, such as some rituals, dances and marks on clay tablets. Although the mimesis are not language, but they do allow us to better communicate.

The third transition, linguistic or mythic culture, the characteristic of early Homo sapiens. Myth, according to Donald, is the primary function of language in a culture dominated by linguistic cognition. In this stage, people can synthesize symbolic art and symbolic language. The symbolic pictures, such as the one founding southern European caves,were used to explore and develop the mythic ideas.

The fourth transition, external symbolic storage employing symbolic material culture, the characteristic of early agrarian societies with permanent settlements, monuments and valuables. With the development of writing, people’s memory was no longer restricted to the bounds of the body, but could be held in external storage systems.

On the one hand, we benefit from this evolution. From the moment when we curved elaborate marks on clay tablets, we know we can handle those symbols. We can communicate with each other efficiently.

On the other hand, we are also hunted by this brain evolution. The development of the brain provides human beings thousands of mood. Those sad and jealous feeling always hurt us.

Kate Wong, “The Morning of the Modern Mind: Symbolic Culture.” Scientific American 292, no. 6 (June 2005): 86-95.

Merlin Donald, “Evolutionary Origins of the Social Brain,” from Social Brain Matters: Stances on the Neurobiology of Social Cognition, ed. Oscar Vilarroya, et al. Amsterdam: Rodophi, 2007.

Terrence W. Deacon, The Symbolic Species: The Co-evolution of Language and the Brain. New                York, NY: W. W. Norton & Company, 1998.

Colin Renfrew, “Mind and Matter: Cognitive Archaeology and External Symbolic Storage.” In Cognition and Material Culture: The Archaeology of Symbolic Storage, edited by Colin Renfrew, 1-6. Cambridge, UK: McDonald Institute for Archaeological Research, 1999.

John C. Barrett, “The Archaeology of Mind: It’s Not What You Think.” Cambridge Archaeological Journal 23, no. 01 (2013): 1-17.

Wikipedia contributors. “Mirror test.” Wikipedia, The Free Encyclopedia. Wikipedia, The Free Encyclopedia, 6 Sep. 2016. Web. 15 Sep. 2016.