Category Archives: Week 4: Symbolic Cognition

“if you take care of the syntax…”

Recently I have been chewing on the concept of the mind as software and as thought as computational expression, so encountering these ideas in the Clark reading was quite captivating.

A week or so ago, I actually wrote this poem as a means of working through my own thoughts as a method of computation.


Machines can miss the meaning in messages. Semantics can complicate computational expression. Maybe I’m like a machine.

Input, output. Something doesn’t add up.

1 + 1 should equal 2, but I’m missing something.

All I’m getting is 1.9999999.

This week’s section theories on symbolic cognition are quite different from the models of communication that we learned about during the first week. The fact that cognition is being taken into account alone symbolizes an energetic shift away from the lexicon of the words we use and instead, the reasoning that goes into choosing those words, and how we as humans make meaning of them. This week’s readings begin to uncover the process of understanding at a level beyond the basic “sender/receiver” model in which the sender encodes and the receiver decodes. Now we are examining how exactly a receiver would decode a message, based on his or her conceptions of the words themselves.

The notion of “formal logics,” described in Clark’s reading, stood out to me as a huge deviation from the original, non-interdisciplinary communication theories, as it describes how meaning can be transferred from person to person without being understood. The example he gives is one of following the instructions to build a bookshelf. Thinking about the concept more, it reminds me a lot of legalistic interpretations of religion, and how people are prone to following rules without understanding the meaning behind them. In medieval times, the Catholic church took advantage of this phenomenon, because as Clark writes, formal logics are advantageous in that the end goal is reached (constructing a building, paying indulgences, etc.) even without the person having an understanding of the symbols embedded within the message conveyed.

I really enjoyed the examples of conceptual metaphors that we read–the relation of love, life, and career to the concept of a journey. Lakoff writes about the difference of contemporary communication theory and the cornerstone theories–that contemporary theories incorporate the importance of the image as a symbol and as a retainer of meaning.

Overall, it makes absolute sense that we as scholars would incorporate cognitive studies into our analysis of communication as it relates to meaning making.


George Lakoff, “Conceptual Metaphor.” Excerpt from Geeraerts, Dirk, ed. Cognitive Linguistics: Basic Readings. Berlin: Mouton de Gruyter, 2006.

Andy Clark, Mindware: An Introduction to the Philosophy of Cognitive Science. New York: Oxford University Press, 2001. (Excerpts from Introduction, Chapters1-2)

Power of Cognition

Storytelling is vital to human cognition in making sense of world and reality that we construct in living our everyday lives. The production of language and symbolic meaning-making in film has many layers from production, mediation, communication within the film and outwards to the audience. In Mindware, Clark presents the mind as a “meat machine” and as a biological computer system that is separated from all other things on this planet by self-awareness and thoughts; our meta flow of thoughts about thoughts and our reason-respecting flow of actions are structured in such a systematic way that we have the ability to watch and comprehend commentary on the degradation of human communication and acknowledge “languages” that we don’t understand. Throughout the readings, I was reminded of the Disney-Pixar film “Wall-E” which can be evaluated by it’s combinatorial meaning systems, grammar and distributed cognition both in it’s creation as a computer animated production and in it’s story and characters. As an example, I will use the clip below to demonstrate how people make sense of the film in it’s current form and how it showcases the three different instances of human symbolism and meaning.

Two major points are to be made of this scene:

1. Wall-E and his robot counterparts do not speak English except for names, yet their beeps and non-verbals are familiar and continue the narrative.   

2. There is obvious meaning to the depiction of humans living cognitively through technology, a warning and foreboding of the loss of “real” human communication system and cognition of the collective group.    

The film follows the life of an A.I that has emotions and “falls in love” with another machine, Eve. The story does not have include a lot of human speaking until Wall-E encounters them on a spaceship. What is intriguing about the movie is the way it moves the story along , knowing or at least expecting that the audience can follow based on verbal and social cues associated with human grammar. The ability to manipulate the movements and “expressions” of the characters through CGI help to make the robots seem “human” with their reasoning and expressive language capabilities despite their machine like experience. Their actions and emotions make sense to us as we follow them on the screen because it plays out a basic narrative that we are all familiar with, boy                                                               meets girl, girl is “kidnapped,” boy goes to save girl. Intriguingly, even though the robots are clearly machines, it is easier to assign them gender roles, provided by the cognitive stimuli in the form of symbolic human tones and actions, to make the story more meaningful and follow the complex and logical structure of the human through process. Or in this case, the belief in lack of logic when involving love. In this sense, and in many other films, this narrative is a form of grammar where the viewer knows the basic story, but as it is reformatted, are able to engage with the people, or in this case robots, based on the affordances of the environment around them.

The second part of the film focuses on Wall-E’s interaction with the humans who live on flying chairs, communicating right next to each other via futuristic skype-like screens rather than face-to-face. This human “dystopia” presents a look into the effects of distributed cognition. Wall-E’s  interactions on the abandoned Earth are mainly with non-living external artifacts, excluding his pet cockroach. He instead watches movies on TV and exhibits a sense of loneliness from lack of communication with others. When he encounters Eve, and later the humans, his cognitive environment changes. Additionally, the humans display an interesting example of the idea of distributed cognition as an interaction between human and technologies and social organization itself as a form of cognitive architecture. (Hollan, Hutchins, Kirsh) Humans have entered a state in which technology has taken over many of the cognitive actions that were once performed face-to-face and mediating all relationships through technology. It is a more extreme example of what many people are arguing today about how technology is rewiring our brain and how we perceive the world and relationships. The entire culture of social practices and interaction with external artifacts has changed in the community of humans in the film, with Wall-E being  more cognitively able than the meat machine of the human brain.

On a side note, this short blog on “The New Yorker” looks into creating a simulation of the human brain in computers: The Brain in the Machine

Andy Clark, Mindware: An Introduction to the Philosophy of Cognitive Science. New York: Oxford University Press, 2001.

Hollan, James, Edwin Hutchins, and David Kirsh. “Distributed Cognition: Toward a New Foundation for Human-computer Interaction Research.” ACM Transactions, Computer-Human Interaction 7, no. 2 (June 2000): 174-196.

Zhang, Jiajie, and Vimla L. Patel. “Distributed Cognition, Representation, and Affordance.” Pragmatics & Cognition 14, no. 2 (July 2006): 333-341.







Language in Context: evolutionary questions in Deacon’s book, The Symbolic Species

This week, our readings expanded on the puzzles about language last week’s readings posed, to connect language to cognition more broadly.  Right away, something in this week’s readings resonated with me more than last week’s.  It seems that cognitive linguists are trying to put meaning and context back into the study of language.  Syntax, for cognitive linguists, is not some sort of independent system formed in isolation and following its own logic, as it sometimes seems Chomsky et al. are implying, rather, syntax ties into evolutionary forces and social forces.

Thoughts on Terrence Deacon’s, The Symbolic Species

Last week in my post, I brought up the film Project Nim. I explained that I felt that scientists were missing a lot by insisting that Nim showed no ability to learn language because he did not learn syntax.  Nim was clearly a sophisticated communicator but the scientists did not deem this interesting.  In his book The Symbolic Species, Terrence Deacon suggests some rich new ways of thinking about non-syntax communication (including the communication of animals, and non-verbal human communication).  His conceptions give me a structure with which to critique the anthropocentric thinking of Chomsky et al. in a way that really resonates with me.

First of all, he reinserts evolution into the study of language.  Evolution is not teleological, he reminds readers.  Language is not some superior, end destination, that any species headed in the right direction will eventually achieve.  Rather, it is an adaptation to a particular environmental niche, evolved by humans for very specific reasons, to solve very specific evolutionary problems, just like bats evolved eco-location to catch bugs, or turtles evolved shells to be safe from predators.

The next thing he does, is reinsert our thinking about language into broader thinking about communication, and he argues against the misconception that things like human body-language or animal communication systems are comparable structures to language.  “Of no other form of communication is it legitimate to say that ‘language is a more complicated version of that.’…. Nonhuman forms of communication are something quite different from language… [and] comparison is misguided.”  This strikes me as a much more effective way of thinking about non-human communication, because it recognizes its complexity, draws connections to rich human non-verbal communication rather than to human language, and allows us to analyze these two systems (verbal and non-verbal communication) without expecting them to act like each other.

I love the phrase Deacon uses to critique how some linguists have downplayed the role of evolution in language development: “the hopeful monster theory.”  This theory essentially posits that it was some sort of one-shot mutation that created full-blown syntactic language out of previously unintelligent humanoids.  This type of theory does not need to address the messy questions of the context-specific adaptivity of language.  For Deacon, the main mystery underlying language is its uniqueness to humans, and not because it is too complex for other animals to master, but because it is fundamentally, structurally different.  He phrases his puzzle thus: “despite the intelligence of other species and the fact that they engage in communicative behaviors that are as complex in other ways as a simple language might be, no other language system exists.” And the fundamental structure that makes language different, is its symbolic reference, in other words, that words refer symbolically to each other and are primarily defined by their interactions with other words, rather than any object or action in the real world.


External Representation in Political Caricatures

Pictorial expression is a powerful kind of external representation, which in many cases surpasses the written word by not being limited to cultural and linguistic differences, and in that it accommodates a very wide range of audiences. An example of visual expression is political caricatures, which are often simple drawings that convey & sup-up complex narratives or theories.  Those caricatures use a kind visual “shorthand” that relies on symbolism, where objects are used to represent ideas, and are arranged or juxtaposed in a manner that helps convey the story or theory. So it is both the semantics (the images themselves) and the syntax (the arrangement) that allows them to work. Information conveyed through political caricatures is much more striking, memorable, and powerful than plain words

According to Jiajie Zhang, external representations, or shapes and positions of symbols, have certain properties that allow them to convey information for people to process in an integrative and dynamic manner. Such properties include that those representations provide memory aids – either long or short term – that help reduce the memory load on preceptors, as symbolism in caricatures does. The exaggeration of certain qualities in their drawings (such as exaggerating facial features of political figures to make them comic) allows for those images to stick in the viewer’s mind. Also, those representations need to hold information that is easy to perceive with minimum effort yet be formulated explicitly, and minimize abstraction of ideas to aid understanding and processing, which is done in caricatures by using drawings of concrete objects to represent abstract concepts.

A recent example and trend of political caricatures is seen in the Syrian Revolution. Protestors heavily rely on those caricatures in communicating their demands, feelings, and ideas. In Kafranbel, a small town in the suburbs of the Idlib province, political caricatures have become a nationwide – even international – phenomenon, where this town, previously unheard of, gained stardom and fame among Syrians and people following the revolution all over the world. It is frequently referred to as “the icon of the revolution” and its caricature banners are considered as powerful summaries of events and ideas that represent the whole of Syria. Here are a couple of examples:

In this caricature, sectarianism, which is an abstract concept, is pictured as concrete object – a balloon – being blown by Assad and infiltrators of the revolution (again, hypocrisy is pictured as a mask being worn by an infiltrator) and the revolution, pictured as a person wearing the Syrian flag, pops this balloon. It serves to explain how Assad is the cause of any sectarianism present, and how the revolution is eliminating it.

Similarly, in this example, Assad is shown as if he is a  battery low on charge, where his biggest ally and supporter, Vladimir Putin, plugs him with a charger. It conveys how Russian support to the Assad dictatorship has been the main reason why he has lasted this long.

Finally, here are a couple of interesting quotes on political caricatures by Dr. Paul Parker, a political science professor from Truman State University:

“Relying on symbolism and caricature, experimenting in fresh imagery, political cartoons help people think about politics. Whether their purpose is to promote the status quo, raise social concerns, or to spur people to fight hard for change, political cartoons have changed the face of history.”

“Political cartoons are a unique creation–pictorial editorial and artistic social commentary. The medium of the political cartoon, which combines the political and the artistic with journalism, allows them to make social commentary beyond the boundaries of the written word.”




Dan Youra – The Art of Editorial Cartoons & Political Caricature

Jiajie Zhang and Vimla L. Patel, Distributed cognition, representation, and affordance. University of Texas at Houston / Columbia University

Reading this ad, as my species would

According to Deacon’s Symbolic Species “symbolic reference derives from combinatorial possibilities and impossibilities and therefore depend on combinations to…make use of it”. Thus, in order to do an analysis on a modern media artifact as full of meaning-making symbols, we must conduct a combinatorial analysis of said artifact.

I found a public service announcement (PSA) that can serve as the quintessential example of our symbolic faculties being compelled.

Here it is:

Here is my combinatorial analysis of this PSA:

I first identify the cacophony of symbols that arrest the visual, audio, textual, and discursive intuitions of our “Symbolic Species”:

Genre: PSA, which is publicly funded, didactic in nature, short, televisual, and has the aim of changing behavior for the “public good”.

Audio: The background music is an up-tempo, classical, symphony that is focused on a commanding trumpet beat. This is akin to the music usually used in blockbuster adventure movies in scenes where the protagonist/hero is in the middle of an epic battle or going through obstacles.

Identity words: The constant use of ‘you’ by ‘Nicki’, the woman dressed as snuff, refers to the baseball player she is nagging. Her use of ‘me’ and ‘I’ though she is snuff establishes the metaphor. The use of ‘you’ by the voice-over refers to the spectator, establishing a personal message.

Metaphors: The Nicki character is a woman dressed as snuff and nagging the man dressed in baseball gear as if she were his annoying and persistent ex-lover or ex-girlfriend that he is trying to avoid and ignore with all his might. Her name ‘Nicki’ is probably a personification of Nicotine, an addictive substance found in smokeless tobacco. Her merits that she uses to plead for him to come back to her are symptoms of snuff-use but conveyed in the language of an ex-lover’s platitudes: ‘We’d always be together’, ‘we have a commitment’ regarding the addictiveness of snuff; ‘You’ll never get through the game without your little Nicki’ regarding the reliance on snuff users have; ‘If you ignore me, I’ll play mind games with you’ regarding the withdrawal mental symptoms of quitting smokeless tobacco; and ‘Make your heart pump’ regarding the high from snuff.

Characters: There are two characters, a male and a female, who seem to have had a history of an intimate relationship that has recently been severed. The man is dressed for playing baseball and seems intent in going to his game. He never utters a word or attempts to utter one, establishing the fact that he is trying to ignore the female and that she is not an ex-lover, but an ex-addiction: snuff, which is an inanimate object. However, the woman that seems to be his ex-girlfriend intercepts him alone on the baseball field as he parks there and she starts nagging him trying in vain to get his attention. This woman’s name is Nicki (see metaphors). She is also dressed up as a giant, green, snuff can labeled as ‘Snuff: Smokeless Tobacco’ akin to a mascot costume common to American sports events. Because she cannot be identified by a human face, she becomes snuff and it is thus understood that she is not an ex-lover but snuff itself, which is compared to a nagging ex-lover by the PSA. The word ‘nagging’ is a very apt word to use in order to describe Nicki’s voice- it is incredibly high-pitched, nasal, whiny, and she speaks very quickly and pleadingly.

Voiceover:  In contradiction to the only other voice heard in the PSA (Nicki’s), the voiceover is a male one with a very low, masculine, pitch and a confident, self-assured tone as he speaks with conviction and omnipresence in a slower, more rhythmic cadence compared to Nicki’s nagging. The voiceover even speaks above Nicki’s nagging at one point towards the latter half of the PSA in order so that the audience hears the message that the clear voiceover is saying (and to hear the point of the PSA) but with Nicki’s voice dimmed in the background as nagging gibberish that is rendered incoherent because of the voiceover, but present nonetheless as a slight annoyance.

Facial expressions: The only facial expressions we see is that of the character that never speaks: the baseball player. At first, when Nicki starts to pester him, his expression is one of a forced ambivalence- stone-faced and fixed. But as she is pestering him even more and he constantly attempts to dodge her, his brows furrow in frustration and his lips curl upward on one side in a grin of held-back aggravation as his face paints a picture of tension but resolution (to not give in to Nicki).

Image/text contradiction: Never is there any real form of a woman or snuff at face value shown.

When we combine our symbolic resources i.e. language, audio, visual, etc., we end up with a media artifact that is laden with referential meaning making that leads to more referential meaning making in such a way that Deacon would posit. What do these previously mentioned symbols refer to in the PSA?:

Woman/addiction/unhealthy vs. Man/will-power/health: Why is snuff represented as a stereotypically annoying ex-girlfriend? There is the classic stereotype of women nagging men but more telling is the stereotype that women are temptresses and the weaknesses of men. Not only does this PSA use the seductress stereotype for the woman in the PSA, but the woman is also portraying an inanimate object that is purely consumed for pleasure and for a temporary high that leads to the downfall of men. When watching this PSA, everything about Nicki is personified and very human except of course for the outlandish and flamboyant mascot-like costume that is supposed to convey to the audience that she is a metaphor for snuff, which is scientifically founded to be unhealthful and highly addictive. The distinction between what is snuff and what is woman is blurred. This is because the seductress is a woman already known by the audience to be careful of, for although she is tempting with her abilities to make your ‘heart pump’, she is inevitably bad for you. Therefore, this feeling of wariness for the seductress is easily transferred to smokeless tobacco. Snuff too is attractive but is also unhealthy to consume. The baseball player is a male stereotype of athletic prowess as he is on his way to a sports event. He also fulfills the stereotype of male reservation and assuredness, as he is the calmer, quieter one in the relationship but also the tougher one as he plays sports and ignores the nagging with stride, dignity, and power of will.


Terrence W. Deacon, The Symbolic Species: The Co-evolution of Language and the Brain. New York, NY: W. W. Norton & Company, 1998.

How Do Interdisciplinary Approaches to Cognitive Structures Modify the Communication Process?

How Do Interdisciplinary Approaches to Cognitive Structures Modify the Communication Process?
By: Sara Levine 

Interdisciplinary approaches to studying our interactions with media and technology seem few and far between. Cognitive science, for the most part, is left out. Additionally, the sciences don’t always include important external factors. With this in mind, I would like to use Shannon’s model of the communication process and Foulger’s model of the communication process as tools for exploring theories on human cognition.

Fig. 1 Image source: <>

Shannon’s model reminds me of John Searle’s “Chinese Room” thought experiment from Andy Clark’s “Mindware.” We can decode the characters easily, but “symbol manipulation alone is not enough (Clark 34).” There is a lot more going on in a communication system than a simple transmit-and-receive interaction (Fig. 1). The source and destination usually manifest in human form and therefore should demonstrate a few of the cognitive processes that occur within the human mind. Humans rely heavily on symbols and symbol-oriented language. We use metaphorical structures and mapping when interacting with transmitting and receiving devices. These devices may have a certain level of consciousness because they are programmed to respond to scripts and instructions as we use them. It is also important to note that the device’s design has an overwhelming impact on the way the human uses it and thinks about how it is used. Hollan, Hutchins, and Kirsh use the example of the devices in the cockpit of an airplane. The panel that shows the pilot interacting with the fuel gauge, for example, is treated as both a symbol and as the fuel system that it represents (Hollan, Hutchins, Kirsh 185). Finally, the “noise source” should be expanded out to include environmental factors in the communication process.

Fig. 2

Let’s apply this infinitely more complicated model to a Skype call (Fig. 2). Person A calls Person B over Skype. They both enable video and start a conversation. There are a multitude of both hierarchical and parallel processes occurring at once. The conversation between Person A and Person B is based in English, which is a largely metaphorical language. The meaning-making processes within their minds are working to interpret sentences and to form ones in return. At the same time, Person A and Person B are interacting with Skype technology. They are navigating the Skype interface in order to facilitate a fluid communication experience. This requires Person A and Person B to interpret the symbolic structure of Skype, much like the pilots assign symbols and meaning to the devices in the cockpit of a jet. Person A and Person B’s computers are displaying their own level of consciousness in this interaction because they are following the software instructions of Skype as it responds to the situation at hand. The environment, WiFi connections, and other external factors are also important factors throughout this example of communication. These revisions to the Shannon model are not final. There are probably many other cognitive and technical processes occurring that we may not be aware of.

Fig. 3 Image source: <>

Foulger’s model is designed to be more applicable to media. However, it can also be modified to include several meaning-making concepts and theories (Fig. 3). Foulger does an excellent job of representing cognitive functions with words such as “imagine” and “interpret”. However, as Lakoff describes in “Conceptual Metaphor,” much of our communication is mapped onto other metaphors. Consequently, there is an entire structure of metaphorical language and symbolism that the Creator employs to distribute her or his message. The Consumer employs similar structures to interpret the message. In between those two actors is media, which functions as another symbolic architecture through which these two actors communicate. The cognition processes and structures differ depending on the type of medium and technology that produces media. Additionally, a multitude of external structures (social, cultural, embodied, etc.) affect this model as well.

Fig. 4

A more specific example of this could be an American romantic comedy film (Fig. 4). The creator enacts the communication process even before she or he has written the script because thought is an interpretive interaction. However, she or he eventually produces the story through a visual medium. In addition to the symbolic structure of the English language, she or he has also encoded the story through the structures embedded within both film and the romantic comedy genre. The couple has a “meet cute” at the beginning of the film, for example. The editing choices for each shot hold meaning in terms of focus, timing, and composition. Perhaps all “meet cute” scenarios are shot with similar pacing. Therefore, due to an indexical cognitive learning process, audiences know how to interpret this sequence of events as a “meet cute”. Some external factors may include the history of film, the history of romance as a genre, American culture shaping the cognitive processes of audience members, etc. 

These revisions are not intended as permanent alterations, but as an exploration into how cognitive theories and concepts might reflect on models of communication. My previous experience with studying communication processes was through the narrow lens of microsociology as described by Erving Goffman. He used theater metaphors to explain the seemingly mundane gestures and conversations of everyday life. Goffman used terminology such as “performance,” “front stage,” “backstage,” and “audience.” Consequently, the dramaturgical (as his theories are labeled) perspective may be another angle of approach for cognitive and communication processes. Goffman’s metaphors could even be studied under the cognitive lens as an internalized structure for shared cognition amongst sociologists. The potential for interdisciplinary exploration into the communication process seems to indicate that there is more to be discovered through further collaboration between multiple fields of research. 

Works Cited

Clark, Andy. Mindware: An Introduction to the Philosophy of Cognitive Science. New York: Oxford University Press, 2001.

Day, Ronald E. “The ‘conduit Metaphor’ and the Nature and Politics of Information Studies.”Journal of the American Society for Information Science 51, no. 9 (2000): 805-811.

Deacon, Terrence W. The Symbolic Species: The Co-evolution of Language and the Brain. New York, NY: W. W. Norton & Company, 1998. Excerpts from chapters 1 and 3.

Foulger, Davis. “An Ecological Model of the Communication Process.” A Ecological Model of the Communication Process. N.p., n.d. Web.

Foulger, Davis. “Models of the Communication Process.” Models of the Communication Process. N.p., n.d. Web.

Goffman, Erving. The Presentation of Self in Everyday Life. Garden City, NY: Doubleday, 1959. Print.

Hollan, James, Edwin Hutchins, and David Kirsh. “Distributed Cognition: Toward a New Foundation for Human-computer Interaction Research.” ACM Transactions, Computer-Human Interaction 7, no. 2 (June 2000): 174-196.

Lakoff, George. “Conceptual Metaphor.” Excerpt from Geeraerts, Dirk, ed. Cognitive Linguistics: Basic Readings. Berlin: Mouton de Gruyter, 2006.

Read This Title As You Are a Human Being Who Understands English

“We don’t read and write poetry because it’s cute. We read and write poetry because we are members of the human race. ”   — Mr.Keating, Dead Poets Society

A quote from the film Dead Poets Society may illustrate the light and beauty humans have discovered in poetry, as well as the primitive men read the distinct smell in the wind as a warning of a dangerous animal’s approaching. We used to recognize and interpret sign, and generate its meaning. Briefly, Deacon concludes: “Breaking down the term re-cognition says it all: to think [about something] again.” (Deacon, 77) Deacon’s piece at length talks about the interpretive process, the tricky relationship between Peirce’s “icon, index and symbol”. At first, the triad reasoning seemed confusing, until I found Deacon’s comment on Peirce’s contribution to the semiotics very interesting: “Peirce’s most fundamental and original insights about the process of interpretation: The difference between different modes of reference can be understood in terms of levels of interpretation.” (Deacon, 73) Indeed, the mystery of a sign is that it can be interpreted in a million ways: anything is a sign once it is interpreted as a sign. And that means, one thing can simultaneously be treated as an icon, an index, and a symbol as the content is defined by the context.

It’s a recursion, infinite creations in finite materials. Deacon does mention the word “recur”: “A languagelike Signal would exhibit a combinatorial form in which distinguishable elements are able to recur in different combinations.” (44) I find it fascinating. Think how many times a simple object “apple” has been interpreted and how fast these interpretations and meanings have gone through your brain. A picture of an apple is an icon when we see it, we think of the round, red, juicy apple, but wait, it becomes a symbol as the forbidden fruit in the Garden of Eden, a symbol of the original sin, also, isn’t it the apple that kills Alan Turing? And what about Steve Job’s Apple? The multiple mappings make our existing world so complex and colorful, and we as humans seem to naturally have the ability to associating, to make sense from nonsense, and make more senses out of that.

Take my meaningless title for instance, “Read This Title As You Are a Human Being Who Understands English”, if the person who reads it does not understand any language, he/she can’t see any words or letters but only the strokes of various shapes, maybe the character font is “cambria”, and the word size is “12”, at this point, the printed sentence is just an icon. It’s an index when I point out that it’s the title of this blog, to relate the sentence as a whole to the function of a title, and even a quote by myself. It becomes a symbol when you really read it and almost immediately understand its meaning. By that means, any words, sentences, letters, any languages are symbolic signs – Peirce’s symbols. By generating ideas and connecting different indexical signs, we create layers of symbolic significance, and that extends to metaphors. My sentence “Read this Title As You Are a Human Being Who Understands English” is no longer a simple sentence with literal meaning, but a metaphor for this week’s readings, a joke about semiotics and our mind. At this point, the metaphor is much more important than the language itself.

According to Lakoff, a metaphor goes deep into different domains of experience. “A metaphor can be understood as a mapping from a source domain to a target domain.”(Lakoff, pg190) How boring our lives would be without metaphors? Poetry will lose the depth of soul, and TV shows will lose their fun and vigour. Joe Wang, a Chinese comedian in America who has a PhD in bio-molecular, I always found his talk shows full of metaphors that are rich humor and let people laugh. In his performance at RTCA dinner, he joked about the questions from American history lessons: “Who is Benjamin Franklin?” And he thought: “The reason our convenience store gets robbed?” When asked: “What is the Second Amendment?” He thought again: “The reason our convenience store gets robbed?” Similar examples are too numerous to list, and my point is that, although we can treat thought as computation as Clark mentions in his piece, the great difference between human and machine lies in what the signs and symbols have endowed us: we don’t just follow the procedures but generate our existing ideas and keep digging new meaning in our lives. We laugh when we understand that new meaning, and the laugh itself is a representative of joy.



Terrence W. Deacon, The Symbolic Species: The Co-evolution of Language and the Brain. New York, NY: W. W. Norton & Company, 1998. Excerpts from chapters 1 and 3.

George Lakoff, “Conceptual Metaphor.” Excerpt from Geeraerts, Dirk, ed. Cognitive Linguistics: Basic Readings. Berlin: Mouton de Gruyter, 2006.

Andy Clark, Mindware: An Introduction to the Philosophy of Cognitive Science. New York: Oxford University Press, 2001. (Excerpts from Introduction, Chapters1-2)


Chinese Literati Paintings: Behind Symbols

The literati painting is a genre in Chinese painting that has its root in Han Dynasty, thrived in Song and has been one of the most important component of Chinese culture throughout history to this day. These paintings are artworks from members of the literati class, the well-educated intellectuals whose works are more contemplative than anything practical. Here I will use them as examples of interpreting some of the notions in the studies of the human symbol system.

First lets look at these paintings at their representational surface. An overwhelming majority of literati paintings use a limited number of imageries, such as plum trees/blossoms, bamboos and mountains. All of these are objects that exist in nature, in Chinese everyday life (at least for those who can afford it). Most people have seen them at least sometime in their lives. This is what Deacon would define as an “iconic” relationship (74). For example, when one knows what he/she sees in the painting of bamboo is the kind of plant he/she has seen in the garden, he/she does not make a distinction between the art and the plant.

Yet for someone who has seen bamboos, the represented art form looks far different from what one sees in real life. It’s two-dimensional, it’s colorless, and it’s highly abstract. This is where an “indexical” relationship (Deacon 77) comes into play, in which case the abstract art serves as an index item that point to the viewer’s idea of real bamboo.

But for the relationship to be considered “symbolic”, the most essential of which is combinatorial possibility (Deacon 83). None of the paintings above reveal an exact copy or exact abstraction of the real bamboo. In the procedure of making the art, items (such as butterflies) are added or subtracted, spatial relationships are altered, and compositions are manipulated. And these are not done randomly, however “free-thinking” the literati class have branded themselves to be. In fact, especially in the Ming dynasty, painting manuals that specifically teach about related rules were sold in large circulations that reached all tiers of society (Park).

Now lets look beyond the paintings themselves. All of the imageries have explicit meanings. For example, plum means endurance and pride, bamboo implies modesty and integrity, and mountain often refers to aloofness and detachment. These are metaphors shared by the entire Asian culture nowadays, but when they were initially used, they were symbols understood and used among the elite literati class. Such is what Hutchins would classify into “distributed cognition” (175), extending the cognitive process beyond boundaries of one’s mind to encompass interactions between people and with resources and materials in the environment. For the symbol system to work and be passed down through generations, it has to be externalized and weaved into the culture, be it group or social. It would be interesting to look into when they first come into being, but it is clear that once a core group or even one prominent figure has started using it, others have followed suit and reinforced the associations via the annotations and poems on the paintings, via the literature works which also circulate in the class group, and via the painting manuals that extend the influence of the symbol system to the entire society. This way of thinking about an art genre can be applied to all arts or all media, each with their different combinatorial rules and different cultural preconditions, which is where the real fun is.

Deacon, T. W. 1998. The Symbolic Species: The Co-evolution of Language and the Brain. New York, NY: W. W. Norton & Company.
James, H., Hutchins, E., & Kirsh, D. 2000. Distributed Cognition: Toward a New Foundation for Human-Computer Interaction Research. ACM Transactions, Computer-Human Interaction, 7(2): 174-196.
Park, J. P. 2012. Art by the Book: Painting Manuals and the Leisure Life in Late Ming China. Seattle, WA: University of Washington Press.

“What Feelings Sound Like”

Elizabeth-Burton Jones
Week 4

“What Feelings Sound Like”

A river flows alongside the grass and clashes abreast rocks. The fish, the snakes, the frogs take refuge in the flow. It moves, and with this movement the river’s environment is introduced to different terrains some calm and some tumultuous. And in one moment this river flows into a fall. The river starts to disconnect and the environment that once was flowing, becomes a scrambled flow, still moving, yet ever-changing in the movement. Ever-changing in the fall.  Splat. Splash. Mist.  But with all of these changes, it is still flowing. As water cascades down from a waterfall so too do thoughts flow from one aspect to the next. Just as the river mentioned above, thoughts begin with an original flow. Even though the thoughts are introduced to different feelings (happy or sad) and can at sometimes become disconnected, the thoughts still flow from one thought to the next.  For this week, I would like to look at the stringing of the mind and the mind’s musical flow.

I was inspired by the Clark and Deacon reading.  In the Clark reading, the “reason respecting flow” was mentioned. In short, the reason-respecting flow suggests the introduction of one thought breeds introductions of other thoughts and feelings (“sun->sunscreen->paradise”). To connect to the prompt, I tried to link the relation between the flow of thoughts and the sound of music. This made me wonder about the validity of the flow of thoughts and how they can be emotionally shifted when music is introduced.

For instance, right now when you hear any one of these songs, how do you feel? What do you think about? – Phil Collins -Red Red Wine – Welcome to the Jungle – Giving Up – Moonlight Sonata – Pop –  Bust the Windows


Or when you listen to music from Teddy Pendergrass, John Legend, or Barry White how does that change your emotions?

The examples listed demonstrate a wide variety of feelings. The songs stimulate anger, excitement, sadness, relaxation, etc. So I wonder how these feelings can be attached to music and how the music can activate these feelings and continue the flow of other thoughts. For instance, if a person is sad, and they play music that is upbeat, how can this change the flow? Also, music can change the flow and cause a mixed emotion.

Another example of the aroma of music could be an emotional attachment. When looking at music, often certain feelings are brought to the surface. According to the Deacon reading the certain words can bring other aspects. (“Figure 1.1” “stringing together words in a sentence leads the listener to bring together images in the mind”). The same could be said for music. By stringing together different instruments and voices and tones, different images could be created in the mind.

Every one has different reactions to music, some reactions could be created by thoughts and feelings. For instance one song could be connected to your first Kiss or first date. In the movie, “Silver Linings Playbook” the main character has a certain connection to the above song. The Stevie Wonder Classic was his wedding song. However, one day when he came home he found his wife with another man, the music that accompanied his wife’s lip-locking with another man was in fact his wedding song, the Stevie Wonder classic. Needless to say, the string of thoughts became tangled up and resulted in his mental breakdown.

To try to answer the question about combining music and a symbol a plethora of things could happen. Someone could have a nervous breakdown (“Silver Linings Playbook”), someone could be reminded of a happy moment. These strings create a tapestry of musical representations that are visible and invisible.

Respecting the flow in music

Respecting the flow from feelings to musical arrangement.

Finally, there is a connection between spoken word and the transition to song. Through this transition the flow of thoughts and emotions tries to be equally translated. This form of music is difficult to achieve because it takes time to learn how to channel one’s thoughts into another form of speech. By pouring out raw emotion into music, the artist respects the flow and a greater level of mastering of music is achieved.

In conclusion, the connection of music and symbolism is inextricably linked to the flow of language. Without the flow from one subject to the next and the continuous activation of thoughts, music would not be able to have a powerful impact. However, if music is built from thought, (which through this week’s lesson we can conclude that music is a product of the flow) then we can have meaningful emotional connections to music and “we’ll all float on ok”.

Work Cited:

Combinatorial Meaning in Pottermore

A scene of Platform 9 3/4 from the Pottermore scavenger hunt. Credit:

One of the defining literary and cinematic events of my childhood and early adolescence were the Harry Potter books and movies. Like most other kids at the time, I was completely absorbed into the fantasy realm created by J.K. Rowling, and followed both forms of media until their completion. It seemed like there would always be a Harry Potter related mode of entertainment to enjoy, so when the last movie truly ended the series, fans like myself were looking for something to fill the void. Created by J.K. Rowling and Sony Entertainment, the website was introduced in 2011 as a supplement to the Harry Potter books. This website meant to give fans access to unreleased written material and facts from the author by way of an interactive game.

The website is set up as a community, but allows users to create their own profiles and independently go through the Harry Potter books chapter by chapter, collecting new facts along the way. Each chapter presents a few interactive settings in which users engage in a scavenger hunt or game to gain new information and the ability to move to the next chapter. The farther a user gets into the book, the more “connected” he or she becomes to the Harry Potter world. For example, once users reach the chapter in which Harry gets sorted into a house, the user gets sorted as well and can begin interacting with other members of their house.

The large amount of information combined into one platform makes Pottermore a great example of a combinatorial meaning system. Not only does the website require users to have previous knowledge of an imaginary world that is then symbolized online, but it also assumes that an entire real-life community can be created and symbolized digitally from this knowledge. The most noticeable means of combinatorial meaning are the visuals used on the site. These visuals provide a sense of “reality” to the feeling of actually being “in” Harry Potter’s world by way of animations, photos, icons and videos. The sounds associated with the movements of many of these animations and visuals also allow users to feel a multidimensional sense of cognitive reality, regardless of the fact that they are engaging in actions that are digitally expressed (not to mention, based on imaginary plotlines.) Written instructions tie the whole experience together, and depend on the symbolization of language to tell the user what to do next, as well as describe the new facts they may encounter along the way.

Going through the Pottermore website, three ideas from this week’s readings became apparent to me. First, Gibson’s idea of “affordance” as described by Zhang and Patel, is very relevant to interactive multimedia websites like Pottermore due to their holistic approach in expressing information. Since “what we perceive when we look at objects are their affordances, not their dimensions or properties” people experience Pottermore in its entirety, rather than just seeing or hearing one part of it at a time, or just seeing the interactive experience as a simple function performed by their computer. (337) One of the main ways this website allows for the holistic aspect of affordance to be apparent, is the manner in which the point of view of each scene is set to be from the perspective of the user as though he or she is seeing the visuals and performing tasks head on.

This holistic aspect of affordance leads in to the example of the way a pilot behaves in a cockpit while interacting with his technological interface, as described by Hallen, et al. The way in which the pilot manipulates the interface automatically without physically altering some form of his machinery or environment is applicable to the icon manipulations a Pottermore user completes. For example, if a user is prompted to combine ingredients to brew a potion (not until he or she is an official student, of course) he or she will not actually transfer the ingredients into a cauldron in real life, but would rather understand that the actions are only demonstrated as manipulations on the website. By doing this, players are “[manipulating] the properties of a representation to encode information that does not pertain to and is not about the thing that the representation represents.” (186)

(For a link to a screenshot of a Pottermore potions class, click here.)

Finally, the interpretations of each chapter’s activities are highly cognitively symbolic within themselves, particularly in relation to how the “difference between different modes of reference can be understood in terms of levels of interpretation.” (73). Each player will see exactly the same setting in each chapter the format of the site presents, but each player will also most likely have different interpretations as to where they should start clicking to begin their scavenger hunts, or what they want to read first. This would create different levels of interpretations of the same information even though initially all players saw the same visuals.

Having a better understanding of the massive amounts of symbolic thinking that is involved in multimedia projects like, really demonstrates how much goes into the creation of a modern digital artefact. Additionally, thinking about each component of internal and external communication truly brings to light how much we take the complexity of online communication for granted.


Terrence W. Deacon, The Symbolic Species: The Co-evolution of Language and the Brain. New York, NY: W. W. Norton & Company, 1998.

Hollan, James, Edwin Hutchins, and David Kirsh. “Distributed Cognition: Toward a New Foundation for Human-computer Interaction Research.” ACM Transactions, Computer-Human Interaction 7, no. 2 (June 2000): 174-196.

Zhang, Jiajie, and Vimla L. Patel. “Distributed Cognition, Representation, and Affordance.” Pragmatics & Cognition 14, no. 2 (July 2006): 333-341.


The Distributed Cognition of Television

A media artefact that I usually like to view things through is television. I thought that it was particularly useful this week with so many concepts being introduced to take things back to where I’m comfortable. This week introduced distributed cognition, and the ideas of how language and symbols interact with the human mind, how that has evolved, and also how we use metaphor to make meaning. According to Hollan, Hutchens and Kirsh, the theory of distributed cognition looks beyond the individual cognition of a person and includes interactions between people as well as with the outside environment and materials. Some components of this are the coordination of internal and external, and the idea of memory. Things that happen now can be products of earlier events, and rely on that memory. “The relations between internal processes and external ones are far more complex, involving coordination at many different time scales between internal resources—memory, attention, executive function—and external resources—the objects, artifacts, and at-hand materials constantly surrounding us.” (Hollan, et. al).

The television medium shows a modern example of distributed cognition. Shows rely on a lot of different players and parts to be successful. The interaction between the creator and the writers, the producers and the actors, the editors and the musicians all make a television show what it is. Beyond that is the audience. The people behind a show rely on the memory of the audience to keep up with a long-standing story. Television is an ongoing art form and something that requires the worn-in feel that they talked about wanting to replicate in modern HCI. Symbols, language, and meaning making are also extremely important to a successful television show. Especially in genre shows, there are certain signs that let you know what to expect. In a comedy, you expect a physical pratfall or a even a laugh track which reminds you when to laugh. In a spy drama, you expect twists and turns. In the horror genre, you know that once someone runs up the stairs, they’re probably going to get killed. Metaphors can also be hugely effective to evoke an emotion or to impart a feeling without having to write it out as exposition.

The example that pops to mind is The Americans, a show that just premiered last week. The show is set in 1981 in D.C. and features a married couple who run a travel agency in Dupont Circle. Except it turns out that they’re really KGB agents planted in the US to gather intel during the Cold War, and their marriage is arranged through their government. To make matters more complicated, they’ve been in their marriage for 15 years and have had children together and might be starting to actually have feelings for each other. One of the largest aspects of the show is the idea of the Cold War as a metaphor for their marriage. With all of the espionage and lies, can a spy really know or trust anyone? Can you really ever truly know your spouse? There’s giving and taking and changing relationships. There’s diplomacy and cooperation. There’s a sense of allegiance, whether or not you’re having a good day, to each other. Beyond that is the symbols of a bunch of different things. There’s the indicators throughout that denote the time period: the fashion, the music choices, the references to Reagan and certain astronauts. There are symbols of a(n outdated) spy show: wigs and disguises, interrogations, high-speed chases. The writers behind The Americans are betting on the audience’s memory of the new characters and the plot, but also of the historical implications of the time. They’re trusting you to get the genre and to roll with it. And I think there’s the certain feeling of responsibility to give you what you’re expecting – the drama, the high stakes, the close calls. On top of all of this is the fact that the show is based off of a true story of Russian sleeper cells that were discovered living in New Jersey in the early 2000s. Does that cognitive memory need to be tapped?

I think overall that television requires that sort of distributed cognition due to its serialized nature. Shows with big casts and multiple seasons require an almost working relationship between viewer and writer. There is an entire world that is made in which people need to suspend their own realities and join to understand. The grammar of serialized television includes all of these elements.  By combining so many different elements, such as the written words, the visual components, the music, and the story, a lot of different meaning and symbols need to be deconstructed to be understood as a whole. This also requires the knowledge from viewers of each of these aspects, and their reaction can ultimately impact the show. If things aren’t working, the creators will be forced to fix it or they’ll be at risk of going off air. Since television lasts longer than film, it has become much more collaborative with its audience, which can add even more to the distributed cognition. There’s an additional landscape putting in their inputs.



Deacon, Terrence W.  The Symbolic Species: The Co-evolution of Language and the Brain.        New York, NY: W. W. Norton & Company, 1998.

Hollan, James, Edwin Hutchins, and David Kirsh. “Distributed Cognition: Toward a New   Foundation for Human-computer Interaction Research.” ACM Transactions, Computer- Human Interaction 7, no. 2 (June 2000): 174-196.

Lakoff, George. “Conceptual Metaphor.” Excerpt from Geeraerts, Dirk, ed. Cognitive Linguistics:            Basic Readings. Berlin: Mouton de Gruyter, 2006.

Poetry, The Website, and Making Meaning in a Combinatorial Symbolic World

(1) consider a media artefact as an instance of a combinatorial meaning system (an artwork, a Website, a movie, music composition) with some features investigated in readings

An artefact inside an artefact

Building off last week’s concepts regarding language and symbolic cognition, I was able to relate a recent experience to the concepts of symbolic cognition and human meaning-making. Last week, I attended a live discussion event by Dr. Maya Angelou at the Smithsonian National Museum of the American Indian. Angelou is frequently referred to as “a poet, educator, historian, best-selling author, actress, playwright, civil-rights activist, producer and director” according to her official website (2013). I remember growing up with her work as a child, and also coming across her work throughout college.  After digesting this week’s readings, I found it relevant to examine her website, as a media artefact. This website allows individuals to interact with her metaphorical works in a drastically different means compared to 30+ years ago when Dr. Angelou was just as influential, but could only share her work through written text (books, poetic literature) and oral traditions (speeches and conversations). She still does this now, but her website is yet another outlet. Humans are constantly looking for ways to formulize meaning through all our media channels, and as we will see the website is basically a culmination of our ancestors’ histories and symbols that have accrued over time.

Angelou’s website is the epitome of how combinatorial meaning systems work and interact with humans today. As for nearly any website, key symbolic materials such as language (English words) visuals (videos, photos, graphic designs), sound (from the videos) are presented in an orderly fashion on Angelou’s website. To the majority of users, the resources make sense because they follow proper sentence structures like “Welcome to Maya Angelou’s Official Site”. According to Deacon (1998, p.98) “creating a larger sentence in a human language cannot just be accomplished by stringing together more and more words. It requires use of hierarchic grammatical relationships, as well as syntactic tricks …” One could also argue, though, that this artefact (the website) follows a certain grammatical/syntactical flow in the form of media content just described. The videos, text, images are not arbitrarily placed on one immeasurable page; instead there is a central organization and layout of media intended to appeal to users.  This example of her website appears to follow Clark’s (2001) definitions of syntactic and semantic properties [1].

What Maya Angelou is primarily known for is her poetry, and not her website. Nevertheless, her website bears weight because other artefacts (her poems, books, interviews) are placed or inserted “on” this seemingly grandiose artefact. In Renfrew’s (1999) view, artefacts have critical roles both symbolic and practical for humans. Considering just her poetic work, the terms ‘symbolism’ and ‘metaphor’ are often associated with poetry. For Wong (2005), symbolism can be regarded as “the invention of external storage of information – whether in jewelry, art, language or tools” which dates back to the evolutionary period (p.89).  Angelou’s poetry indeed holds symbolic power in the form of literature, but also as an art form and tool she uses to express her thoughts, beliefs, and experiences.

One of Angelou’s famous works

Furthermore, Angelou’s form of language is primarily metaphorical. Lakoff (2006) describes metaphors as “instances of novel poetic language in which words…are not used in their normal everyday sense” (p. 185).  Taking a look at one of Angelou’s most famous poem  I Know Why The Caged Bird Sings it is possible to apply Lakoff’s contemporary theory and conceptual framework surrounding metaphors, just like he did with the love poem that stated ‘Look how far we’ve come/ It’s been a long, bumpy road/ We can’t turn back now.” (p.189)

         This poem follows similar metaphorical patterns of ‘improper’ syntax (“on the back of the wind/ and floats downstream”) but makes complete metaphorical sense in the eyes of many poets or less syntax-focused individuals. I agree with Lakoff’s point that “the metaphor is not just a matter of language, but of thought and reason” (p.192) and that language takes a backseat in these instances. Here, Angelou’s website could perhaps be seen as a depository of information that allows for the combination of multiple artefact styles. Another relevant point that can relate to the artefact of Angelou’s website is the growing field of cognitive science. If the theory of ‘distributed cognition’ from Hollan et al. (2000) is applied to this case of media artefacts, it leaves room for new insights about the unique field of interface-centered technology. The distributed cognition theory basically implies that there is a need to understand the interaction between technologies and people, furthermore stating:

            “Digital objects can encode information about their history of use. By recording the interaction events associated with use of digital objects (e.g., reports, forms, source code, manual pages, email, spreadsheets) it becomes possible to display graphical abstractions of the accrued histories as parts of the objects themselves” (Hollan et al., p.187, emphasis added) 

A user can explore her history through the artefact (website)

Hollan et. al (p.187) provides the opportunity to try and apply Angelou’s website and poetry to this inquiry. A website can serve as a vast depository for history and as a projector of constantly revolving information. Once put on the website, Angelou’s poetry (her metaphorical language) transforms into a digital object. Once a tangible artefact, it is now an artefact comprised of computing numbers and codes. But the end user (the person who reads her poems through a computer screen) does not need to decode the artefact in a drastically different way than before – it can still be read, analyzed, copied, etc., but in slightly different measures. No matter which environment the artefact is positioned, the “highly-enriched digital” object (p.188) is an overarching representation of her poetic ability and history. What researchers mustn’t do, according to Hollan et al., is neglect the fact that culture and cognition studies should not be separated. If someone is trying to grasp how a user interprets the poetic artefact via a second artefact (a website), culture (which is comprised of person’s history, language, customs, etc) may very well influence their cognitive functions and abilities to create symbolic systems of meaning. Some can only wonder what the Internet will look like ten to twenty years from now, but I foresee higher levels of interaction through the “graphical abstractions” between humans and media artefacts


[1] Clark defines Semantic properties as “the ‘meaning involving’ properties of words, sentences, and internal representations,” and Syntactic properties  as “nonsemantic properties of, e.g., written or spoken words, or of any kinds of inscriptions of meaningful items” p.10,



Clark, Andy. Mindware: An Introduction to the Philosophy of Cognitive Science. New York:          Oxford University Press, 2001.

Deacon, Terrence W.  The Symbolic Species: The Co-evolution of Language and the Brain.        New York, NY: W. W. Norton & Company, 1998.

Hollan, James, Edwin Hutchins, and David Kirsh. “Distributed Cognition: Toward a New   Foundation for Human-computer Interaction Research.” ACM Transactions, Computer- Human Interaction 7, no. 2 (June 2000): 174-196.

Lakoff, George. “Conceptual Metaphor.” Excerpt from Geeraerts, Dirk, ed. Cognitive Linguistics:            Basic Readings. Berlin: Mouton de Gruyter, 2006.

“Maya Angelou.” Maya Angelou, The Official Website. Accessed 4.      Feb 2013.

Renfrew, Colin.  “Mind and Matter: Cognitive Archaeology and External Symbolic Storage.” In     Cognition and Material Culture: The Archaeology of Symbolic Storage, edited by Colin     Renfrew, 1-6. Cambridge, UK: McDonald Institute for Archaeological Research, 1999.

Wong, Kate. “The Morning of the Modern Mind: Symbolic Culture.” Scientific American 292, no. 6 (June 2005)