Author Archives: Yasheng She

Value and the Meaning System – Yasheng

Value and the Meaning System

Abstract

This paper aims to demonstrate the indexical nature of interface through a case study on the dual-function of Library of Congress (LOC), it is simultaneously an museum and a library. The paper illustrates the differences between an artifact and an interface and their relation to value. The main goal of this paper is to understand the significance of reproduction, especially digitalization, in preserving an artifact’s interface to its meaning system, and, therefore, protecting the integrity of the collective human experience, or in other words, universal cultural memory.

 

One of the biggest complaints people often bring up these days is that people do not read anymore because technological advancement has taken away people’s motivation for reading. This statement is, however, not factual. The Pew Research Center conducted a survey in 2014, their findings suggest that Millennials are reading more books than the over-30 crowd.[1] Why? It turns out that the Internet has helped to cultivate a reading habit among Millennials through the widening access to information.

But can digital books replace physical books? Personally, I enjoy the textual of a physical book and somehow find a digital book less valuable and less personal. There are many reasons for it, such as I can doodle on a physical book, not have to worry about charging my tablet, and, most importantly, the tangible quality of a physical book makes my copy unique and valuable.

This train of logic leads to the thesis of my paper. What makes something valuable? The most obvious answer is scarcity. This is why rare metals are expensive. But what about culture? How do you determine the value of a cultural product? Is it determined by how well it is made or how many copies of it still exist in the world?

To answer this, we must contemplate the significance of reproduction of artifacts. Morris Eaves, who has studied William Blake extensively, states that an interdisciplinary approach from fine art and technology engenders a new perspective on the relationship between Blake’s term “machine” and art. He maintains that,

The perfect system of reproduction would reproduce anything perfectly. Because there is no perfect system, the reproductions will always be imperfect. The technological gap between the original and the reproduction is what interests us, because we can safely guess that the difference that occurs in the gap is significant and affects the artists’ originals not only now but also in the future and affects as well the audience, its understanding of the work, judgment of the work, artistic expectations, and so on.[2]

The gap Eaves mentioned here is also what Walter Benjamin laments in his response to the rise of photography. Benjamin goes a step further warning the ramification of digital reproduction,

The situations into which the product of mechanical reproduction can be brought may not touch the actual work of art, yet the quality of its presence is always depreciated. This holds not only for the art work but also, for instance, for a landscape which passes in review before the spectator in a movie. In the case of the art object, a most sensitive nucleus – namely, its authenticity – is interfered with whereas no natural object is vulnerable on that score. The authenticity of a thing is the essence of all that is transmissible from its beginning, ranging from its substantive duration to its testimony to the history which it has experienced. Since the historical testimony rests on the authenticity, the former, too, is jeopardized by reproduction when substantive duration ceases to matter. And what is really jeopardized when the historical testimony is affected is the authority of the object.[3]

Walter Benjamin maintains that the loss in authenticity in art caused by reproduction will eventually jeopardize the original artifact’s value. A valid argument indeed, yet it lacks certain productivity. Lack of productivity in terms of Benjamin never offers a feasible solution to the issue he sees. Artifacts are mediums of universal cultural memory. They help us understand the collective human experience. If an artifact’s interface is forever lost, the meaning system that it associates with will be lost as well. André Malraux offers different take on this issue, he claims that reproduction, though the quality of such is not as favored as the original, can grant access to those who would not have such access otherwise. Putting together a collection of art works is like building a “museum without wall”[4]. Reproduction takes arts out from their esoteric environment and do so ensures their presence in universal cultural memory. Malraux maintains that,

Nowadays an art student can examine color reproductions of most of the world’s great paintings, can make acquaintance with a host of second-rank pictures, archaic arts, Indian, Chinese and Pre-Columbian sculpture of the best periods, Romanesque frescoes, Negro and ‘folk’ art, a fair quantity of Byzantine art… Hitherto the connoisseur duly visited the Louvre and some subsidiary galleries, and memorized what he saw, as best he could. We, however, have far more great works available to refresh our memories than those which even the greatest of museums could bring together. For a “Museum Without walls” (musée imaginaire) is coming into being, and (now that the plastic arts have produced their printing press) it will carry infinitely farther that revelation of the world of art, limited perforce, which the “real” museums offer us within our walls. [5]

This is a video about how Library of Congress (LOC) preserve and extrapolate infarction from old and damaged artifacts using technology.

The speaker in video, Sarah Werner, maintains that digitization ensures the integrity of cultural memory.[6] The work Werner presents validates Malraux’s claims, digital reproduction, though will always be getting better, preserves artifact and its interface to its meaning system.

So back to my thesis. We are facing a dilemma between damage to the value of original artworks due to reproduction and the benefit of valuable access gifted by reproduction. So how do we determine value in any meaning system?

In order to answer this, my paper will divide into three sections;

  1. Index’s relationship to value
    • Why index cannot retain value
  2. Artifact and Interface
    • Artifact retains value
    • Interface as index
    • Interface’s relationship to value
  3. The necessity for digital reproduction for it is a means to protect the value of universal cultural memory

Index and Value

Listed below are two indices of books. Index A is an opinion piece on “what are the most influential books ever?” This list is compiled by “an expert panel of academic book-sellers, librarians, and publishers, and members of the public were asked to vote online for their top 20.”[7] Index B is where you can locate the most influential book, On the Origin of Species by Charles Darwin, using Library of Congress Classification (LCC), which is adopted by the majority of research and academic libraries in the U.S.

Index A

Index A

Index B

Index B

Index A ranks these books according to their assigned value whereas index B maps out the location of the number one book in the LCC system. At first glance, two indices appear quite different. The main difference is in purpose. Yet from a semiotic perspective, these two serve the exact same function according to Charles Sanders Peirce’s definition of an index, “a guidepost, which points down the road to be taken, or a relative pronoun, which is placed just after the name of the thing intended to be denoted, or a vocative exclamation, as ‘Hi! there,’ which acts upon the nerves of the person addressed and forces his attention.”[8] The two indices have no physical property of their own, only serve as pointers to unveil the underlying meaning systems. In other words, these two indices are signifiers that signify different connotations, and the difference between the two only lies within what they signify, the signified.

Pierce’s semiotic theory is undoubtedly more productive when considering the relationship between the signifier and the signified, yet Ferdinand de Saussure’s idea on the arbitrary relationship between the two is helpful when we consider the index and the underlying meaning system it indicates. It is true that index A seems more subjective comparing to index B and, therefore, it is more arbitrary. In some way, that is true. Yet I will cite Martin Irvine’s definition of arbitrary so to proceed my argument,

The term (arbitrary) should not imply that the choice of the signifier is left entirely to the speaker (… individual does not have the power to change a sign in any way once it has become established in the linguistic community); I mean that it is unmotivated, i.e. arbitrary in that it actually has no natural connection with the signified.[9]

This argument remains valid outside linguistic as well. Now going back to the two indices, it is clear that relationship between index A and its meaning system is just as arbitrary as that of index B. Though Index A aims for objectivity since it is compiled through a variety of measures conducted by both professionals and the general public to ensure the accuracy of the ranking, subjectivity is still the fundamental constituent. Nevertheless, the indexical ranking is still unmotivated in the linguistic sense mentioned above, simply a reflection of the outcome.

Index B is also arbitrary even though it is more “objective.” This image shows how LCC classifies different books,

LCC

LCC

Yes, index B is arguably more objective in the way that it treats any given book equally, categorically assigning a random value to one type of book, but the process itself is arbitrary as there is no natural relation between letter and the category.

Value, in these instances, is either a qualitative or quantitative manifestation of a “subjective” or “objective” evaluation. In short, subjectivity and objectivity have nothing to do with the arbitrary nature of signs.

Furthermore, as stated earlier that since indices do not have the physical property to contain value, therefore they do not have value yet retain the ability to signify such.

Artifact and Interface

Then again, what is the difference between LLC and my personal way of categorizing book? The answer is standardization via institution. Institution justifies an arbitrary instance of signifier/signified correlation into an established standard. Bruno Latour’s idea will be very helpful for examining the case here. We delegate the task of categorizing books to LOC, and through standardization, LOC conditions us to understand its classification while freeing us from the labor of using different systems in different places, hence forming a complete circle of meaning system[10].

This, however, is not the only job LOC has. On their official website, one can easily spot the two major functions (ignore the ask a librarian function here):

LOC

LOC

Beside setting standards for book cataloging, LOC is the institution to organize and protect books and doing so, ensuring the integrity of the collective human experience we call the cultural memory[11]. The duality of its function here presents an interesting dialectical dichotomy, LOC as an artifact and LOC as an interface. But before I delve into why it is significant for me to dissociate artifact from interface, I shall first define these two terms in the context of my writing and second admittedly state that these two concepts can be considered as a single instance.

So what is an artifact? Irvine eloquently explains that, an artifact is “something humanly constructed by design.”[12]

Artifacts are the fundamental makeup of the collective human experience, or universal cultural memory. Books, arts, and music are all instances of human artifacts and more importantly vehicles for cultural memory. Furthermore, institutions, like museum, are also instances of artifact, or physical symbol systems[13]. Regardless what the specific type of instance it is, any of these examples shows to possess a physical quality, or in other words, tangible. But artifacts are not just physical manifestation, according to Michael Cole summarized by Irvine, “Artifacts are simultaneously ideal and material. They coordinate human beings with the world and one another in a way that combines the properties of tools and symbols.”[14] [15] The physical quality of an artifact constitutes the “material” description, whereas the concept of interface provides structure for it.

Herbert A. Simon maintains that,

An artifact can be thought of as a meeting point–an “interface” in today’s terms–between an “inner” environment, the substance and organization of the artifact itself, and an ”outer” environment, the surroundings in which it operates. If the inner environment is appropriate to the outer environment, or vice versa, the artifact will serve its intended purpose[16].

Accordingly, there is no a definite division between an interface and an artifact as the former is the ideal manifestation of the latter. However, I argue it is productive to separate the two when considering where value is distributed. For instance, this is a blog documented by a teacher about her students’ field trip. When they visited the LOC, she posted a picture of the Gutenberg Bible and wrote,

screen-shot-2016-12-10-at-1-44-31-pm[17]

A simple analysis here reveals that there are several layers of meaning here.

  1. She values it for its technological significance in the printing history as students were taught in school.
  2. She took the photo because she wanted to document the filed trip.
  3. She understands the historical, spiritual, and religious significance of The Gutenberg Bible as it is being featured in the LOC.

So here is the question, she did not take photo of the Gutenberg Bible because she wants to read it. Gutenberg Bible as the interface is missing in her photo. But to the teacher, there are still enough reasons for her to document her and her students’ close proximity to this artifact. It is the same reason that people want to take selfies with famous paintings in museums, they attribute value to the artifact for its material property not for its interface.

Researcher who are interested in the artifact’s interface might find value in what artifact’s interface mediates. But even in this case, value does not come from the interface, but what the interface indicates.

Case Study

This comparison reveals dual functions of LOC, it is simultaneously a museum and a library.

In order to further explore this idea, I decide to evaluate the distribution of value using LOC as an example and its dual functions as a museum and library will explains this perfectly. Instead of using words, I shall employ annotated system diagrams below to illustrates my thought processes.

711-graphic

If how we perceive value derives from various meaning systems, then the indexical nature of interface would indicate its ability to call upon value and it inability to retain such value. Interfaces are, after all, instances of extended cognition, which takes away physical constraints and maximize our cognitive process.

I will cite Irvine here again to illustrate this point,

So, “meanings” are never “in” any artefact as a physical or material thing; meanings and values emerge in the correspondences in a cultural meaning system that uses clusters of signs in recognizable patterns.[18]

Meaning and value are closely associated with each other. If the interface is jeopardized, the artifact’s meaning system will be lost forever. Through preservation and digital reproduction, the interface of such is protected, hence ensuring the access to the artifact’s meaning system. Benjamin, Malraux, and others’ concerns about technology are valid for the original is eternal better yet Malraux’s notion of musée imaginaire ensures the value of the cultural memory. This video here shows not only books but other instances of artworks can be preserved through digital reproduction. It is true that the value of the original will be decreased once they have been reproduced, yet technology ensures the integrity of its intended purpose, which is invaluable.

[1] Zickuhr, Kathryn, and Lee Rainie. “Younger Americans and Public Libraries.” Pew Research Center: Internet, Science & Tech, September 10, 2014. http://www.pewinternet.org/2014/09/10/younger-americans-and-public-libraries/.

[2] Eaves, Morris. “Blake and the Artistic Machine: An Essay in Decorum and Technology.” PMLA 92, no. 5 (1977): 903–27. doi:10.2307/461845.

[3] Walter Benjamin, The Work of Art in the Age of Its Technological Reproducibility, and Other Writings on Media. Edited by Michael W. Jennings and et al. Cambridge, MA: Harvard University Press, 2008.

[4] Malraux, André. The Voices of Silence. French, Les Voix du Silence, 1951. Translated by Stuart Gilbert. Garden City, NY; repr. Princeton: Doubleday; Princeton Univ. Press, 1953.

[5] Malraux. The Voices of Silence.

[6] CBS Sunday Morning. America’s Film Heritage Preserved at the Library of Congress. Accessed December 10, 2016. https://www.youtube.com/watch?v=SEhMxp2os3M.

[7] Thomson, Stéphanie. “The 20 Most Influential Books in History.” World Economic Forum, November 15, 2015. http://weforum.org/agenda/2015/11/the-20-most-influential-books-in-history/.

[8] Martin Irvine, “The Grammar of Meaning Making: Signs, Symbolic Cognition, and Semiotics.”

[9] Martin Irvine, “Key Writings on Signs, Symbols, Symbolic Cognition, Cognitive Artefacts, and Technology.”

[10] Latour, B. [as Jim Johnson] (1988). “Mixing Humans and Nonhumans Together: The Sociology of a Door-Closer” Social Problems, Vol. 35, No. 3, Special Issue: The Sociology of Science and Technology. (Jun., 1988), pp. 298-310.

[11] Dietz, S., Besser, H., Borda, A. and Lévy, P. (2004) Virtual Museums (of Canada): The Next Genera on, Canadian Heritage Information Network. Available from: <h p://besser.tsoa.nyu.edu/howard/Papers/vm_tng.doc.> [Accessed 9 March 2015].

[12] Martin Irvine, “Working with Semiotic Concepts and Methods: From Peirce to Computer Interfaces.”

[13] Herbert A. Simon, “The Sciences of the Artificial.” Cambridge, MA: MIT Press, 1996.

[14] Irvine, “Key Writings.”

[15] Michael Cole, “Cultural Psychology: A Once and Future Discipline.” 1996.

[16] Irvine, “Key Writings.”

[17] Finnerty, Valerie. “2016 LMS Washington, DC Trip: Day 4!” 2016 LMS Washington, DC Trip, May 26, 2016. http://2016lmsdctrip.blogspot.com/2016/05/day-4.html.

[18] Irvine, “Key Writings.”

A Case Study on Google Arts & Culture (Ruizhong & Yasheng)

A visual illustration of different layers of Interface

TEAM: BEAUTY♡ PRETTY☆ SOCIETY♀ (Ruizhong & Yasheng)

The Princess from the Land of Porcelain (La Princesse du pays de la porcelaine)
James McNeill Whistler
1863-1865
FS-7754_01-651x1024 copy

We decided to use this painting in the famous Peacock Room as an example to demonstrate different layers of interface involved in Google’s Arts & Culture project. We will illustrate different steps of representation in the process of digitalizing the experience of viewing this painting.

Layer 1: The painting itself

The description of the painting can be found on Freer Museum’s website as well as on the Google Arts & Culture site;

This painting, popularly know as “The Princess from the Land of Porcelain”, which hangs over the mantel in the Peacock Room, was part of a series of costume pictures undertaken by Whistler in mid-1860s in which western models appear in Asian dress, surrounded by Chinese and Japanese objects from Whistler’s own collections. Whistler never visited Asia, and his creative borrowing of eastern objects and influences was motivated by a desire to suggest the temporal and spatial distance of a foreign, and therefore imaginary realm, rather than by an interest in Asian cultures per se.

As described in the description, this painting is not intended to showcase affinity towards Asian culture per se, rather a display of the imaginary. Orientalism aside, the painting is quite interesting as its dialectic between the West and the East is represented through both its inside and outside architecture. The inside architecture – the painting as a whole and symbolic element in the composition, generating multiple interpretants to feature the clash between the Eastern and the Western cultures. The image of a white woman in vaguely Eastern outfit standing among Eastern items comprises a number of tokens together function as icons (e.g. images of the Chinese vase), Indices (e.g. the posture of the lady pays homage to the Japanese style – Ukiyo-e), and Symbols (e.g. the mixture of different cultural elements indicate the painter’s desire to showcase his love for arts in all forms).

Now the immediate outside architecture – the golden frame of the painting. The frame of the painting is a solid golden frame with detailed decorations and this frame adds a new layer of interface by enhancing the luxurious quality of the painting.

Screen Shot 2016-11-22 at 9.00.03 PM

So the painting itself is an interface between the author and his audience, through which they can form a mutual understanding of what the painting signifies.

Layer 2: The painting in the Peacock Room and the Freer Museum

A further outside architecture of this painting is also a codependent element – the Peacock Room. Codependent in the sense that they compensate each other to signifier the underlying meaning of this room. From this image we can see that all these objects work together to add a new layer of interface for people to have an immersive understanding of the Peacock Room. Of course, one can make the argument that the room and the painting together is an interface as a part of the Freer installation, but we decided to separate them because we are emphasizing on the painting. From this image, we can see a clear four-step meta happening, we are looking at an interface (the painting) within an interface (the room) that resides in another interface (the museum) through a digital interface (the computer screen).

And this brings us to the next layer of interface – digital representation.

Layer 3: Digital computer interface of painting

We go meta again when we watch the painting on the computer screen in the context of Google Arts & Culture website.

The material screen is a pixel-mapped substrate. The painting itself in fact is represented by a set of pixels. These pixels are organized in a specific way to resemble the original painting. In the computer screen mediated painting, the palette of the original, as well as the figures and objects depicted in the original are highly resembled. Apart from the painting, the explanatory texts and audios beside the digital mediated painting also an example of imitating the description in paper and the curator’s voice in the physical museum. Therefore, the digital representations are taking advantage of preexisting interfaces.

Picture1

But it also has something new, which an actual museum cannot do. In Alan Kay’s vision, this is a win for the virtual museum.Picture2

However, what we see from the screen are a series of semi-static “screenshots” of the painting as well as the surroundings, the settings in the Peacock Room. The 3-dimensional virtual tour of the room imitates the experience of on-site visiting. Watching the painting on screen is definitely different from watching the painting in the Peacock Room. Something might be missing during the remediation, such as the texture of the layered brushstrokes of the oil painting, the smell of the ancient wooden shelves and furniture, the ambient light in the room, etc.

Layer 4: Digital computer interface of Google Arts & Culture

By mapping around the webpage of Google Arts & Culture, the construction of the website serves to guide us around the website by the side bar and the scrolling content.

Sidebar

Sidebar

Scrolling Content

Scrolling Content

First, all these texts and images are buttons that link to another page. This function is realized by the hyperlink. Hyperlink as a meta language has no physical quality, rather an indice that directs you to somewhere else.

Second, the digital representations like icons, indices, and symbols are everywhere on the website. By looking at the side bar, all the icons are taking advantage of the cognitive affordance of other icons on other websites, they are not by any means new.

Take a closer look, the Google Arts & Culture website is a digital representation of a collection of museums in terms of multiple ways of categorizing and sequencing. First, the side bar tries to “guess” what’s in your mind and what might interest you. Second, they try to help you map out how to organize different exhibits. These icons are designed to aid this cognitive process without any physical constraints.

The scrolling content are image driven and divided into different sections so that you can always see more. The design of such website is to let the icons do the work so that the symbol don’t act as a form of “distraction” (Alan Kay).

Conclusion

The Google Arts & Culture project is great in terms of extending and enhancing the influence of museum, but we think it cannot not fully re-mediate or replace museum as the mediate of arts. As we stated earlier, virtual museum, through digital representation, can achieve a lot and even beyond “a lot.” Yet it lacks certain authentic quality – you cannot smell the wooden structure in the room, feel the room with your own body, and see the light bounce off the object. Google Books is a more possible project in the way that most people read book in analog now and maybe it will be the future (though the two of us prefer the tree-killing texture of a physical book) .

Reference:

Martin Irvine, " André Malraux, La Musée Imaginaire (The Imagined Museum) and Interfaces to Art". Overview and Excerpts from Malraux's text.

Presentation (Irvine): "Semiotic Foundations (2): The Museum and Artworks as Interfaces: Metamedia Interfaces from Velázquez to the Google Art Project"

Nancy Proctor, "The Google Art Project." Curator: The Museum Journal, March 2, 2011. http://www.curatorjournal.org/the-google-art-project/.

Martin Irvine, "Working with Semiotic Concepts and Methods: Momentary Concluding Exercises for CCTP-711."

 

Random thoughts – Yasheng

Some random thoughts and questions.

There is a lot to cover for this week so I will just write down whatever pops into my head.

For me, there is a clear progression from myth to enlightenment, in terms of my understanding of computation, taking place while attending this class. Computing makes sense form a semiotic point of view. It makes sense in term of technological progression, from worshiping gods of thunder to manipulation of electricity, from manipulation of electricity to representing thought using electricity. From this perspective, computers are not some alien technology that came out of nowhere, but a natural progression, or in other words, an artifact.

About design philosophy

Murray’s paper excellently explained the design philosopher of modern technology, or technology in general, “in digital as in mechanical design, our goal should be to remove distraction, to allow the user to focus on the task at hand rather than the tool, by making the interface elements transparent, and allowing the interactor direct manipulation, as we do, for example, when we drag a file folder to the trash can” (Murray 61). I understand the frustration that comes with “unfriendly” interfaces because just like anyone else, I grew up using technology without questioning what is really inside the black box. Though I didn’t comprehend the mechanism of any technology, I did know which hardware/software is better simply because it has better affordance. Just look at the evolution of website design, nowadays every “good” website is clear, clean, and image driven. Just look at CNN’s website as an example, they hide all the menus, use images as headlines, and decrease the visibility of text on their website.

CNN Website in 2016

CNN Website in 2016

CNN Website in 2000

CNN Website in 2000

However, the removal of “distraction” can be understood as an act of black-boxing. So my questions here are:

  • Is it patronizing to set the golden design standard as “make it as simple as possible?”
  • Should we focus on raising computation literacy instead of chasing after what the market requires?

On the same topic but a different perspective, I want to talk about the softening line between hardware and software. Murray’s “direct manipulation” also explains why everything these days are either touch screen or voice control. These technologies are great because they eliminate a large number of physical constraints and allow us to do so much more. In this way, black-boxing becomes necessary again. Yet giving it a deeper thought, I wonder do we need to be efficient all the time? I understand great affordance frees our cognition up to do so much more, but the question here is, what can we do? Don’t we need distractions?

Emotional Interpretants

Furthermore could we explore more on the additional interpretants? Peirce describes the division of interpretants as being emotional, energetic, and logical. Logical can be further divided into normal, dynamical, and immediate, energetic into physical and mental, whereas emotional stands on its own. I want to ask about emotional interpretant and what is the relationship between emotional Interpretant and semantics?

  • Emotional Interpretant: A subjective element of thought, a feeling as it occurs at a specific moment, indicating the presence of a manifold of impressions that need to be brought to unity.

Computation is very clear if it is understood in a semiotic fashion, but I have to say that I still have the fear or at least a minor aversion about this “clarity.” This is also why I am intrigued by the concept of emotional interpretant. I still naively think the human experience is too complicated to be fully presented digitally. I guess the idea of digital representation of human emotions is way to down in the uncanny valley for me. Since there are so many progress made on semantics, I wonder if we are getting close to breaking down our emotional experience?

Sorry for the disorganized post m(-_-)m

Use your voice to code? – Yasheng

First of all, I am not saying Echo is a great product and I actually don’t have one because personally I don’t really see the point. But I have to note that Echo is definitely something I did not perceive as the next step in technical advancement. In my mind, the so called “next step” looks more like VR technology where humans directly interact with the symbolic environment, touch, draw, and create new layers of abstractions without any physical constraints like this video:

Voice control, on the other hand, is somewhat imperfect in my opinion because there are so many memes about “Siri fails,” and frankly it looks kind of dump when people are using it… like echo:

Anyways, back to this week’s topic. The readings reveal that the evolution of interface and interaction is not a matter of one way of mapping replacing another, rather a process of softening the divide between hardware and software. For instance, the keyboard we use to type on our smartphones are based on real life keyboard, and that keyboard is just a step forward from typewriters.

In the case of VR and voice control, both technologies fit the description yet have the opposite approach. VR maximizes its affordance by creating a neutral language between human and machine via a “friendly” interface. Voice control technology maximizes its affordance through making the machine function more like human by teaching them syntax and semantics. In my opinion VR is better than voice control because it’s more friendly and direct, which allows me to do more things. After this week’s readings, however, I think voice control might be the future.

Dictation technology exist long ago and I remember my old desktop PC has Dragon Dictation installed when I was learning English. It wasn’t a big deal for commercial purposes until Apple introduced “Siri,” then comes “Cortana” and “Alexa” (All have default female voices and feminine names btw). It seems that modern technology is able to make voice control software more reliable through a lot of engineering and semantic training. And Apple is pushing this idea by implementing Siri to laptop OS updates (Sierra) and creating Apple Watch, which has a small interface so that users are encouraged to give voice commands. Amazon’s Echo pushes this concept further by elimination a visual interface all together, forcing the users to directly “interact” with the machine.

This is where things get interesting, human language has historically been one of the reasons for the separation between hardware and software, because the human language is too nuanced for machine, and machine code is incomprehensible to regular humans. The voice interface between machine and human makes the dynamic between human and machine more natural because it allows machine and human to directly interact at the symbolic level without any physical constraints. With further development in voice control, we might be able to do so much more because we can offload all the complex syntax building labor to machine and directly engage new ideas. So imagine one day we can code using just our voice rather than earning a degreed in computer science or intensive training on Codeacademy.

Python and Computational thinking – Yasheng

The eureka moment came when Evan explains coding and representation work: “instead of enumerating a mapping between all possible character sequences and the natural numbers, we need a process for converting any text to a unique number that represents that text (10).”

Denning and Martell clarifies this process in their writing,

A representation is a pattern of symbols that stands for something. The association between a representation and what it stands for can be recorded as a link in a table or database, or as a memory in people’s brains.

  • There are two important aspects of representations: syntax and stuff.
  • Syntax is the rules for constructing patterns; it allows us to distinguish patterns that stand for something from patterns that do not.
  • Stuff is the measurable physical states of the world that hold representations, usually in media or signals.
  • Put these two together and we can build machines that can detect when a valid pattern is present (372).

These explanations are very useful this week when I was deblackboxing Python in Codecademy.  The second module of Codecademy teaches the newbie coders to use python to calculate tips – an annoying task that happens to everyone who eats out at restaurants that requires tipping. Tipping is a process we all familiar with, and we all know how to do it step by step, and of course sometimes we ask help from the calculator.

Just from looking at the interface of Python, the steps of calculating tips became very clear.

Screen Shot 2016-10-25 at 10.08.13 PM

We, humans who have learned the basic procedures of tipping, first identify what is in play when calculating a tip: how much does the meal cost, what is the percentage of tax, and what percentage should my tip be? These questions are defined as “meal =,” “tip =,” and “tax =.” Then when values are assigned to these variables, an equation is formed to conduct proper calculation based on these variables.

Python and I both calculate tips in a same manner, the only differences are 1. I am not as fast as Python at calculation (I am an Asian who cannot do math well L) and 2. I filter out certain steps in the process of calculation, like defining variables, because I do it subconsciously though it is still very much a step I take in my cognitive functioning. So there is actually little difference between computation thinking and human thinking. Wing puts it perfectly, “A way that humans, not computers, think. Computational thinking is a way humans solve problems; it is not trying to get humans to think like computers (35).” When we have a problem, we identify it, isolate different elements that are involved, and follow a logical sequence to solve it. Or put in Denning and Martell’s terms, Python and the humans employ same syntax when conduction cognitive works yet the stuff human have can be more nuanced.

Questions:

What if a pattern that is not visible?

What happens when the “stuff” is too nuanced for computers to interpret?

Venmo’s Affordance (Ruizhong & Yasheng)

TEAM: BEAUTY♡ PRETTY☆ SOCIETY♀ (Ruizhong & Yasheng)

The technology we are analyzing is Venmo.

Venmo allows people to conduct small to medium scale of financial transaction with the experience of social media.

After the user allowing Venmo to access her bank account, the users off-load tasks with the bank to Venmo. This allows users to directly transfer money to another person.

Venmo helps to eliminate the process of using actual money to pay back someone or going to ATM and withdrawing money, it allows direct transaction bypassing time and space constraints.

Think about a scenario where you have to pay someone back, you need to have a quantifier to signify the value you owe that person. That’s the cognitive function of paper bill in the sense that it allows people to easily conduct value transaction with certainty.

However, the reality of using money, or paper bill to be more specific, is that there are a lot of physical constraints, including organize bills, deposit/withdraw them from bank, and having extra change to allow accurate transaction. This system diagram showcases the complicated steps of paying someone back (made by Ruizhong).

WithoutVenmo

Venmo eliminates the physical constraints of paper bill, offloading the tasks of using paper bills, withdrawing money from the bank, and having to talk to person about financial transaction.

We, humans, delegate several processes of value transaction to Venmo, the technology, so that we are free of the labor of financial transaction. This system diagram showcases the simple steps to allow financial transactions (made by Ruizhong).

WithVenmo

Venmo functions as an extension of our cognition in value transaction in terms of calculating exact amount without worrying about if that person has the exact change, or the awkwardness of having to remind someone to pay back. Furthermore, Venmo takes advantage of the light-heartiness of social media by taking off a lot of social pressure and anxiety of financial transaction.

The Affordance of Venmo

We decide to use the interface of Venmo as an example to demonstrate the affordance of its interface design.

Diagram

Reference：
Andy Clark and David Chalmers. "The Extended Mind." Analysis 58, no. 1 (January 1, 1998): 7–19. 
James Hollan, Edwin Hutchins, and David Kirsh. “Distributed Cognition: Toward a New Foundation for Human-computer Interaction Research.” ACM Transactions, Computer-Human Interaction 7, no. 2 (June 2000): 174-196. 
Jiajie Zhang and Vimla L. Patel. “Distributed Cognition, Representation, and Affordance.” Pragmatics & Cognition 14, no. 2 (July 2006): 333-341.
Itiel E. Dror and Stevan Harnad. "Offloading Cognition Onto Cognitive Technology." In Cognition Distributed: How Cognitive Technology Extends Our Minds, edited by Itiel E. Dror and Stevan Harnad, 1-23. Amsterdam and Philadelphia: John Benjamins Publishing, 2008.

png vs. jpeg

So one of the best memories of taking 505 in CCT was FINALLY understanding how jpeg works. It actually felt like opening a black box and unlocked the secret of a magician.

With one question answered I have a dozen more. I started to wonder how other image formats work, especially why PNG can create transparency in image and jpeg cannot. Hence I did some research and found out why. Instead of using difficult jargons, I will try to explain the difference between PNG and jpeg through my crude words.

So jpeg is made with different segments, each segment has a marker in the beginning telling what color a pixel is and how many pixels of the same color exist sequentially. These segments are structured using binary codes. With the correct way of decoding, one will be able to view the image on any device that supports jpeg. Because jpeg’s syntactic structure, it will be difficult to represent absent of color and how absent some colors should be, therefore it is difficult to represent transparency.

Though PNG is also coded in a binary manner using only 1 and 0. The structure of PNG allow it to do more than just tell what color a pixel is and how to arrange the order. Each segment in PNG has a special task such as organize the frame, adjust the color, and how much space is between one color and the other. Therefore, PNG can have transparent background and jpeg cannot.

What is so cool about these two format is that one image can totally different syntactic structure in different formats. This also an evidence to Pierce’ claim that signs as vehicles do not possess meanings. Like these images, they are only meaningful to us when they are properly decoded, or decompressed in this case. Taking a photo and make that into a jpeg means to translate that into a string of binary codes. These codes become vehicle of what the photo represent. These codes are then decoded on our computer and then we can interact with the image and create meanings out of it.

It is remarkable how much we can achieve through digital representation, and how much we can make the convey of meaning more efficient and accurate. Here is an example. The video shows the compression speech difference between PNG and FLIF, a newly developed image format.

Though the meaning we can get from these images are the same, the speech of FILF makes the delivery of meaning so much faster and that is meaningful in a different manner.

Understanding symphony through semiotics – Yasheng

This week’s readings are very technical and I am not sure if I understood them 100%. But I will attempt to illustrate how Jackendoff’s parallel architecture of language can be extended to music.

My wonderful friend Iva often invites me to performances by the National Symphony Orchestra at the Kennedy Center. Because I don’t know a lot about symphony or classical music in general, I often rely on the program booklet for ways to appreciate the performance. One of the most hauntingly yet disturbingly beautiful piece I’ve heard is the second movement of Dmitri Shostakovich’s 11th Symphony entitled “The Ninth of January.”[1] This movement depicted the massacre of protesters carried out by the Tsarist autocracy in 1905. One specific segment of this movement especially helps to illustrate the “hauntingly yet disturbingly beautiful” quality.

To demonstrate the meaning of this movement, I will attach an excerpt of the program here,

Allegro (The 9th of January)

The second movement, referring to the events of the Bloody Sunday, consists of two major sections. The first section probably depicts the petitioners of 22 January 1905 [O.S. 9 January], in the city of Saint Petersburg, in which crowds descended on the Winter Palace to complain about the government’s increased inefficiency, corruption, and harsh ways. This first section is busy and constantly moves forward. It builds to two steep climaxes, then recedes into a deep, frozen calm in the prolonged piccolo and flute melodies, underscored again with distant brass. Another full orchestra build-up launches into a pounding march, in a burst from the snare drum like gunfire and fugal strings, as the troops descend on the crowd. This breaks out into an intense section of relentless strings, and trombone and tuba glissandos procure a nauseating sound underneath the panic and the troops’ advance on the crowd. Then comes a section of mechanical, heavily repetitive snare drum, bass drum, timpani, and tam-tam solo before the entire percussion sections breaks off at once. Numbness sets in with a section reminiscent of the first movement.[2]

The highlighted sentence coordinates to 11:00 to 15:09 in the YouTube video I shared above, and it is the segment I find “hauntingly yet disturbingly beautiful.” I will now attempt to explain processes, in my opinion, of the creation and appreciation of this segment in a semiotic fashion.

During the production of this segment, Shostakovich treated the sound of snare drum, bass drum, and others as icons to mimic the sound of firearms on the day of the massacre. Sounds coming from these instruments are “perceived as resembling or imitating the signified, being similar in possessing some of its qualities.”[3] These sound patterns are then choreographed by Shostakovich into different notes on a score. This can be perfectly explained through Jackendoff’s Parallel Architecture.[4] The process of representing one sound pattern using another is visible at the interface between the phonological structure and the conceptual structure. And the process of Shostakovich organizing different sound patterns into music notes can be perceived as the syntactic structure interfacing the conceptual structure. Furthermore, these music notes remain as abstract representations until they are played by their associated instruments – just like a speech remains as an abstract representation until it is orally delivered by someone.

On the receiving end, the process of decoding requires various layers of interpretations of different “stacks” in the structure of this segment. I commented on the piece using the phrase (or symbol in Pierce’s term) “hauntingly yet disturbingly beautiful” earlier, indicating that there are (at least) two processes of interpretation happening when I was listening to this segment.

  1. I, without knowing the context of the symphony, found the music very engaging thanks to the drums and other instruments playing in a rapid fashion. [syntactic-phonological interface]
  2. I, understanding what the icons represent, found the music disturbing as it revokes my feeling towards death by shooting. [phonological-conceptual interface]

The first interpretation comes from direct interpretation of the music, whereas the second interpretation is aided by the program booklet. Hence, I arrive the conclusion that Jackendoff’s parallel architecture model can be extended to this particular instance and even to other instances of music in different manners. This can be used to demonstrate why music functions like a language.

[1] Laurel E. Fay, Symphony No. 11 in G minor, “The Year 1905,” Op. 103 (1957), American Symphony Orchestra Program Notes

[2] https://en.wikipedia.org/wiki/Symphony_No._11_(Shostakovich)

[3] Daniel Chandler, Semiotics: The Basics. 2nd ed. New York, NY: Routledge, 2007. Excerpts.

[4] Ray Jackendoff, Foundations of Language, selections on the “Parallel Architecture” model of language as a combinatorial system. Chap. 5.5, pp. 123-128; Chap. 7, pp. 196-200.

I promise there are logical sequences between my paragraphs – Yasheng

What is Language?

Before feeling sad and confused by all the linguistic math, I came across this post on Tumblr (best place to procrastinate):

  • Am I the only one that finds it weird that I can transfer data from my brain to someone else’s by opening my mouth and pushing air with vibrations in their direction.
  • How high are you?
  • 5’4”

LOL aside, whoever this person is, s/he is correct. In the wonderfully informative video by Steve Pinker with the good hair, he explains that the noises we make using our mouth are called language, which we use to share ideas and create meaning. [1] There is, of course, a difference between written language and spoken language within one culture. Like I can write s/he, but I have to say she and he when I am reading it out loud. Despite this difference, both written and spoken language follow one specific grammatical structure. Language is symbolic in nature, which allows us – beings with semiotic competence – to understand each other as long as we are following certain grammatical and contextual rules. But if someone were to ask me about what language is, I’d probably refer them to the Tumblr post I cited earlier.

Though I just complained about how I am not good at math and the all the logic equations made me sad, I did found Jackendoff’s parallel architecture exceptionally useful in explaining how we use language once I understood (I hope I did) how parallel architecture model works.

“…language as a whole can be thought of as a mapping between sounds and meanings; phonological structure is the specifically linguistic encoding of sounds, and conceptual structure is the encoding of meanings. Syntactic structure serves as a ‘way- station’ between these two structures, making the mapping between them more articulate and precise.”[2]

This quote perfectly explained the components of language and this makes me wonder, can we use this mapping to understand other things?

What about idioms?

Jackendoff’s writing made me thinking about idioms, especially because idiom’s syntactic structure is somewhat unique. Idioms make the language richer, more colorful, and unique, which makes translation of idioms often difficult. Because of this uniqueness, idioms are exclusively connected to their culture.

I am wondering what idiom’s CC-CS interface looks like because idioms have to be within certain allowance of its syntactic structure to be understood. For instance, “it’s raining cats and dogs” makes sense to most English speakers, yet how big of change to this expression can make it less comprehensible? Would the phrase “it’s raining pigs and goats” convey the same meaning? Would the phrase “it’s pouring cookies and cream” convey the same meaning?

Another question: I am also curious about what idiom’s interface between semantics and pragmatics look like? I think I will read chapter 6 to find out.

What about Chinese idioms?

I know I am getting off topic, but this has been on my mind ever since I started reading syntactic structure. So Mandarin Chinese’s grammar is modeled after English grammar and tMandarin Chinese is digraphia for it has both Chinese characters and pinyin (Romanization system for Chinese characters). Because we now learn Chinese using grammar modeled after English, we constantly need to make exceptions or introduce new grammatical rules when learning idioms and ancient Chinese (Wen Yan Wen).

Here is an example:

Screen Shot

贤妻良母：Xian (Smart) Qi (Wife) Liang (Good) Mu (Mother) is an idiom to describe virtues of women (very patriarchal and Confucian indeed). Break it down, we get:

1111

Seems pretty straight forward right? But there is a special grammar ancient Chinese often use (mostly in Min Dynasty: 1368–1644) called Hu Wen. This is when the adjectives in a phrase simultaneously modifies nouns, noun phrases, or other elements. So if we follow this logic, the phrase 贤Xian (Smart)妻Qi (Wife)良Liang (Good)母Mu (Mother) becomes difficult to represent in English grammar because English syntactic structure is linear. When I rearrange Xian (Smart) Qi (Wife) Liang (Good) Mu (Mother) following Hu Wen, the meaning of such phrase becomes: smart and good wife / smart and good mother. Two virtues for being a woman as two roles. The syntactic structure should look like this:

2222

Why the circles, you may ask. The outer circle is similar to the function of S to represent the idiom as an independent phrase. The middle circle represents the two adjectives simultaneously modifying the two nouns. And center empty circle represents women as the invisible implied subject of the idiom. This makes the idiom more powerful and meaningful as it is in its original context.

The unfortunate truth is that we do not know if the person who created this phrase meant as a linear manner or Hu Wen manner. Because Chinese has changed a lot to chase after global standardization, the Chinese language might have lost something in this process.

Syntactic structure for art?

Chinese idioms are considered as a form of art in China alongside of painting, music, opera, and so many others. Chinese aesthetics are imbued with the concept of emptiness – the invisible and intentionally emptied space in things. Because of this, how can syntactic structure to illustrate such emptiness and can it be represented as computing codes for others to appreciate?

[1] Big Think. Steven Pinker: Linguistics as a Window to Understanding the Brain. Accessed September 19, 2016. https://www.youtube.com/watch?v=Q-B_ONJIEcE.

[2] Ray Jackendoff, Foundations of Language: Brain, Meaning, Grammar, Evolution. New York, NY: Oxford University Press, USA, 2003.

Language is a Technology – Yasheng

Deacon notes human language is unique to human while adding that what other animals use to communicate is either inferior or superior.[1] Maybe I watched too many whale documentaries (#FreeTilly), one thing these documentaries all claim is that whales and dolphins have more developed brains and can communicate better than humans. 5e4dc177b5303d2a9f61dbba3f2a2f09Living in a world without air to transfer sound, whales are able to create meaning, value, and social structure within their small communities. What we read last week from Saussure entails that language, as a signifier, can never fully represent our inner thoughts, the signified. So in a way, our language is not the perfect tool for communication for the lack of full representation.

Deacon also notes that humans understand the world through matter recognition and symbolic representation – language. If our inner world is too complex to be fully represented through symbols, and pattern recognition is also limited by context. In other words, the original meaning  of a message gets lost twice, first time when the sender is encoding her or his thoughts into language, and the second time when the receiver is trying to decode the symbolic meaning of what the sender is trying to express. Accordingly, wouldn’t it be that case that language actually creates isolation between self and other, rather than engender empathy among the two?

Also consider the fact that computing is based on human language, therefore it also lacks full representation of the signified. So how can computing makes us better at establishing effective communication with others?

Well, on the other hand, I guess one thing whales cannot achieve is passing on their existence through their communication tools, or at least as far as I know it. Humans, on the other hand, are able to pass on knowledge through the language we speak. In a way, what we use to communicate on a daily basis is part of the collective human experience. Donald claims that individuals become more powerful by connecting to their “culture network”.[2] From all the evidence presented by Wong, it is clear to see that many early human cultures evolve exponentially thanks to the advancement of their culture network. Different cultures create different traditions, contrary to the West, many Asian languages encourage group mentality and indirectness. The Eastern speech is delicate and I remember that is one of reasons Japanese people don’t really get sarcasm (This is not a joke).

Another fun fact about Eastern language, Chinese in this case, I want to bring up comes from a psychological research done by my previous professor. The research shows that,

Chinese speakers had stronger connections leading from an area of the brain called the anterior superior temporal gyrus – which has been identified as a “semantic hub” critical in supporting language – to both Broca’s and Wernicke’s area. This increased connectivity is attributed to the enhanced mapping of sound and meaning going on in people who speak tonal languages.

The second difference showed activation in an area of the brain’s right hemisphere, but only among the Chinese speakers. This brain area, the right superior temporal pole, has been implicated in Chinese tones before but – perhaps more importantly – has until now been considered completely separate from the classic language network in the left hemisphere.[3]

This makes me wonder if there is a technology that builds Eastern language processes, would it be something different than what we have today.

[1] Terrence W. Deacon, The Symbolic Species: The Co-evolution of Language and the Brain. New York, NY: W. W. Norton & Company, 1998.

[2] Merlin Donald, “Evolutionary Origins of the Social Brain,” from Social Brain Matters: Stances on the Neurobiology of Social Cognition, ed. Oscar Vilarroya, et al. Amsterdam: Rodophi, 2007.

[3] Taylor, Larry. “If You Speak Mandarin, Your Brain Is Different.” The Conversation. Accessed September 11, 2016. http://theconversation.com/if-you-speak-mandarin-your-brain-is-different-37993.