Author Archives: Ruizhong Li

Mediating the Social-Cultural Function of Museums: A Case Study of the Digital Exhibition of “Landscape with the Fall of Icarus” – Ruizhong Li


In this paper, I am going to analyze an online exhibition of the painting Landscape with the Fall of Icarus as:

  1. A digital artefact mediating the social-cultural function of museums
  2. An interaction design as a cultural practice

The paper will include:

  1. An illustration of the multilayer of interfaces embedded in this digital artefact and the affordances implemented by this digital exhibition;
  2. An analysis of how this online exhibition remediated the social-cultural function of museums; what changes brought by the digital museum idea to on-site museum-visiting experience and human cognition of art history.


Digitization has made it possible for human to tour the world without leaving their desk. Digital representations can remediate almost every material artefact bit by bit. Remediation, in essence, is a process of creating an embedded structure of meaning systems as time goes by. Therefore, in order to de-black the remediating process, we need to figure out what’s going on in each layer of interfaces and how they connect to the larger meaning systems. The digital exhibition I want to discuss in this paper is a case of mediating a material painting in a physical-existed museum, a meta-(already)-meaning system. Therefore, before I go further into looking at the remediation of the social-cultural function of museum (as suggested in the title), I would do an visual illustration of symbols remediated and affordances implemented in each layer of interface. Then, based on the detailed illustration, I will do an analysis on how this online exhibition remediated the social-cultural function of museums. Done with every piece of detail about this digital exhibition, I want to compare it to another form of digital representation within a larger realm of virtual experience, to see what might be a special application of the technical mediation technologies for this instance. Finally, I will extrapolate from the case to discuss the differences between the virtual experience and on-site experience of visiting museum, and how digital exhibition changes human cognition of the art history.

Multilayer interfaces of the Digital Exhibition

“All human symbolic activity and representation in any material form are artefacts of symbolic cognition in a long cumulative continuum of technical mediations for human sign systems.” – Martin Irvine

Computation, software, digital media are artefacts of human symbolic cognition, and thus all computing is fundamentally “human(istic) computing. Human sign system is the fundamental meaning system underlying all of the technology mediated platform. Therefore, I will start off from the “human” interpretation of the painting and go further into the computer-mediated meta world.

First of all, for background information about the painting, please click here for details.

  • Layer 1  The Painting: Landscape with the Fall of Icarus

Fig 1. Landscape with the Fall of Icarus (undated)

The painting Landscape with the Fall of Icarus, featuring a subject from Greek mythology, depicted the fall of Icarus described by Ovid in his Metamorphoses. The painting included almost all the elements in the story, but at the same time, it is a very personal interpretation of the story.


Fig 2. Bruegel’s interpretation through composition

The composition suggests that what Ovid accentuated in his passage, the fall of Icarus, was understated by Bruegel (the painter). In the bottom right-hand corner of the painting, the legs of Icarus himself can be seen desperately flailing in the air. What’s more, the rest of the world remains unperturbed; the ploughman, the shepherd, even the fisherman were showing indifference to the fall of the mythical hero.

According to Peirce’s triadic model of the sign as a process with material forms, the following diagram presents and explains the ObjectInterpretantRepresentamen, and the Semiosis.


Fig 3. Semiosis of the Painting

The Object in this symbolic productivity process is the original Greek mythology, which is already a meaning system organizing assorted folklores into a formalized framework. Interpreted by Bruegel, he decomposed the established institutions and relationships of the characters and elements (the symbols) in the story and reconstruct the symbols into a personal interpreted meaning system via the proportion, composition, palette, etc. Understating the fall of Icarus, Bruegel emphasized the landscape and other characters in the painting. In terms of proportion, Icarus takes up only a small space of the painting while the other characters in the foreground and the panorama of the port in the background take up a large proportion of the whole painting. In terms of composition, the character of Icarus was depicted in the shadow on the right-bottom end of the diagonal – the least conspicuous position. In the foreground, Bruegel depicted a tangible reality that ordinary people were minding their own business: ploughman was steering his plough, shepherd was gazing at sky while grazing his sheep, and fisherman was engrossed in his toil; none of the characters pay attention to Icarus flailing in the water. In the background, Bruegel delineated a panorama of the port, island and the surrounding town, which shift audience’s attention to the marvelous landscape of the vast ocean. In terms of palette, Bruegel used three hues one after the other (browns, greens and blues) to create the impression of depth. Icarus was on the blurred boundary of two hues, where hardly can audiences notice. These correlation-making components reflected Bruegel’s responses to the story, which function as the Interpretant in the symbolic activity. The painting, the one hanging on a wall in the Royal Museums of Fine Arts of Belgium is the Representamen. It is the “sign vehicle” representing the material-perceptible structure with the interpretable features in the painting realm.

The painting provided a personal interpretation of the Ovid’s passage, which is a special genre of the Greek mythology. It is not a direct interpretation of the original story, but it serves to present an interface for audience to understand the Fall of Icarus story via a medium of painting.

  • Layer 2  The Painting in the Museum
Fig 4. Brugel Artworks Exhibition in the Royal Museums of Fine Arts of Belgium

Fig 4. Brugel Artworks Exhibition in the Royal Museums of Fine Arts of Belgium

In the room where exhibit Bruegel’s painting artworks, the paintings are decorated (or protected) by wooden frames hanging on the wall. The walls are painted in peaceful green, providing a harmony tone for better visual experience for audiences while watching the artworks. Paintings are hang on the wall aligned with a horizontal line at a suitable height according to people’s watching habit. The paintings are not tightly attached to the wall; instead, in order to solve the problem that the painting may reflect light, there is certain degrees between the plane of the painting and the wall. In terms of location of each work, the most famous and delicate artwork of Bruegel, The Fall of Rebel Angel is placed at the center of the wall, which is faced with the designated auditorium (the deck in the middle of the room). Beside this painting, there are Landscape with the Fall of Icarus (right) and Winter Landscape with Bird-trap (left). These paintings work together to serve to map audiences into another meaning system to interpret any one of these paintings. The painting as a whole is a single symbol of the “room meaning system”. For instance, Landscape with the Fall of Icarus functions as a related artwork to the middle painting for the same topic of the fallen angel, at the same time, serves to guide audience through the room to the next painting, leaving room for audiences to imagine the possible correlation between these two artworks.

The painting Landscape with the Fall of Icarus with the wooden frame is a part of the Bruegel’s room, and functions as an interface for audiences to understand the larger meaning system of Bruegel’s artwork collection. Observed from the photograph (in fact, a digital copy) above, we have a general feeling of the installations of the Bruegel’s artwork room. Unconsciously, we have interpreted the painting through a digital interface, the computer screen. This brings us to the next layer of interface – digital representation.

There are so many possible ways to present a painting in a digital form, photograph, video, even audio commentary. The digital exhibition Landscape with the Fall of Icarus (… and the surrounding controversy) combines several media together to present a in-depth interpretation of the painting.

The digital exhibition is a part of Bruegel Unseen Masterpieces projectThe project is presented by the collaboration of The Royal Museums of Fine Arts of Belgium and Google Arts and Cultural. 

“Drawing on a wide spectrum of virtual and on-site experiences, this unique initiative offers everyone the chance to immerse themselves in Bruegel’s works by honing in on the details of each painting and accessing expert knowledge. By delving deeper into the artist’s world, the viewer will discover the unexpected elements in Bruegel’s works which constitute the pinnacle of the Flemish master’s craft. … This innovative concept is the fruit of in-depth thinking on current transformations in the field of museology as it adapts to the digital era.

From the description, we instantly know that we have immersed in a 3-step meta process when we watch the painting (layer 1) that resides in the museum (layer 2) on the computer screen (layer 3) in the context of the digital exhibition as a part of the whole project.

Accessing the Bruegel. Unseen Masterpieces virtual exhibitions on the Google Arts and Culture platform, we are interact with a material screen which is a  pixel-mapped substrate. What makes the digital representation different is the difference in material representamen. When visiting the on-site museum, what we see using our naked eyes is an artwork painted on a canvas using oil painting. However, when we watch the painting in the online exhibition, we are looking at a set of pixels, which remediate the original painting into a digital copy. These pixels are organized in a specific way (abstraction, recursion, …) to resemble the original painting.

Apart from the painting itself, other functions of museum are also imitated by the virtual online museum. The commentary text and video helps explain the composition and the dynamic path that eye follows across the composition, which is also an example of imitating the description in the card and the curators in the physical museum:

Fig. 5 Christine Ayoub, guide at the Royal Museums of Fine Arts of Belgium, explains the path that the eye follows across the composition

The sequence of the exhibition and other paintings involved in the digital presentation is an example of imitating the installations in the on-site museum as well. In the physical museum, paintings are organized by author, at the same time, paintings of the same authors are placed according to specific logic. In the digital exhibition, paintings that are related to the presented painting, according to assorted needs, will be added into the exhibition:

Fig. 6 Related paintings are added according to assorted needs: 1) A parallel painting displaying the composition of sweeping landscape; 2) Bruegel's interest in depicting port; 3) A painting adapting from the same Greek mythology

Fig. 6 Related paintings are added according to assorted needs: 1) A parallel painting displaying the composition of sweeping landscape; 2) Bruegel’s interest in depicting port; 3) A painting adapting from the same Greek mythology

Therefore, the digital representations are taking advantage of preexisting interfaces.

But it also has something new, which an actual museum cannot do. In Alan Kay’s vision, this is a win for the virtual museum.

The high resolution of the photographic copy of the painting allows people see even more details than what we can see using our naked eye in the museum. With the zoom-in function, we can even see the little cracks of the oil paints:

Fig.6 Look closer at the masterpieces

Fig.7 Look closer at the masterpieces

Also, the digital exhibition enable to guide audiences to specific part (often the neglected details) of the painting:

Fig. Discover the secrets

Fig.8 Discover the secrets

The technical mediation enable audiences to see more than we could see: the head of a man lying in the undergrowth, the horse equipped with blinkers, a knife and a sword on the same rock, a seed bag leaned against a rock, the partridge, the fisherman, the mythology hero falling into the water, the dark ewe stands amongst the sheep, the mediating shepherd, the wind-filled sails and masts, the remote island, the surrounding town, and the sun disappearing over the horizon – all these details are emphasized by zooming in, which enable audiences to get a detailed appreciation of the intricate painting.

Affordances of the Digital Exhibition

"Think of the computer not as a tool but as a medium." - Brenda Laurel

The platform for the digital exhibition resides in computer. “All digital artifacts are made of a common substance: programmable bits that can be used for symbol manipulation.” In essence, computer is a common medium of representation. According to Murray(1997), computer is encyclopedia, spatial, procedural and participatory. Based on the four representational affordances, the following analysis of the digital exhibition will focus on the interaction design as a cultural practice.

  • Procedural Affordances

The digital exhibition is able to represent and execute conditional behaviors. The experience of visiting the digital exhibition of the painting takes the format of sequential slideshow.

The progress bar at the bottom is based on the metaphor of any sequential visual medium, exploiting the procedural (and participatory) affordances of the medium:

Fig. 9 Progress bar

The progress bar is the embodiment of the abstraction and algorithm of the dynamics of this digital presentation. Interacting with the bar, audiences are interacting with a conceptualized model of executing conditional command initiated by the audiences. The jump-off window suggests the flexibility of the program, providing possibilities for audiences to jump back and forth between sections in a unisequential design. Also, the jump-off window is an abstraction of that specific slides, which signifies that specific slide and gives audiences a preview of the content.

  • Participatory Affordances

“The relationship between the interactor and any digital artifact is reciprocal, active, and open to frustrating miscommunication.” The participatory design concept of this digital exhibition is displayed by the automatic language switch based on the language settings of different login accounts. This is one of the preset algorithm.

Sometimes the script is more flexible. Some digital conventions are so familiar that they script us in a transparent way. For example, on the first page of the digital exhibition, there is an arrow on the right of the screen:


Fig. 10 Start arrow

To start off, audiences automatically relate the arrow with the starting: the arrow at the right edge of application windows cue us to scroll the screen; when we put the cursor on this symbol, the arrow turns into a hand, which cues us to click on the symbol.

The same logic, the restart arrow serves to the restarting of the procedural affordances.

Fig. 10 Restart arrow

Fig. 11 Restart arrow

The loading circles signifies “please wait”:

Fig. 11 Loading circles

Fig. 12 Loading circles

These digital conventions aids the participatory design; the interactive design employs these conventions to stimulate human actions to realize their expectations.

  • Encyclopedic Affordances

The word encyclopedic emphasizes on computer’s capacity of storage and transmission, and its inheritage of the tradition of knowledge collecting, preserving, and transmitting.

The digital presentation of this painting is a part of a larger project remediating Bruegel’s artworks. Within the large project, exhibitions are linked one another according to meticulous segmentation and classification. For example, exhibitions can be classified by stories:


Fig. 13 Exhibitions organized by stories

Videos embedded in a specific exhibition could be reorganized into a video section:


Fig. 14 Scattered videos are reorganized into a section

Landscape with the Fall of Icarus, together with other paintings consist of this collection of paintings with the label of Bruegel’s artworks. However, this is not the only way for Landscape with the Fall of Icarus to present among numerous paintings. For example, this painting can be assigned to another project collecting all the adaptions of the Greek mythology, which can include music works, poems, passages, etc. The flexibility in positioning the paintings in the huge collection, displays the inclusiveness of the encyclopedic affordances provided by the computer.

  • Spatial Affordances

Computer creates virtual spaces that are navigable by the interactor, which rests upon the procedural and participatory affordances of computation.

Visual design manipulate the space to represent the hierarchy of the items. By looking at the horizontal-placed paintings, we know the equal relationship among the paintings; the consistency in design suggests the equal level in the structure. We are not going to lose navigation during the browsing experience, because the coherent spaces and the scripted digital conventions.

A remediation of the social-cultural function of museums

The Museum is an organizational system. How museums organize the paintings is highly correlated to the social and culture institutions. Museums can be considered as implementations of human cognitive art history. This predominant museum idea “preceded and pre-interpreted any artefacts selected for representation”. Museum establishes the standard: it set up limited categories for selecting and assigning artefacts to some established categories (which we take for granted now): periods, styles, genres, cultures, etc. The institutionalized idea of museum makes it properly to suggest that museum is an ideal meaning system: it turns the actual museums into embodiments of its conceptual model, in the way of determining what to present and how to present.

In terms of the social-cultural functions of museum, the most primary goal for museum is to provide a ordered representation of art history. Most of the museums are divided into relative independent spaces in order to enable visitors to make sense to the installations and organizations of the artworks. Each space creates a real space for a meaning system, in which establish correlations among artworks within the same space. The meaning system is open, allowing unlimited interpretations of each artwork itself and the relationship between any pair of artworks. The meaning system is going through an on-going changing process, for the possibilities of occurrence of further interpretants expressible in new or additional signs.

How to evaluate the performance of the digital exhibition in fulfill the social-cultural functions of museums? The digital exhibitions earn advantages in flexibility, accessibility, and compatibility.

Flexibility. “Artworks continue to be receives as art works by means of further technological mediation and representation, but are also continually reinstantiated as art works by the institutional framing of “art history” and the museum function in culture.” There are so many ways of presenting only one painting. For Landscape with the Fall of Icarus, it can be assigned to the collection of Bruegel’s artworks, it can be assigned to the collection of Renaissance period paintings, it can be assigned to the collection of artworks adapted from Greek mythology. What digital museum can do is to reorganize the collections by manipulate symbols on the website, totally getting rid of the trouble caused by the material artworks.

Accessibility. The virtual exhibition helps the artworks go beyond the boundaries of time and space. Make the artwork accessible to people with a terminal device with Google Arts and Culture platform.

Compatibility. With multimedia, text, image, audio, video, involving in the digital exhibition, these media also present an embedded relationship in the exhibition. A text annotating the image, a video commentary interpreting the character in the picture … all these work well together thanks to the inclusiveness of the semiosis – an emergent process.

Concerns. Digital museums exist in an intricate network connecting to unlimited cultural symbols and dynamic meaning systems. When we depict the digitalizing process of museum, we always use the word “simulation”. Even with the high level of resemblance, when visiting virtual museum, we are still conscious about something different. It suggests the deficiency in the current condition how human manipulate symbols in an opening and emerging interface. Computing is fundamentally “human computing”, the reason we feel it is “non-human” is related to our current ideological, political-economic conditions, and processes of education and socialization about computers. Even though we have concerns, we are always open to the possibility of remediating human meaning system into computer version. Digital museum is never an enemy, so does computer.


Clark, A., & Chalmers, D. (1998). The Extended Mind. Analysis, 58(1), p. 7-19.

Janet Murray, Inventing the Medium: Principles of Interaction Design as a Cultural Practice. Cambridge, MA: MIT Press, 2012.

Martin Irvine (2016), André Malraux, La Musée Imaginaire (The Museum Idea) and Interfaces to Art. Communication, Culture & Technology Program, Georgetown University.

Martin Irvine (2016), Introduction: Toward a Synthesis of Our Studies on Semiotics, Artefacts, and Computing. Communication, Culture & Technology Program, Georgetown University.

Being mediated … we are getting used to it

We are living an increasingly mediated life. As we intensively interact with computers in our daily life, two sets of symbols, human meaning system (usually referred to natural language) and “computational” meaning system (mediated meaning system simulating and representing human meaning system), are being used simultaneously and interactively. In fact, even without physically manipulating a computer, habitually, we are using computational thinking to solve problems, in which we are mediated by the technologies.

This mediation has been institutionalized; it has been collectively and culturally accepted after intersubjective practice. The reason why we are happy with interacting with computers is that we can gain efficiency from this process to some extent. The relative efficiency is realized through the affordances provided by the technologies. By offloading or distributing some part of our cognition functions into physical technologies, we are externalizing cognitive process, which was believed to be “imperceptible”, into perceptible devices. This embodiment is a remediated process and realized by the invisible and intelligible affordances. As we mentioned in the post of week 9, Venmo provides the affordances of calculating, initiating transaction, commenting, etc. While using this application, we are getting used to the icons, index and symbols intuitively in a human thinking way. If the process of accommodating ourselves to the context of this application is natural and smooth, we consider this application as a good example of humanized technology, which narrowing the gap between human and technology. If we get lost in mapping ourselves into this new set of symbols, the application is failed in simulating and remediating human semiosis.

It is why we are limited by the affordances as well. I happened to read a paper called “Are digital media institutions shaping youth’s intimate stories? Strategies and tactics in the social networking site Netlog” (Sander De Ridder, 2015). In this paper, Ridder insisted that SNS has established some institutions to shape youth’s intimate storytelling online. She also argued that the digital media reproduce the mainstream culture and make it even more prevailing. The “strategies” refer to the software design of the SNS, which provides affordances or options for audiences to self-represent themselves online. The “tactics” refer to audiences’ responsive behavior to accept or resist this predefined software context. Online storytelling involves information representation and information process as many other mediated symbolic cognition processes do. Researching how digital media institutions shaping online storytelling inspires me a lot to think about how we are limited by the affordances. For example, on Facebook, formerly we are only allowed to choose between male and female – just like there are two sides of a paper. Now, we are glad to see that we can custom our gender, getting rid of the “limitation” on self-representing ourselves online to some extent.


I think it is a vivid example of how computational context or digital media shape our information representing via predefined affordances.

The influence brought by the technology can also be observed in the case excluding the interaction with computers. Human’s (only human) computational thinking process is an example of how human are jumping back and forth between our own meaning system and “computational” system. The redefinition of computation and the term “computational thinking” put the emphasis on information representation, which is more inclusive to engage the human agent in this process. Different from delegating agency to the technologies and using them to extend and distribute cognitive abilities, we “computationalize” ourselves as well. In other words, we are not going to “physically” turn into a machine; we can reinterpret our thinking process in terms of computational terminologies. When we are solving a problem, we take discrete steps to formulating a solution. From the former step to the current step, we map out a “correct” path, excluding all the “wrong” path. In essential, excluding uncertainties to find out the certain path is an information process. The only difference between how human process information and how computers process information is that we are using different meaning systems. Nevertheless, we can still term the process of finding the solution to the problem as “computation”; we can still term our way of working out the problem as “computational thinking”.

In general, it is a “conceptualized” post that allows me to connect the key concepts to explain what happened between different sets of symbols. The basic argument is that “humanization” and “computationalization” can happen simultaneously, and these phenomena represent the computation process, the transformation between one set of symbols to another set of symbols.


  1. Irvine, Martin. “Introduction: Toward a Synthesis of Our Studies on Semiotics, Artefacts, and Computing.
  2. Denning, Peter. “What Is Computation?” Originally published in Ubiquity (ACM), August 26, 2010, and republished as “Opening Statement: What Is Computation?” The Computer Journal 55, no. 7 (July 1, 2012): 805-10.

Everything starts with Smalltalk – Yasheng She / Ruizhong Li

TEAM: BEAUTY♡ PRETTY☆ SOCIETY♀ (Ruizhong & Yasheng)

I (Yasheng) just started learning how to do basic coding on Arduino and, to my surprise, the learning process was actually painless.

Here is an example on why learning to code is easy:


Arduino’s software comes with a lot of examples and you can just open them up to practice, and when you feel like you are confident in what you are learning, you can just modify the codes to make a new project. The information in the red block in the right of the graphic shows that the programmers of Arduino explained what how to utilize different classes and their instances. It feels like learning a new language in just minutes and I did not even have to take exams. My experience is consistent with the essence of Kay’s vision: “to provide users with a programming environment, examples of programs, and already written general tools so the users will be able to make their own creative tools. (32)” The reason why I feel the basics of coding is easy to acquire is that coding is not a brand new language, rather it uses “already existing representational formats as their building blocks, while adding many new previously nonexistent properties. (23)” This makes us think, why didn’t we learning computing this way from the beginning? Instead of treating it like a blackbox, we should have studied how programing works from a fundamental level. The interface of Arduino helps me to interact with the hardware directly, and we can see evidence of such system in Kay, Nelson, and others’ theoretical framework. We take advantage of Kay, Nelson, and others’ system of interaction – meta-systems, to “support the processes of thinking, discovery, decision making, and creative expression. (53)”

I (Ruizhong) have been learning Javascript (p5js) for about one year. What impressed me is the “all-in-one” style of this programming language website. The programming environment is easy to manipulate. I started to learn the syntax of the language by playing with it. With bunch of examples, I could adjust some parameters in the code and simultaneously see what changes happened in the output. It is not necessary for me to start from drawing basic shapes to complicated-structured image. I could start from a macro-view of how all these codes function, and acquire the knowledge of syntax through practice. With the libraries imported into the code, I could also make use of to the “mature” “building blocks” which function as a whole with only a few lines of code to create my personalized meaning tool.

The experience of learning p5js contrasts with my learning experience with Python. With a tutor teaching us in a traditional way, I feel it is a little bit “overcautious” while we moving forward to the next stage. With a problem proposed at the beginning, we practice the routine of solving a problem using Python code from drawing flowchart to writing code. We cannot directly see the relation between the code and output. I have no idea of how the code should look like. It seems to me that the codes stay as a conceptual level, and can never be brought into reality. Therefore, for me advantage of GUI and software is enhanced with this contrast in my learning experiences.

It is not surprising to us anymore that how much we can do using computers yet it is surprising to find out how the idea we take granted is formulated by people like Alan Kay. We are especially fascinated by Kay’s concept of Smalltalk. Smalltalk, as an early stage of all object-oriented programming software, is remarkable in the sense that it simplifies the thought process into small building blocks and creating meaning using these building blocks feel just like using our brain – only externally. Thanks to technological advancement, we can now do almost everything on a small laptop, from drawing to making music, or in Alan Kay’s term a “personal dynamic media” system (metamedium) that grants us endless possibilities.

No wonder people call Alan Kay a visionary – he was correct in thinking of a computer as new media generation engine during a time when a computer metamedium was only coming into existence.

According to Alan Kay, “symmetric authoring and consuming is quite lacking in today’s computing for general public.” In Kay’s vision, people are allowed to create, manipulate, sequence and share media across the world with the software. To realize his vision, modern technologies like notebook, tablet, and smartphone have fulfilled his expectation in terms of physical forms. However, the problem lays within the lack of open source. Open source is not a new concept, yet it still receives little attention comparing to massive success of Smartphone, PCs, and other products. Kay’s vision is closer to the Linux system, manifested as Raspberry Pie, Arduino, and other small appliances. These small appliances all have an open source library and easy to understand interface, yet they require a certain level of literacy for users to fully take advantage of it usability. Smartphones and tablets, on the other hand, are made into more and more like black boxes, allowing people to communicate and to create only at a surface level. If there is a way to combine the flexibility of Raspberry Pie with the affordance of smartphone, a Kay-styled Dynabook can be possible.

Speaking of flexibility, Kay’s vison is still not made possible. We still have the overarching notion that technology evolves does not evolve with us, but it continues follows a standard practice. Kay maintains that, “There is also the QWERTY phenomenon, where a good or bad idea becomes really bad and sticks because it is ingrained in usage.” So in terms of usability design, a truly “dynamic” interface should grant users freedom to truly personalize their experience with the machines. Furthermore, literacy of technology should be reduced to teaching people “building blocks” instead of teaching them to “Press to unlock” (formally known as “swipe to unlock”).

  • Lev Manovich, Software Takes Command, pp. 55-239; and Conclusion.Follow Manovich's central arguments about "metamedium", "hybrid media", and "interfaces" and the importance of Allan Kay's "Dynabook" Metamedium concept.
  • Jay David Bolter and Richard Grusin, Remediation: Understanding New Media. Cambridge, MA: The MIT Press, 2000Martin Campbell-Kelly and William Aspray. Computer: A History Of The Information Machine. 3rd ed. Boulder, CO: Westview Press, 2014.
  • Kay, Alan C. “Microelectronics and the Personal Computer.” Scientific American 237, no. 3 (September 1977): 230-44.
  • Alan Kay and Adele Goldberg, “Personal Dynamic Media” (1977), excerpt from The New Media Reader, ed. Noah Wardrip-Fruin and Nick Montfort. Originally published in Computer 10(3):31–41, March 1977. (Cambridge, MA: The MIT Press, 2003), 393–404.
  • Kay, Alan C. “Microelectronics and the Personal Computer.” Scientific American 237, no. 3 (September 1977): 230-44.
  • Interview with Kay in Time Magazine (April, 2013). Interesting background on the conceptual history of the GUI, computer interfaces for "interaction," and today's computing devices.
  • Butler Lampson's original 1972 memo on the Xerox Alto computer, the first "personal" computer implementing a GUI Windows and mouse system and networked via Ethernet.

Have a conversation with computer, where are we on the continuum between human and computer? – Ruizhong Li

Interacting with computers, we human are on a continuum between ourselves and computers. According to Licklider’s depiction of “Man-Computer Symbiosis”, in order to enable the effective man-computer interaction, human and computers should work on the same display surface. In that case, before we get to that point when human and computers literally can work collaboratively, where is human’s position on the contimuum in this process?

It is interesting to know that Licklider’s effort to clarify “man-computer symbiosis” is considered as a way to humanize computing. We are getting computer adapted to human thinking. However, in the process of getting computer stayed tune with human, human is adapted to computer in this process as well. Extrapolating from it, it is true that human function as an agent in the development of technology, but technologies are self-evolved according to their innate mechanism. The role human play in the process is to take advantage of the technology and, according to human’s need, to decide what’s the main technology human would probably make heavy use of in the next decade. In this process, human can benefit from what affordances provided by the technologies, but also cannot get rid of the constraints of the technology.

As we look back into the history, it’s not hard to find that many of the computing technologies were developed during the post-War period. We cannot neglect the social context when we recount the history of computing. In the post-War period, there were growing demands of accuracy and efficiency in computing. It seems that the atomic blast that ended the World War II have an unforgettable impact on post-War scientific research. The eager for nuclear technology created the need for developing the basic computing technology. The most important feature of Sketchpad when it was invented in 1960s, was to perform rapid and accurate calculation, and model the architecture design: both of the two features are related to the preparation of arm race.

But look at the Sketchpad technology today, it is widely employed in our daily life, like handwriting system in our mobile phone, and children’s eco-friendly sketchpad.



It seems that using Sketchpad technology to do these daily things are like employing a steam engine to crack a nut. Is it? I think it is the necessary “backwards” of the technology. They are sharing the similar mechanisms, but they are differentiated by their usage. Using a mature technology to find radical new ways of using such technology. It reminds me of Gunpei Yokoi’s philosophy: lateral thinking with withered technology.

“Withered technology” in this context refers to a mature technology which is cheap and well understood. “Lateral thinking” refers to finding radical new ways of using such technology. Yokoi held that toys and games do not necessarily require cutting edge technology; sometimes, expensive cutting edge technology can get in the way of developing a new product.

It is exactly what is going on between human and technology. If we could slow down and have a look at the way we were coming through, human are driven by the social needs all the time. That’s why we are in a passive position in the development of technology. We do not have enough time to understand the computers, and we lock the secret of how computer working in a little box. We are going too fast to understand what’s going on in the world. Using Sketchpad for daily use is a sign that we could change our way of thinking and be ready to embrace another breakthrough.


Martin Irvine, “Introduction to Affordances and Interfaces: Semiotic Foundations

Mahoney, Michael S. “The Histories of Computing(s).” Interdisciplinary Science Reviews 30, no. 2 (June 2005): 119–35.

Bush, Vannevar. 1945. “As We May Think.” The Atlantic.

Conery, John. 2010.  “Computation Is Symbol Manipulation.” The Computer Journal, 55, no. 7.

Licklider, J.C.R. 1960. “Man-Computer Symbiosis”. New Media Reader. Wardrip-Fruin, Noah, Nick Montfort, ed.. 74–82. Cambridge, MA: The MIT Press, 2003.

Computational thinking is all about human thinking – Ruizhong Li

Among the computing principles, the COMPUTATION category is the most impressive one. The computation category addresses the problem of converting complex problems into a number of computational steps to achieve a solution.

When I was learning Python on campus, in order to solve a problem, the tutor always started with a flowchart. For example, if she wanted to print out “Hello World” for 5 times, here is her workflow:

First, she would draw a flowchart:


Computational thinking is a way humans solve problems. In the process of drawing the flowchart, we are thinking in human’s way. Print out “Hello World” for 5 times, intuitively we have to define how many times and what to print, and then print it out. These steps in our mind are exactly depicted by the flowchart, as well as the code.

| while loop |

i = 0

nbrTimesToPrint = 5

while (i < nbrTimesToPrint) :

print(‘Hello World’)

i = i+1

In the code, we use the variable i as a counter to count from 0 to 4, which is 5 times. Using the while loop, the little program repeats the “print” action for 5 times in only 5 lines. The number of the lines will be the same even if we want to print it out for hundreds of times, thousands of times.

Therefore, the advantage of the abstraction is manifest. If you cannot abstract “printing ‘Hello World’ for 5 times” as a repeating process, it is impossible for you to think about having a loop in your code. The flowchart will be like this:


and the code will be:

print(‘Hello World’)

print(‘Hello World’)

print(‘Hello World’)

print(‘Hello World’)

print(‘Hello World’)

The number of lines is the same as the loop one; this version of code is even more explicit. It is coder’s call. For now, the requirements are easy to meet with: even with the second one without loop, the problem can still be solved in a few lines, but for further consideration, the loop one is more advantageous for sure.

It should be noted that human thinking come first and then computational thinking. The decision of whether to loop or directly to command for 5 times, has been made in human’s mind. The idea of loop is not taught by the computation or software; it is our developed logical thinking that enables us to simplify the steps by making use of loop.

We can observe the existence of RECURSION from this tiny program as well. Making use of the value of the counter variable i, with the comparing between the counter and the variable nbrTimesToPrint, the program knows when to stop. As we can see, the counter i needs an initial value i=0, and then with i = i+1 at the end of the program, the counter adds 1 when the program run through the whole process for one time. The counter is the descriptive feature of the program accounting for how many times the program has run, and it is this value that determines the end of the program.

Computing is the automation of abstractions. Computational thinking focuses on the process of abstraction. The computation and software rely on human minds, and by writing out the codes, human have a chance to recount what’s going on in their minds in an observable way.


“Key Concepts in Technology: Week 7: Computational Thinking & Software – YouTube.” Accessed October 27, 2016.
“Jeannette M. Wing – Computational Thinking and Thinking About Computing – YouTube.” Accessed October 27, 2016.
“Denning-Great-Principles-of-Computing.pdf.” Google Docs. Accessed October 27, 2016.
“Evans-Introduction_to_Computing-2013.pdf.” Google Docs. Accessed October 27, 2016.

From Bits to Meaning – Ruizhong Li

Information Is a Physical Order

This week’s readings about information theory remind me of the book I read last semester: Why Information Grows: The Evolution of Order, from Atoms to Economies. The author, Cesar Hidalgo, maintains that information is not a thing, rather, it is the arrangement of physical things. It is a physical order. The contrary side to the order is randomness, therefore, the information grows by overcoming the randomness. This point of view aligns with Claude Shannon’s opinion. Borrowing the idea from thermodynamics, Shannon calls the random, uncertainty principle “entropy”. What is the most important, Shannon proves that the entropy can be measured and controlled, which provides possibility to reduce the uncertainty to generate and communicate information. Since entropy can be measured, the same logic, the information can be measured as well. The basic measurement is used to quantify how much information will potentially be used to encode, transmit, and decode electronic signals. The measurement introduces the concept, bit (binary digit), which comes from Boolean logic.

Why can’t we extrapolate from the “information theory” model to explain transmission of meanings?

Because the information theory model excludes the meaning.

The signal transmission model is based on binary math, combined with the logic of probability theory. It is the method for signal transmission, not meaning transmission. People always blur the boundary between signal transmission and meaning transmission because we have the ability to interpret messages and infuse them with meaning. But what is going through the physical wires is not the meaning, but the signal. The information itself doesn’t contain any meaning.

Where are the meanings in our understanding of messages, media, and artefacts?

Meaning is created during semiosis.

Meaning exists nowhere but the process when we perceive the signals. As socially symbolic beings always live in technically mediated symbol systems and use information to exchange meanings. Since human are social and collective animals, the meaning-making happens during a certain community, in which people share a common ground on interpreting specific signals. The way that people interpret signals mean is embedded in the social structure and context. It doesn’t mean that the meaning exists somewhere. The meaning-making is a dynamic process, although to some extent it is subject to the social context.

What is needed to complete the information-communication-meaning model to account for the contexts, uses, and human environments of presupposed meaning not explicitly stated in any specific string of symbols used to represent the “information” of a “transmitted message”? 

Information theory + semiotics = the whole story.

Shannon’s signal transmission model perfectly explain how information encoded and decoded in a physical aspect. What it leaves out is the meaning system. Semiotics bridges this gap between bit and meaning. Semiotics addresses the process from symbols to meanings, which help complete the process from bit, a kind of symbol, to meaning. Information theory only talks about the input and output on the both sides of the black box, but semiotics clarifies what is inside the box. For example, the triadic structure and the parallel architecture help understand the process of interpreting the symbol from object – representamen – interpretant, in phonological, syntactic, and semantic aspect simultaneously.

[1] Irvine, Martin. “Introduction to the Technical Theory of Information.

[2] Cesar, Hidalgo. “Why Information Grows: The Evolution of Order, from Atoms to Economies.

It’s a long way to go from a frog to a hate symbol (at least for me) – Ruizhong Li

‘Pepe the Frog’ meme deemed a hate symbol by Anti-Defamation League

When I saw the title this morning, I am not sure whether I can understand what it means, even though I know every single word in this title. As I scrolled down the screen, a picture showed up:

The Deplorables

OK, well, I have to admit that the only thing I know about the picture is that the guy in the middle is Donald Trump. But, wait … what’s that frog? It looks so familiar to me. Oh! that’s the famous meme! The sad frog! As I looked back to the title, I realized that the frog is not named as “sad frog”; it’s called “Pepe the Frog”. And as I continued to read the news, I even found out that the frog was not designed as “a sad frog” at the very beginning. The design of the picture looks like adapting from the poster of The Expendables – nice mimicking, but, what does “The Deplorables” mean? These people (and the frog) are put into the same frame because they are “The Deplorables”. The literal meaning of “deplorable” is “deserving strong condemnation”, but as I continued the search, it turns out that “the deplorables,” means people who are racist, sexist, homophobic or xenophobic.

… STOP! I am overwhelmed now. The series of self-asking and self-answering process shows that several pairs of triadic (as Peirce would say) relationships are going on here. The frustrated feeling accompanies with my superficial understanding – I only know one vertex of the triangle. For example, if I was overhearing someone’s conversation, and they were talking about “Pepe the Frog”, I would probably ask a question: what is “Pepe the Frog”. And they would show me a picture, and as soon as I saw the picture, I would suddenly realize that, “Oh! It is THIS frog”. I only store the symbol (representamen) of the frog in my mind, and merely with the symbol I cannot communicate the referent (object) with people. It is the object that provide people with an interface to communicate the concept and thought (interpretant). There is no direct relation between the symbol and the referent, I think the relation between the symbol (representamen) and the referent(object) is mediated by the thought (interpertant). Since I have a wrong u nderstanding of the origin of the frog, which leads to misunderstanding the relation between the representamen and the interpretant (a relaxed frog was misinterpreted as a sad frog), let alone the relation between a wrong interpretant and the object. Therefore, establishing “correct” relation between three vertices relies upon individual’s experience. I have never seen the comic, where “Pepe the Frog” made his debut. I didn’t experience the process how “Pepe the Frog” became famous on the Internet starting from At the first time I met “Pepe the Frog”, he was already deemed “sad”. I set up a wrong relationship between the symbol in my mind, and the interpretant, and the object. Fortunately, today’s experience helped me integrate the three vertices.

However, sometimes I get lost in setting up the relationship between the three vertices because of the culture or social context. That’s why I say that it’s a long way to go from a frog to a hate symbol (at least for me). Hardly can I relate this frog meme to “White Supremacy”. As Ogden & Richard suggested, “When we speak, the symbolism we employ is caused partly by the reference we are making and partly by social and psychological factors”. We cannot get rid of the social context when communicating. Those people growing up in similar circumstances share the common ground when they are talking and the common ground has been embedded in the society structure, or institutionalized. The cognition process is intersubjective, and is generated by adding on social factors. In this case, people who grew up in America, and had a strong interest in politics and discrimination issues would easily go through the chain of sign-situation intervening between the act (making a reference) and its referent (the outcome of the act). For me, it’s a long way to go, because I know little about the both ends of the relationship, and limited by my culture circumstances, being indifferent to politics has become a normal thing in my life. That’s the reason why I think it is incredible to make such a reference.


[1] “Clinton Expresses Regret for Saying ‘Half’ of Trump Supporters Are ‘Deplorables’ –” Accessed September 29, 2016.
[2] “‘Pepe the Frog’ Meme Deemed a Hate Symbol by Anti-Defamation League – Washington Times.” Accessed September 29, 2016.

Trying to Answer the Questions… – Ruizhong Li

What is language?

Defined by linguists, language is a cognitive system which is part of every single person’s mental structure. Linguists believe that cognitive capacities are the product of structures in human mind. Given the hypothesis, language research is inseparable from the study of human brain and psychological process. Going beyond the individuals, language is an intersubjective system of meaning making dependent on collective cognition. Language is the primary tool for human to communicate information and ideas, however, why language became the primary cognitive system that is collectively shared by the human species remains mysteries.

What is the human capacity for language in general?

Human capacity for language is termed as language faculty. When we say language distinguishes humans from other species, in fact, it is the language faculty that makes the difference. Language faculty is the native speakers’ competence in that certain language. The competence of knowing a certain language equals having a certain mentally represented grammar. Native English speakers share common characteristics in their mental structure and it is same logic for any other language. People speaking different languages maintain different mental states. Regardless of the difference, language faculty is not specific to any one human language. That is to say, despite each language is governed by its distinctive grammar, all these grammars share principles of Universal Grammar, which go beyond a certain language, and define the features necessary for any language to be a certain language.

What is “a language,” What are the essential features that enables a language to be a language? 

As mentioned, the essential features that enables a language to be a language are termed as Universal Grammar. A language works according a set of rules. The essence of a language is phrase structure rules. It is these rules that allow for unlimited creativity of phrases and sentences. To form the grammar that is universal to any kind of languages, four components are indispensable: lexicon, syntactic component, phonology form component, and logical form component. It is comprehensible including the lexicon and syntactic component as a part of grammar. Lexicon is the brick for building a house. The syntactic component is the knowledge of architecture. Something special about language is that speech is generated with a sequence of words, and the “neighbors” of a word may change its phonetic form. For instance, the pronunciations of “west” in “west side” and “to the west” are definitely different. Another special point of language is that language is a tool for meaning-making, sometimes ambiguity occurs without the logical constraints. Think about a sentence: The boy saw the man with the telescope. We are not sure whether “with the telescope” is a complement for “the boy” or “the man” without the logical inference.

What are the implications of using the features of language as the model for other symbolic systems (visual, audio, and multimedia combinations) and for most forms of communication and media?

Language is one certain kind of symbolic system among numerous symbolic systems on the earth. Among all the symbolic systems, language emerges as the primary cognitive system of human being. The collective cognition is the most distinctive characteristics of language compared to other symbolic systems. Is it reasonable to apply the features of language to other symbolic system? I think it is a tentative process. Derived from the hypothesis in linguistics research, we can also form a more generalized term “symbolic faculty”, which could be applied to some specific field and specific group of people. Going to concert and art exhibition, is another way for people to communicate via visual arts or music. The major difference between these symbolic systems and language is the efficiency of communication: the efficiency would be discounted a lot if the participant in the communication share little common sense about visual or audio representation.

It is still interesting to apply the language features to music genre. We can imagine a baby who was born in a “music family”, in a band. His four older brothers had already played band for years, and the baby was born in a music atmosphere, which seems like a natural setting just like English language. You are allowed to play bass with the experienced music language speakers. Are you going to become a native music language speaker as well? Probably the answer is YES according to Victor Wooten. In his TED speech, he narrated his story of learning English and music at the same time, in the same way. It seems that it is possible to make the analogy between language and music as long as the natural setting of English and music are almost the same. You are not going to learn bass by enrolling into entry level class; you just play with it. You are not going to be taught English; you just say it. The special experience of Victor suggests the possibility of applying the language features to music genre, but also indicates the importance of social collective cognition of one cognitive system.


[1] “Irvine-Linguistics-Key-Concepts.pdf.” Google Docs. Accessed September 22, 2016.
[2] “Radford-Linguistics-Cambridge-Excerpts.pdf.” Google Docs. Accessed September 22, 2016.
[3] “Jackendoff-Foundations-of-Language-Excerpts.pdf.” Google Docs. Accessed September 22, 2016.
[4] Big Think. Steven Pinker: Linguistics as a Window to Understanding the Brain, 2012.
[5] TEDx Talks. Music as a Language: Victor Wooten at TEDxGabriolaIsland, 2013.

On readings: Pictionary


The discussion of meaning-making reminds me of a game called Pictionary. Pictionary is a guessing word game played with teams with players trying to identify specific words from their teammates’ drawings. The definition of Pictionary addresses several key texts discussed in the readings. First, the symbolic cognition process is accomplished or implemented via two different sign-systems: language and image. Second, there are several cognition processes happening simultaneously – expressing, representing, interpreting goes back and forth between “describer” (the drawing player) and “decoder” (the guessing player). To complete the communication circus between teammates, players usually experience the following steps:

  1. Describers get to know the object that they are going to describe, which is represented by written format;
  2. Describers draw pictures to describe or represent the object;
  3. Decoders watch the drawing process;
  4. Decoders tell what they think the object being described is;
  5. Repeat Step2 through Step4, until decoders get the correct answer.

These steps are observable. For describers, the word card that represents the object is an input, and what describers draw is an output. The process converting the written word into drawing can be roughly considered as a symbolic cognition process. For decoders, the drawn image is an input, and the spoken word that decoders use to represent what they think the object is is an output. The process converting the image they perceive into human speech is also considered as symbolic cognition.

However, the “converting” process is still a black box. As indicated in the reading: “how meaning happens – how we make, communicate, or “intend” meaning – when we use symbols, signs, and representations is unobservable.” True, what we can tell is the stage we are at, which is static; but we cannot tell is what’s going on, which is dynamic.

To clarify “what is in the black box”, researchers attempt to “build out models to account for what we can observe”. These models are testable, and open to revision and redefinition. Two models are discussed in the readings.

One model was developed by De Saussure. He pointed out that the meaning relation between signifier and signified is dyadic and static. In Saussure’s opinion, the relation between signified (the concept) and the signifier (sound-image) is arbitrary. Yes, the arbitrary principle indicates that there is no natural connection between signifier and signified. No natural connection indicates nowadays established rules of language is the result of social norm or collective habit. This inference leads to another conclusion that the establishment of the connection is a dynamic process (because the socialization is a dynamic process). The relationship doesn’t “born to be like this”. Saussure didn’t realize the dynamic relation between signifier and signified, therefore failed to explain the institutionalization process within a specific community.

Another model was developed by Peirce, who maintained that the meaning relation must be triadic and dynamic. The primary difference between two models is the newly introduced concept: Interpretant. It is this concept that explains the dynamic meaning relationship between signified and signifier. I am wondering is it possible to present the relationship between these two model as follow:

Signifier = Representatum (?)

Signified = Object + Interpretant (?)

It is Interpretant that enables “unlimited semiosis”. According to Peirce, Representatum is the material-perceptible forms of sign, which “equals” the “signifier” in Saussure’s model; Object is what the signs are “used to be” about, and Interpretant is the instant response to the signs – Object and Interpretant consist of the “Signified”. The behavior of Interpretant assumes the existence of recipient. It is recipient that differentiate the meaning of the object from the object itself. Representatum is only an “imitation” of the object, without the Interpretant, the dynamic process cannot continue. At the same time, the Interpretant can be perceived as a new Representatum, which triggers the infinite generation of Interpretants.

Take Pictionary game as example. There is a word “chocolate chip cookie” on the word card. The written word is a perceptible form of the concept “chocolate chip cookie”. For describers, they are the recipient and they are going to interpret the concept – at this moment, what in their mind is a Interpretant, something that is different from but infinitely close to the original concept (which is represented by the written word). Now, the first round of sign process ends up. The second round begins when describers start drawing. They are not recipient but “representer”. At the same time, the Interpretant in their mind becomes an original concept now, and the images drawn by the describers is the Representamen in the second round. The recipient of the Representamen is the decoder, and decoder is responsible for “interpreting” – again what decoders interpret is different from what they originally perceived. When decoders start to guess what the images represent for, the third round starts. It is the first effort of decoders to establish direct connection between the concept indicated by the word card and what they interpret. In the third round, what decoders inference from the image becomes the Representatum of the most original concept. It is a reconfirmation between the meaning and the concept. If the connection fails (in this context), then describers and decoders will repeat the second round and third round to “correct” their perception of the Representatum for them respectively. The process will not stop until the success of “confirmation”.

The Pictionary game is an “embodiment” of what we are experiencing in our daily interaction and communication. It enhance the feeling of that “We Are Symbolic Species”. In the game, we are seeking for the match between symbols and meanings, and we are continuously question the statement that “Are we on the same page?” The only difference between the game and real life is that we are not going to spare much time confirming whether we are on the same page. This feeling are attenuated, but it doesn’t indicate the absence of the sign process and the trouble brought by the sign process. We are fortunate to cultivate something called “Culture”. Culture has enabled us to share common ground to some extent, but it is culture, again, that creates gap between people. You never know the Representatum (a circle with triangle shapes on it) of the phrase “chocolate chip cookie” can be interpreted as something about quantum.


  2. Pictionary.” Wikipedia, the Free Encyclopedia, September 3, 2016.
  3. “Semiotics-Cognition-Technology-Reader.pdf.” Google Docs. Accessed September 13, 2016.
  4. “Signs, Symbolic Cognition, and Semiosis: Intro.” Google Docs. Accessed September 13, 2016.