In our contemporary age where digital remediation of sign systems is common, paper as perceptible material substrate for static, two-dimensional visual sign systems is still being widely used. This is because paper has affordances that new technologies do not have and new technologies also bring about new constraints. In the future, paper will probably be totally replaced because of the further advance of technologies and human beings’ shifted expectations.
Since paper is invented in China in 2nd century, it has been one of the major media of human being’s communication and information storage. In the contemporary digital age, paper is still being used widely and intensively. In 2014, the U.S. consumed 73,093 tons of paper product and China consumed 108,750 tons (Green America, 2017). U.S. uses about 68 million trees to produce paper and paper products every year (The Paperless Project, 2014).
Paper is quite a versatile kind of material which is used in many aspect of human beings’ life. For example, paper can serve as mere wraps or containers that protect or preserve other substances. Or paper itself can form human symbolic artifacts without the participation of other sign systems. The art of paper folding and the Chinese art of paper cutting are good examples. These paper-related arts are themselves symbolic and iconic sign systems that people use to convey many layers of meanings–Chinese paper cutting, for example, does not only represent the image it depicts, but it also has the meaning of good wishes, which is a layer of meaning given by conventions or cultural encyclopedia.
To be clear, what I want to mainly talk about in this article is paper as a media that serve as perceptible material substrate for tokens of static, two-dimensional visual sign systems like written languages and images.
A little history
Signs systems like written languages and paintings have been invented employed by human beings as media of communication since the dawn of civilizations, but paper was not a universally used media until it was invented in 105 AD by Chinese people (Stavrianos, 1998). Before settling down on paper, those two-dimensional sign systems wander through different other material substrates. About 3000 BC, people in the river plain of Mesopotamia inscribed written language in clay, and Egyptians discovered papyrus as a portable but unstable writing surface. Chinese people used bamboo strips linked with thread to create bulky scrolls that store information in an ancient version of Chinese characters around 1500 BC.
People around the world were adept at making use of local materials. Wax, leaves and wood had all been once considered handy materials to store written information from 5th century BC. People started to write and draw on parchment after it is invented in the region of Mediterranean during 2nd century BC.
And then paper stepped onto the stage of history. It made its way from Asia to Arab, to Muslim world and eventually to Europe. By 15th century, paper had become common in Europe. In 19th century, paper went through a major revolution with its main ingredient switched from rags to wood pulp–to cope with the greater demand brought by economic prosperity (worldhistory.net, 2017).
Since then, paper has been used by people all around the world from different cultures that employ various sign systems–different languages, different vocations, different walks of life, all these mean different sign systems, but those static, two-dimensional visual sign systems all conform to paper, their common perceptible material substrate–until its dominance is shaken by new technologies.
Affordances of paper
Affordance is a concept adopted from cognitive psychology that means the action and interpretation something can afford (Norman, 1999). Compared with previous materials that bore the same types of signs systems, paper has many advantages, thus it bears more affordances.
Firstly, paper is light, compared to clay and bamboo strips. There is a Chinese idiom, “学富五车” that can be directly translated into “One’s mind bear more knowledge than could have been contained in five cartloads of books”. It is an expression that can be dated back to the time when books in the form of bamboo strips is so heavy that can only be carried around with carts dragged by horses. Secondly, paper is thinner and more flexible than papyrus and parchment. Also, paper is far less expensive that parchment made from animal skin, so it can be massively produced and became available for people throughout societies.
These advantages in properties give paper more affordances as a communication medium that is suppose to spread sign systems through time and space than previous materials. Its lightness and thinness makes it easy to be carried around. For example, mailmen send one’s mail to where it is intended, which is a practice still used now though in more efficient ways than ancient times. Paper’s flexibility reduces the risk that structures of sign systems reside on it is to be compromised.
Coming down to practical functions, Paper serve mainly as a display interface that connects different systems, which in most cases are two dimensional sign systems and human being’s core operation system to recognize signs. Books are good example for this case. Along the path of technological development, sign systems in the book was firstly handwritten (or drawn), and then printed, but the progress of technological advance has not changed the properties of the interfaces.
Paper also serve as a handy tool for human beings to offload some of their cognitive processes (Dror & Harnad, 2008). A journalist would take notes during interviews on his notepad so that he can keep a record of important facts and quotes. Someone who is heading to supermarket would figure out a shopping list on a piece of used paper before he sets out so that he does not need to murmur all the things he wants to buy to himself all the time. A student who is trying to learn mathematics would be very used to listing some equations or drawing some diagrams on scratch paper. A piece of delicate painting always starts with a rough sketch. As a matter of fact, the concept of this kind of quick cognitive offloading is well adopted by new technologies. For example, in Processing, a java-based programming language, programs are referred to as sketches, in the spirit of quick graphics prototyping (Shiffman, 2015).
Moreover, due to paper’s physical properties, like lightness and flexibility, it can easily be attached to other objects, so that it is frequently used to label things. Tags attached to clothes on a shopping mall’s racks indicate their prices and other things buyers need to know. Paper boxes are used not only for containing objects, but also are often used for giving an illustration of their properties. The concept of labeling has also been adopted by contemporary technologies, e.g. hashtag function in social media and different other forms of metadata.
Nowadays, paper’s position as a major kind of media for human beings’ communication has been challenged by devices that can digitally remediate the types of sign systems afforded by paper.
Computer is the most important example. Computer was designed under the concept of metamedium, medium for other medium (Manovich, 2013). Computer can afford not only the static, two dimensional sigh systems that can be borne by paper, but also more complex time-based media like audio and video. Digital remediation of tokens from existing sign systems also allows users to search and compare among different patterns more easily, which is certainly one of the constraints of paper. More importantly, computer allows users to manipulate the sign systems like editing and re-arranging, instead of mere composing and receiving, by enabling symbols for action, which means symbols for controlling other symbols (Irvine, 2014). Users are also free to offload some of their cognitive processes by employing digital applications like calculator and notepad. The automation of some of the repetitive or standardized processes makes things a lot easier. Also, data can be attached to other data like tags to clothes for the sake of explanation or description. Links can be made to bring groups of data from different locations together, as Vannevar Bush envisioned for Memex, a imaginary prototype for modern computer (Bush, 1945).
E-book like Amazon Kindle is another significant example, for they are designed to imitate the look and feel of physical books, magazines and newspapers. The electronic ink and the designation for users to hold it with one hand are employed by Kindle to conform with human being’s form factor, which means the product tries hard to fulfill human beings’ habituated expectations–in this case, reading printed books. But it can also go beyond printed books. For example, in the process of reading a e-book on Kindle, users can create and share notes, look up words in dictionaries, adjust the font and size of text and search for certain content (Amazon, 2017). In addition to mere interpretation of given signs, e-book allow users to do some manipulation on their own. To sum up, users can quickly get familiar with the product using the obvious inference from the detectable features given by their cultural encyclopedia, and also make use of other desirable affordances to make their using experience more enjoyable or efficient.
Why haven’t they totally replaced paper yet?
New technologies like computer and e-book have introduced many new affordances to the perceptible material substrate that display two-dimensional sign systems, but the reason why paper is still widely used is because new technologies have also brought about new constraints.
One of the most significant is that electric power is indispensible for the digital re-mediation of the sign systems that appear on paper. If users want to access data that are stored online, Internet is also needed. Robinson on an uninhabited island would not choose a computer over a book though potentially he can access all the existing knowledge of human beings through it. This is not the case for paper. After a book is produced, it can work for a fairly long time in terms of displaying sign systems before its material fails without additional power input, though the content and form of sign systems are fixed. Constraints of new technologies indicate that paper as an interface for displaying sign systems cannot be replaced under certain circumstances, such as limited power input and no access to Internet. We are used to carrying a book with us when we take metros or a long flight.
Besides the difference brought by the constant need of power for digital remediation, paper as a tool for offloading cognitive processes have many other advantages over computer, smartphone or tablet. When we want to take some quick notes, we can just grab a piece of paper and a pen, and start writing. We don’t have to pull out a device, turn it on, intiate a certain software and start writing. When writing is done, we don’t have to go out of our way to do the process of saving. (Everyone using electronic devices for sign system manipulation must have gone through the pain of losing hours’ hard work just because they forget to save it.) If we want to take what we have taken down with us, we just tear the piece of paper from the notepad, fold it and put it in our pocket. In the case of electronic devices, however, we have to take the device that is certainly bulkier than a piece of paper with us because the display and interpretation of the digitally remediated tokens requires specific hardware and software. We certainly can’t fold the devices to fit them into certain amount of space because of the current limitation of hardware.
When we are scrawling on a paper, we can write or draw any diagrams, shapes, or charts as we wish with little restrictions. We can even use shorthand that is only recognizable for ourselves, as long as it is interpretable for future reference. But we can’t do this with computers or other current electronic devices. For example, in Microsoft Word, we are enabled to insert in our article different kinds of charts to illustrate our points, but they are all pre-defined and fixed by the software, so that while using this function, we can just follow the existing patterns, but not handily create our own pattern. Along the way of development of computer, scholars have brought up the idea that computer serve as a platform for users to develop their own tools for manipulating sign systems (Kay & Goldberg, 1977), but this vision is yet to be accomplished.
Different perceptible material substrates for the same type of sign systems also have unintended consequences. Firstly, medium is the message, meaning that “message” of any medium or technology is the change of scale or pace or pattern that it introduces into human affairs (McLuhan & Lapham, 1994), thus the media where a type of sign systems is remediated is not indifferent, e.g. digital media brought about the information overload and fragmented reading pattern (Liu, 2005). Secondly, sign remediated by pen on a piece of paper is attach with users’ idiosyncrasy that is recognizable but does not compromise the original communication intention. For example, handwriting is one’s own way of re-tokenize the type of signs from written language. Thus, it always enable one more layer of interpretation than the same type of signs that is digitally remediated.
Amazon Kindle and other e-book have done a good job imitating printed books, but not enough–there are still aspects that do not conform with the human beings’ form factors. For example, its refresh rate–the time that devices take to turn to the next page–prevent users from flipping through the pages, which users are inclined to do while reading a printed book.
Contemporary technologies still have a long way to go from totally replacing paper, but I do think they stand a fair chance, for some of the technical problems are on their way of being tackled down. Further development of solar energy technologies might make electronic devices as self-sustainable as paper. Foldable electronic screens are being developed (Gibbs, 2017). With other constraints being solved, I can’t see a reason why future computers would not cover all the affordances of paper.
But the more important question lies in the human beings’ habituated expectation of paper as a perceptible material substrate for static, two-dimensional sign systems. While people who are born in digital age are gradually replacing people who immigrate to it, human beings will become more and more used to the digital remediation of the familiar types of sign systems that bear more affordances, just as people shift from previous writing materials to paper.
Green America. (n.d.). Retrieved December 15, 2017, from https://www.greenamerica.org/
Facts About Paper: The Impact of Consumption – The Paperless Project – Join the grassroots movement. (n.d.). Retrieved December 15, 2017, from http://www.thepaperlessproject.com/facts-about-paper-the-impact-of-consumption/
Stavrianos, L. (1998). A Global History: From Prehistory to the 21st Century (7 edition). Upper Saddle River, NJ: Pearson.
HISTORY OF WRITING MATERIALS. (n.d.). Retrieved December 14, 2017, from http://www.historyworld.net/wrldhis/PlainTextHistories.asp?historyid=aa92
Norman, D. A. (1999). Affordance, conventions, and design. interactions, 6(3), 38-43.
Affordance-Interface-Semiotic-Intro.pdf. (n.d.). Retrieved October 30, 2017, from https://drive.google.com/file/d/0Bz_pbxFcpfxRLWlnWkRoVGpZY2s/view?usp=sharing&usp=embed_facebook
Dror, I., & Harnad, S. (2008). Offloading cognition onto cognitive technology. John Benjamins Publishing.
Shiffman, D. (2015). Learning Processing, Second Edition: A Beginner’s Guide to Programming Images, Animation, and Interaction (2 edition). Amsterdam: Morgan Kaufmann.
Manovich, L. (2013). Software Takes Command (INT edition). New York ; London: Bloomsbury Academic.
What is the most useful conception for the affordances of strong artificial intelligence or artificial general intelligence, (henceforth, AGI) as it is realized, to draw practical assumptions about the intellectual futures of humanity? Contrary to dangerously simplistic popular notions, the semiotic method tells us that human-like AGI will never be able to internalize meaning the same way humans do, and as a result of which, will not have the capacity to exhibit enough independent agency that warrants a “moral status” or “personhood”. This essay attempts to synthesize research from the fields of philosophy of AI, cognitive science, philosophy of mind and semiotics to address the consequences of automating the full range of symbolic-cognitive capabilities of the human species.
Key concepts: symbolic system; philosophy of mind; cognitive science; philosophy of AI; semiotics; intelligence.
Contemporary AI philosophy and popular science have expelled deterministic predictions about the nature of computing technological development, such as the “intelligence explosion”, “singularity” and “superintelligence”. This speculative narrative assumes that once machines reach human-level intelligence (as a consequence of human scientific development), they will acquire the ability to “self-actualize”, and therefore express its own desires, needs and intentions. My goal in this paper is to challenge this notion and prove that such radical foresight may be fatally mistaken along with why such a hypothetical event is not plausible or even remotely useful. The motivation driving AGI research is not so much so that we want to progenate a new species to provide company to humanity or even replace humanity, but to automate complex human problem-solving in a human-machine symbiotic fashion. However, while this field of study remains largely conjectural, active research has been being carried out since interest in AI sprung up in the 1950s. Whether the AGI technology is realized or not, we are presented with profound scientific questions of the characteristic uniqueness of the human mind and its future.
My inquiry is rooted in a curiosity of AGI “machines” solely based on its imminent ability to communicate in natural language (since it is commonly agreed to be an important criterion for human-level intelligence). I suspect applying the symbolic-cognitive hypothesis reveals important consequences for such a long-term goal (to mechanize general intelligence) in AI research. What motivates my inquiry is an intellectual dispute with the impression that human-level artefacts (artificial minds) will soon emerge self-conscious and develop a “mind of its own”. My contention is that AGI technology, from a phenomenological standpoint, will primarily be a tool, yet a powerful one to augment human cognitive agency that may perhaps exponentially accelerate human intellectual progress. For the sake of argument, I use the term AGI as how John Searle defines it: “a physical symbol-system capable of having a mind and mental states”.
The computer is and always has been an interface to the symbolic world represented inside (which in itself is an artefact of human symbolic-cognition). Therefore an artificial intelligence machine will primarily remain a human-created symbol system. While there are compelling views for the implausibility of manufacturing intelligence based on the simulation argument, that is, just because a computer simulates human thought, does not mean it is the real thing, we are missing the whole point here in plain hindsight. What will be the utility functions of a machine that simulates human thought be? AGI will, therefore, be an extension of the “continuum” of the human symbolic cognition. “Artefactual Intelligence, not Artificial Intelligence.” — we will not be able to replicate human intelligence but achieve just enough to enhance human cognitive capacities.
The consequences of studying the philosophy of AGI are that it re-informs the way we think about the nature of such a speculative technology that purports to simulate all human intelligent behaviour. The symbolic-cognition hypothesis shines light on and urges us to take utmost advantage of the fact that artificial Intelligence is an exponentiation of the kind of cognitive offloading humans have been doing for centuries (50, 000 years or so). Computation, as an artefact of human symbolic-cognition itself, is an automation of semiosis. As such, my inquiry does not particularly bother with the technological feasibility of such a machine but rather aims to study the implications of one by means of insight borrowed from the fields of Piercian semiotics, cognitive science and philosophy of mind.
“What deserves emphasis is not these mundane observations themselves, but their powerful consequences.” ~ John Haugeland
1. Current AGI Research
Computing technological progress has not been deterministic in the past and prompts no reason to assume that it will for AGI. It may be disastrous to conclude precisely the nature of AGI designs we will interact with in the future. However, based on what AI researchers are actively working on, we can perhaps map out a way for what is to come for the sake of philosophical examination.
“the scientific goal of mechanizing human-level intelligence implies at least the potential of complete automation of economically important jobs (even if for one reason or another the potential is never activated)”. ~ Nils Nilsson
While narrow AI research is engaged in multiple different routes or “narratives”, the prevalent goal of all AGI or “strong AI” research seems to be similar. The computing technologies we have had so far have been implemented, explicitly or implicitly, with a goal to “augment human intelligence”. AGI research aims to expand this cognitive need: “to automate rational thinking to make the best decisions with limited computational resources.” (Ben Goertzel)
2. General Intelligence
AGI research is inherently an interdisciplinary field converging insight into the same problem of defining general intelligence from computer science, cognitive science, neuroscience, psychology and linguistics. The standard usage of the term “intelligence” among modern human communities seems to ascribe to an ability to solve problems. This is important because the only way a social community can unanimously accept that a machine is, at any point, exhibits human-like intelligent behaviour, albeit externally, is if everyone can agree that it actually is. Minsky offers a flexible definition:
“Our minds contain processes that enable us to solve problems we consider difficult.“Intelligence” is our name for whichever of those processes we don’t yet understand.”
However, it is perhaps even more critical to outline what AGI researchers are defining the same term in the context of scientific development. Lo and behold, the field is ripe with a variety of growing definitions. For instance, the physical-symbol system hypothesis claims that a symbol system, with an ability to store and manipulate symbols, is necessary for intelligence. Another common view is that the ability to process and communicate in natural language is a testament to intelligent behaviour. Levesque brings up a functional approach, essentially saying that we ought to shift the discussion from defining what counts as intelligence to what counts as “intelligent behaviour = agent making intelligence choices” for a more productive understanding.Lesquevez also draws a crucial distinction in human intelligence- “mechanical” and “ingenuity”, which I think is worth noting. Mechanical is what symbolic-cognitive artefactual activities like algebra demands from us, the strict procedurality of following rules and syntax, with the latter requiring a meaningful absorption.
A quick retrospective survey of the history of AGI ventures and designs unveils that much of AGI research has been directed at natural language processing. This makes sense because to mechanize human-level intelligence, automating the species’ primary meaning-making system sounds like a good start. Most notable successes include ELIZA that was built in the 1960s to model human-like interaction, in 2011, IBM’s Watson beat the best human players at Jeopardy and The Turing test has been commonly accepted as a close approximation of confirming general intelligence of an artefact. Perhaps a more accurate test, such as garnering an insightful response from an intelligent machine when given a piece of artwork, a song or a movie, would prove its understanding of the relational function of the “interpretant”.
“Until the last have no intelligence at all. But that does not yet resolve the more basic paradox of mechanical reason: if the manipulators pay attention to what the symbols mean, then they can’t be entirely mechanical because meanings exert no mechanical forces; but if they ignore the meanings, then the manipulations can’t be instances of reasoning because what’s reasonable depends on what the symbols mean. Even granting that intelligence comes in degrees, meaningfulness still does not; that is, the paradox cannot be escaped by suggesting that tokens become less and less symbolic, stage by stage. Either they’re interpreted, or they aren’t.” (Haugeland)
IBM’s ‘Watson’: What we already have
What we expect of an AGI design
3. Semiotic Theory
The human invention of the stored-program computer escorted a revolutionary distinction in the functions of symbols (data) of ones that can do things and those that mean things. Here, I lay out two important semiotic principles that guide my discussion.
PRINCIPLE 1: The first signs have a material-physical-perceptible form
“So we normally take symbols to be objects in one domain that, by virtue of similarity or convention, stand for something, objects in another domain. The first domain is usually concrete and easily seen or heard, while the second one may be less concrete, less accessible, and sometimes purely abstract.” (Lesquevez)
“A sign is something by which we know something more” (CS Pierce). Here the sign is the perceptible-physical-material substrate and the something more is the abstraction, which is another set of signs and symbols.
PRINCIPLE 2: Dialogism
Hobbes said that human thought is inner discourse which makes sense because according to semiotic theory meaning is dialogic that is activated by symbols and we have already established that all human thinking is in symbols, according to the Pierce view of semiotics. The most pertinent role played in semiosis is the that of the interpretant which conducts the relational or mediating function of producing another set of symbols for the first sign. This is the key activation that anchors an entity’s cognition with shared ideas and the outside world.
4. Symbolic Cognition
As the core human operating system which remains the source of computation and all other artefactual cognitive agencies, this concept hypothesizes that cognitive faculties such as reasoning and decision-making are a result of “automatic” symbol manipulation. This unique human symbolic capacity to make meaning, however, depends on shared symbols within a community, rather than occurring within an individual’s mind. Human symbolic-cognition engages with symbols for two purposes: for communication (to send information) and problem-solving (computation). However, in the realm of cognitive technologies, both activities are interdependent. Information processing is required for communication across time and distance, and communication (control of information flow) is required for pulling the right data during a computation. Over an evolutionary period of time, humans have become smarter because they have been able to make higher levels of abstraction based off of the stuff the previous generation left behind in extended memories, in a continuum of symbolic-cognitive advancements (the cumulative or ratchet effect).
“Symbolic cognition enables us to “go beyond the information given” (Bruner, Peirce) in any concrete instance of an expression or representation. We go beyond physical perceptions to activate meanings, ideas, higher levels of abstraction, values, and responses that are not physical properties of symbolic forms but mental and physical associations that we learn how to correlate with them — and “output” our meanings in further symbols, actions, or emotions.” (Martin Irvine)
The crux of my argument is that it is highly unlikely that we can build artificial “minds” with a capacity to assimilate into its nature, the meaning of signs and symbols that it is able to manipulate like humans do because it takes a community with shared concepts and ideas to make an exchange, not solely a single superintelligent entity in isolation. The process of meaning-making, in other words, semiosis, transpires not even in the human mind, but in a community of minds. The AGI technology could mediate, communicate, transmit and store meanings, but it is according to semiotics, impossible for an artefact to realize it itself. With the advancement of NLP, computers could begin to process the vast array of information on the web, but the results will only point out to more “signs and symbols” by means of computational routes.
Perhaps no other evidence underscores my impression better that John Searle’s Chinese Room. This thought experiment involves a room divided by a door, with one filled with native Chinese speakers and another non-Chinese speaker, with no knowledge of the language. However, the non-Chinese speaker is provided with a rule-book for exactly the response to elicit for particular Chinese characters and symbols. As a result, the entities in two rooms are able to carry out a meaningful conversation in Chinese (by means of slipping papers underneath the door separating the rooms), even though the non-Chinese speaker has no sense of the meanings transmitted through the symbols. The thought experiment clearly demonstrates that since the symbols themselves contain no semantic content, the machine computing them will never be able to understand the meaning the symbols in function elicit. However, this marks another fundamental question for AGI: does general intelligence require “meaning-making”, or if “symbols-processing” sufficient? After all, the need to develop AGI is to help us (humans) solve problems not, to put it dramatically, relinquish the responsibility of human civilization to a colony of robots. Information theory was about preserving structures across time and distance. This is the process of meaning-making, aka semiosis, but that does not mean the electricity on its own understands the meaning of the information humans exchange with its help.
“The Turing machine itself has no way of guessing what the symbols are intended to stand for, or indeed if they stand for anything.” (Levesque)
“A sign, or representamen [the physical perceptible component], is something which stands to somebody for something in some respect or capacity. It addresses somebody, that is, creates in the mind of that person an equivalent sign, or perhaps a more developed sign. That sign which it creates I call the interpretant of the first sign.” – cs pierce
The only kind of “interpretation” the system is engaged in is the actions it performs as it is being designated with symbols “that do things”, rather than them “meaning things”. Hence, what humans naturally do with symbols, as a result of the species’ innate semiotic competence is make meaning, but also compute as a secondary function, whereas what computers do with symbols naturally is merely compute.
“Interpretation implies a special form of dependent action: given an expression, the system can perform the indicated process, which is to say, it can evoke and execute its own processes from expressions that designate them.” (Newell and Simon)
CONSEQUENCE 1: Why self-replication is unlikely upon the “intelligence explosion”
The “display” of intelligence does not mean the real thing. Therefore, a machine may be able to accurately manipulate symbols based on other human symbol-systems. However, since meanings are not properties of the signs or symbols themselves, and rather the process of the interpretation of them, rendering the event unobservable, we will never be able to discern the machine as engaging in meaningful communication. Such a machine can only “represent” meaning, not internalize it itself because symbol-cognition is exclusively a human capacity (species-specific). Hence, a machine that is as intelligent as humans will never be able to “express” meaning that comes from its own desires, values or goals. It can represent meaning that is being communicated, transferred and exchanged among humans across space and time.
“What an automatic formal system does, literally, is “take care of the syntax” (i.e., the formal moves).” (Haugeland)
CONSEQUENCE 2: Intersubjectivity
For meaning to be “meaningful”, it requires two or more agents. Therefore, semiotic theory proves that intelligence does not require an understanding of the semantic content, in the way we coin the term intelligence in the technology community today.
“Meanings are public, not private.”
“Entering language — and crossing the symbolic threshold — means entering a world of meanings that are always collective, intersubjective, public, and interpersonally shared and communicable.” ~~ (Martin Irvine)
All this talk about “machine bias” insofar produced by narrow AI systems, would have been predicted if previously theorized from a semiotic perspective. Symbols may or may not mean anything. They only stand for other symbols which stand for something else (surprise, more symbols). Even if a computer can “process” natural language, in actuality, it would be converting the signs in the language such as words, sentences and verbs into numerical representations which lose the same intended meaning of the initial signs.
“Meanings and values learned and discovered in our sign systems are not “located” “in” anyone’s head or anywhere, but are cognitive-social events that we activate and instantiate by using our “cognitive-symbolic resources.” (Martin Irvine)
Imagine machines can finally process natural language as efficiently as humans do. I do think it is worth thinking about what it would look like when two or more such machines will be able to communicate with another. Will it be meaningful? Or would it just be a set of logical responses to a set of questions? According to semiotics, it will never be in a symbolic-cognitive position to say “I understand you”, “You know what I mean?” or “Was that clear” except of course if it was programmed to use those expressions as conversation fillers to display intelligent behaviour, sort of like faking it.
CONSEQUENCE 3: Externalized, Distributed and Collective Cognition
If we ever reach a point of automating the complete linguistic competencies of the human species, AGI will still only be an interface to a more powerful distributed human cognition according to the “delegated agency or cognition” concept first provided by Latour. Consequently, human-level intelligent machines will remain an“external symbol storage”, lending itself to an extension of human symbolic-cognitive capabilities, rather than a replacement.
Cognitive semiotics (embodied cognition) also offers a different view to the mind-body problem- essentially all the physical stuff point to or “stand for” all the mental stuff such as intentions, beliefs, desires, thoughts and values. These sign relations or functions are the fundamental causation of human meaning-making. Hence, our intellectual capacities do not necessarily reside in our minds but exist pervasively across networks of meaning-making in human communities.
Understanding AGI through the semiotic method reveals implications for the philosophy of AI, including the premature drive to anthropomorphize potential human-level intelligent artefacts. This helps us see clearly the progressive nature of cognitive artefacts as products of human symbolic-cognition can not even remotely pose an existential risk, essentially nullifying false speculations fear-mongered by popular scientific experts. What do we get as a consequence of automating human symbolic-cognition?
Much of the dangers supposed of the hypothetical event of the technological singularity emerges from an overestimated extrapolation of current approaches to narrow AI which are fundamentally different from those needed to achieve artificial general intelligence. Therefore we can put on a show to make a machine look like it is truly understanding the world through its interactions, but we don’t have a valid proof test. Since signs are literally anything in the physical universe, automating the “core human operating system” can strengthen communication and computation, perhaps naturally leading to better connectivity and more efficient complex problem-solving.
Okay, sure, once we have formalized the entire symbolic-cognition, machines might be able to form thoughts based on the rules we teach it, that’s how humans “make” meaning after all (rule-governed). However, what purpose will that serve? What will be the utility functions? My aim is not to just pose rhetorical questions, but rather to urge us to think deeply about the effects of developing increasingly symbolic-cognitively powerful computing technologies. More relevant questions would be, what kind of cognitive activities will we most likely off-load? What will that mean for human experience in a much more technologically advanced world?
While we may succeed in automating human-level intelligence (perhaps even human-level linguistic competencies) into a material artefact, semiotics helps us accept that AGI machines will never be as semiotically competent as human beings are and have been since the birth of the Homo sapiens.
Jiajie Zhang and Vimla L. Patel. “Distributed Cognition, Representation, and Affordance.” Pragmatics & Cognition 14, no. 2 (July 2006): 333-341.
Stuart Russel and Peter Norvig, “Artificial Intelligence: A Modern Approach”, 3rd Ed., 2009.
Martin Irvine, “The Grammar of Meaning-Making: Signs, Symbolic Cognition, and Semiotics.”
Martin Irvine, “Introduction to Linguistics and Symbolic Systems: Key Concepts”
Martin Irvine, “Semiotics 1.0: Basic Introduction.”
Martin Irvine, “Introduction to the Theory of Extended Mind and Distributed Cognition”.
Ben Goertzel, “Artificial General Intelligence”, eds. Pennachin, Casio, Springer, 2007.
Nils Nilsson, “Human Level Artificial Intelligence? Be Serious!” AI Magazine, Volume 26, Number 4 (2006).
Newell, Allen and Simon, Herbert, “Computer Science as Empirical Inquiry: Symbols and Search”, Communications of the ACM, No. 3, Volume 19, March 1976.
Allen Newell, “Intellectual issues in the history of AI”, Carnegie-Mellon University, 10 November, 1982.
Marvin Minsky, “Society of Mind”, Simon & Schuster, 1988.
John Haugeland, “Artificial Intelligence, the very Idea”, The MIT Press Cambridge, Massachusetts, 1985.
Ray Kurzweil, “The Singularity is Near: When Humans Transcend Biology”, 2003.
Ray Kurzweil, “How to Make a Mind”, 2013.
Patrick Tucker, “The Singularity and Human Destiny”, The Futurist, 2007.
“I feel drawn to experiment with ways that technology can interact with notions of intimacy, because so much of technology is done in a way that’s very cold and has such an opposite effect.” Jaron Lanier
This paper is a multi-case study. The main argument developed in this paper is that when imply digital interface in the field of outdoor interactive advertisements (shorted as OIA in this paper), the normal principles of interactive principals in new media interface don’t always fit. Being metamedia, those ads use new technologies and strategies to combine pictures, videos, even holograms to create novelty, high-jack attentions and to enhance user experiences. Although being called interactive in its name, most of the OIA are designed with closed systems. They are not necessarily intuitive and transparent as other interactive machines are. They often are designed with a choice of share. They are designed to have narrowed options and using the three elements of advertising to direct the process. They sometimes are designed to offer treats as feedbacks. They follow the rules of semiotic and cultural practices.
This paper simply introduce the basic concept in advertising design and the some technologies used in OIA, then use multiple cases to illustrate the interface design values in the normal interactive machine and in OIA.
OIA have many forms. In this paper I only concentrate on the one kind which contain digital interfaces, and interact with human instead of with environment. Without specific pointing out, the OIA in this paper are references for the ads as above.
In the movie The Minority Report (2002), the OIA have been evolved into personal detailed, holograms and cloud sharing information ad. The billboard can call you by your name at first sight, provide shopping history and recommend new items based on your preferences. These personal ads seem a lot ahead, while Peter Schwartz, chairman of the Global Business Network, thinks differently. “This is not too distant future. This is imminent.”It’s never too bold to imagine the future. I personally agree that the technology and the interaction used in the field of advertising which happened in the movie will soon be realized.
Nowadays with more and more non-traditional forms of outdoor advertisements transformed to new media, the interactivity of ads also catch a lot of attention. Will it work? How will it work? These questions has haunted advertisers’ mind from day one. While the technologies have become more and more mature, a lot of ideas of interactive outdoor advertisements have been realized. Research shows that in the high-involvement conditions, experienced users will possess more positive attitudes during interaction than the inexperienced users. This result suggests that in OIA, although creating a whole novel experience to hijack attention and to promote share is vital, retrospect and remediation is also important. From traditional media to metamedia based on time and space, how to design the interfaces of OIA becomes a valuable question to ask.
3. New Medias Remediated in OIA
The appliance of new media in the field of OIA is a tendency all over the world, yet not the only choice. It should be granted that traditional media without digital applications can also work its magic out perfectly if doing it the right way, as in many successful cases in the market. In this paper, I focus on the kind of OIA which are perceived as a time-based metamedia, containing digital interfaces. Being metamedia, those ads use new technologies to combine pictures, videos, even holograms to create novelty, hijacking attentions and to enhance user experiences. Although quite different from the traditional ones, OIA media remediated the old in many ways and will continue doing so in the future. The technologies and the applications listed below are the realistic foundation of the follow-up interface design.
3.1 Technology support
To realize the even simplest OIA requires some sort of algorithm and furthermore digital interaction design. With new media used in the outdoor advertising, OIA create novel and entertaining interactions thanks to the development of the following technology support. These technologies differentiate the new media from the old static ones, letting the audiences feel new and fresh meanwhile not too far away from the experiences of the traditional media.
3.1.1 Eye Tracking
Eye tracking technology is a methodology that allow the researcher to detect where a user is looking at, and furthermore to depict a path and hot spot map of user’s eye movements. Moreover, we can detect how long the user looks at something. When the concern is the identification of elements in visual scenes like an interactive application, the technology called “point of regard (POR)” (Young and Sheena, 1975) is used to measures the orientation of the eye in space. This technologies are widely used in many fields as research method, including psychology and marketing. Meanwhile it can also be used in OIA. The most common way of using it aims at data gathering. By collecting and analyzing the data of audience attention, the information is fed back to advisors to improve the creation and interactivity of advertising. With the eye tracking technology becoming more and more mature, there’s no doubt that a new situation has been created for the OIA and a more accurate delivery of outdoor ad can might well be realized.
3.1.2 Face and Gender Recognition
Face recognition contain the face detection and expression detection. In OIA, these technologies have high values concerning the interaction they might realized. At least, it can base on the number of face it detected to create some changes, such as the London’s Women’s Aid Poster showed below. With each more face recognized, it will reduce the bruise piece by piece, in doing so call for attention on family violence towards women.
Also, the use of face and expression detection can also realize digital image simulation and therefore satisfy the ego-self-identification need of the users, creating an OIA with high entertaining rate. Moreover, nowadays a camera with software can detect whether the passer-by is a male or female. This gender recognition technology can be used in OIA to adjust its ad on the digital screen accordingly. Similar technologies include the age, race and probably identity recognition. With the technology more and more similar to the one in The Minority Report, surely arise the considerations for privacy protection, which will influence the willingness of interacting in the OIA.
3.1.3 Skeleton Tracing
Like face recognition, skeleton tracing can tell the information about the audience by detecting the 3D images. It can calculate the position and the movement of human bones and therefore simulate the movement on the digital screen.
3.1.4 Near Field Communication and Techs to Link Devices
Near field communication technology is a kind of high frequency radio communication technology used between digital devices to carry out non-contact point-to-point data exchanging. First developed by Sony and Philip, now this technology is more commonly installed in mobile devices like smartphones. Similar technologies like bluetooth are also widely applied in OIA nowadays. Normally, audience use one of the technologies that allow them to contact their smartphones with the OIA, by doing so receive “information”. Like the one below, by putting phone to the cued place, the device will connect and showed a song in the list using the tech of near field communication.
3.2 The application of ad design elements
In the field of advertising, the design is mainly constructed by three basic elements, which are image, text and color composition. Coded in the application, these three elements become core of the visual design, conveying information deeper and further than themselves. In that way can be also perceived as semiotic system in the field of visual design. Different from the traditional ads, OIA use media that can exert these elements in a dynamic way, and meanwhile remediate the print media.
3.2.1 From Static to Dynamic
Transition from the traditional media like magazine, internet and bus stop billboard, OIA use new media that are can apply dynamic footage which not only have widespread commercial value and prospects, but also provide a space for advertisers’ creation. Thanks to the contents of new media, the three elements in OIA possess abundant transformation and composition effect. The speed of changing images and texts, the movement path, the dynamic color can all be used to attract attention (and furthermore to lead their attention to specific cued place).
Like the London Women’s aid ad I mentioned before, that OIA use seemingly static images as part of the ad. However once detect faces in front of it, the image will start changing by slowly reduce the bruise. Another part of the ad is the huge text. It stay static while leaving the model’s face changing more evidently. Meanwhile, it chooses dark and solid color as background, making the only change more prominent. These three elements combined together, making OIA have more novel way to attract attention and to lead attention.
Also, audience can be led to change attention from place, thanks to the dynamic elements. This feature can hardly be realized by traditional media.
3.2.2 From Presenting to Interacting
Traditional outdoor advertisements are used to present, showing changeless information to all the audiences. While OIA use new media to convey different information to different audiences by interacting. The design elements of advertising are changed in speed, scale, position and color according to the external input. In which case, it is far more advanced and novel than the video billboard which present time-based footage.
On the one hand, mostly the dynamic elements are activated by external input, aka human interaction. In the same Women’s Aid ad, only when faces of passerbys are detected, the image on the billboard will start to change. In other word, the remediation from the video billboard set a switch in the new media, which allow the OIA’s design elements to change from the stimulation of the external world. One may call this switch “interaction”. On the other hand, from the interaction, OIA allow the audience to decide the elements composition they were shown. In OIA, audience has the choice not to interact, while in the old media and traditional outdoor ads, the advertising is always ongoing. In other word, the switch is put in audience’s hand, not the advertisers. From the perspective of the audience, the ads are not just some static background. They listen to them and seemingly change “at their will”. This feeling is what traditional media cannot elicit.
3.3 The application of the five senses
Advanced technologies create more method for advisors to achieve creativity, sometimes including the immersed experience of five senses: vision, audition, olfaction, tactile and gustation. Traditional media can also apply five senses in their ads. Vision and audition have been applied since day 1 of advertising. Even olfaction, tactile and gustation can be realized with traditional media, like the one below. PlayStation 2 stick bubble wrapped to the bus board, letting audience burst bubbles while waiting for bus.
New media allow OIA to apply more complex interaction involving the five senses, but the underneath purpose is generally the same: to let the audience not only see and hear the ad, but to touch, smell and taste them like in real times. Different from the traditional media which involve olfaction, tactile and gustation experiences, the new media allow the OIA to looks like personal designed, only activated when the external input is entered. Without external activating, the ad will be anything like the traditional ads. Like the one below, the smell of barbecue will only come out when audience tap their card as cued.
To conclude, the new media of OIA remediate the traditional media in three ways, which are shown above: technologies, design elements and five senses. The specific application will be elaborated in the following chapter.
4. Interface Design in OIA
4.1 Interactive Machine Interface Design
4.1.1 From and Based on Turing Machine
Turing Machine, as the first electronic digital computer used a closed, non-interactive system, can only carry on sophisticated computations. However, once the algorithm started running, it will stop receiving external inputs and go on operating until the computation stops. On the other hand, interactive machines can be engaged by the third parties during the operating. It can learn from the outside world and adapt to experience. The results yielded from interactive machines are depended upon unpredictable external actions, in doing so endow the machine smartness compared to the algorithm in Turing machine. Human-computer system has been recognized as the first truly interactive systems according to Beaudouin-Lafon. The SketchPad designed by Ivan Sutherland use digital pen to input messages and engage with a cathode ray tube (CRT) display. This pioneer technology can still be seen today, like Ipad. Until nowadays, graphical interface design has the core feature of interaction.
Also, from the basis of Turing machine, scholars and scientists propose new ideas on the interactivity that could be applied to machines. Van Leeuwen and Wiedemann offer a new implication of interactive computing. The interactive Turing machine (ITC), as they called it, captured the essence of computing meanwhile evolve interactive systems. This new model of computation is “in theory” more powerful than classical systems. Goldin et al extend the Turing machine system based on dynamic stream semantics, called Persistent Turing Machine (PTM). The PTM will carry on normal computations once they read an input from input tapes and will end once they produce outputs on outputs tapes. The additional worktape is responsible to retain its content from one computation to the next. Goldin’s model of PTM falls in the general class of interactive transition system. However in my point of view, it’s more expressive than interactive. With this sequential feature, PTM can yield results based on computations that remain a process of ongoing.
4.1.2 Principles of Interactive Design
Janet Murray provides perfect principles in the fields of interaction design. In her point of view, as in human computer interaction, human users are unreliable, so the designer should get ahead of the time. In other word, all of the principles in the interactive interfaces design are treating users as toddlers.
Firstly, the interface shall be intuitive. The patterns of operation, the semiotic meanings of the icons and indication signal should be drawing from our conscious expectations about the digital behaviour, experiences and subconsciousness of the world. Janet provides an example of trashcan icon. In the interface if a user see a icon similar to the looking of a trashcan in real life, it’s logical and certain to refer this icon to the function of “deleting”. It’s vital to provoke “intuitive” response which serves similar functions in real world. Also, in consideration of the complexity of the semiotic reference, one icon can logically refers to several meanings according to conventions that govern our engagement with the world and the media. Therefore the designers should also be able to distinguish among the many possible conventions while designing the interactive machine interfaces.
Secondly, the interface shall be transparent. By transparent the most evident explain is “to provide immediate feedback”, and in Murray’s opinion it’s a better design value than intuitive. Immediacy allows the interactive machine to be more similar to electronic devices that we are familiar with. Like a switch can turn on a light immediately, the mouse pointer, light pen, or touchable devices should allow the users see the changes immediately after the operation. This is helpful when a user encounters a new interfaces and be a faster learner due to familiar interaction patterns.
Also as she mentions, “a robust digital media process should pay attention to the values of all the relevant design disciplines and media traditions.” As a cultural practice, interactive interface designers should always bear in mind to consciously and in advance exploit the users’ convention and pre-existing knowledge.
In conclusion, intuition and transparency are two main features of interactive machine. To realize these two goals require the insight in the cultural tradition.
4.2 Closed Systems in OIA
Technically, the media of OIA can hardly be called as “interactive system”. As illustrated above, most of the OIA are not open systems. Instead they are modeled by algorithms and directly yield the outputs as coded once being activated by the inputs. No matter how interactive it may looks like in the outside, deep down the interaction is determined by a closed, non-interactive system, shutting out the external world once receive it inputs, and can not learn from the outside and adapted to experience. However, it is stilled called interactive advertisements as long as it can respond to audience’s operation. So although it is wrongly named technically, the OIA’s interactive system along with its interface create a novelty in the crowd and bring interaction to the audience. And their design value and principles are quite different from the interactive media I presented above.
4.2.1 Unnecessarily Intuitive and Transparent
After hijacking the attention, the OIA have to keep the audience interested until they make the right interaction. Under this circumstances, intuitive and transparent are vital. Furthermore, applying the three design elements and five senses to OIA, the immediate feedback can be designed in limitless possibilities.
On the one hand, it is important that the interfaces design follow the rules of the audience’s convention and their experience. They should not only be able to know what to do, but know that immediately, by right of pre-exist experiences. Like the bus stop OIA made by Guardian of the Galaxy, which remediate the old radio machine and let audiences to know right away the plughole is used for earphone. And from the early experiences, they are expecting to hear music after they plug in.
However on the other hand, intuition and transparency are deliberately avoided in OIA. As advertising strategies, ads often create a sense of suspense or appeal for shockness to make a strong impression and henceforth improve brand recall rate. So do OIA. So sometimes, the ultimate results to audience’s engagement are delayed by using visual effects to make suspense and reinforce the artistic appeal; sometimes the feedback is not necessary to meet expectations of the audiences, create surprise moment or shock feelings. Like in the following OIA made by Nestle Contrex in Paris, audience only know that they are wanted to ride on the bicycles at first. And once they ride the pedal, the beam is emitted from the bike, while no one know what happens next, until the lights on the wall converge and start to transform into a show. Although there’s beam emitted right after audiences’ engagement, the beam per se simulates the electric current, yet is way slower than real speed of electron. In this scenario, the audience may have a feeling about the results from their interaction, but the feedback is deliberately slowed down. The whole execution works against the value of transparency.
Furthermore, some OIA expect the passer-by to “discover” the feedback instead of showing them intuitively, by doing so create surprise and make impression. In the example showed below, the bus stop OIA combined the technology of augmented reality to cater to Axe’s global angle’s falling campaign. By showing only a bottle of Axe on the billboard, audience can’t know what to do or what will happen when they press that icon. In fact, when they press the button, an angel will fall down to the ground and walk towards you and wave and say Hi. These following sequence are not pre-exist conventions in the user’s head. Since the audience cannot trace any directions base on their previous experience and conventions about the world, the value of intuitive is completely abandoned in this kind of OIA.
In recent years, mobile applications and devices gradually change people’s way of consumption. The conventional agreement is that the consumer behaviour pattern has changed from AIDMA to AISAS. The most significant part of changing is that consumer stopped being passive receiver of the information, and turned to be positive searcher. This process makes “share” more and more important. OIA cost much more money and time per audience than other mass media, so the feature of “shareable” becomes economical and beneficial.
One of the strategies used is “personal designed advertisements.” These ads will be activated by single stimulus which will give audience an illusion that this ad is designed just for him or her. When seeing this kind of ads, it will for a moment attract audience’s fully attention. Meanwhile the fantasies and self-ego identifications which the interactive design provided them will not only correspond with the brand or product, but will also arouse the desire to share. In the OIA below, the ad allow the audience to create superhero selfies and slightly adjust it. After being absorbed in this interaction, the OIA provide an option of share. You can share it directly through the interface, no matter to your own device or your friends. Furthermore, audience also may choose to take a selfies with the OIA billboard together, since it’s self-ego complacent to see one’s face on street billboard to some extent.
Another strategy called “augmented reality” provides similar choice of sharing. Even if there’s no direct choice to send through the OIA itself, the unique experience will encourage audiences to record this moment, and share. In other word, the value of “shareable” in OIA media design is two sided, one is providing option for audience on the interface (like the example above), one is satisfying audiences’ self-ego identification and spontaneously arouse the desire to share. A lot of augmented reality OIA have been very successful due to the above reasons, such as Axe’s Falling Angel Campaign, National Graphic Campaign, Hugs for Health Bus Stop Ad, etc. One common point is in these videos, you can see a lot of audiences taking pictures of themselves with the OIA after joining the fun, no matter in what country, in what kind of locations.
4.2.3 Directing the Process by Giving the Only Option
All interfaces include some degree of guidance directing users to operate the system. However, as a metamedia with extreme transparent and simple affordances, OIA interfaces shows the guidance feature very evidently, and more salient than real interactive interfaces with open systems. With a closed system, audience’s options and responses are often narrowed down to only one option. People should understand what this OIA want them to do at the first sight and this require the interfaces to be designed with directions wrapped up by the design elements elaborated above.
Using text to lead audience is the most common and efficient way. The Women Aid ads use “Look At Me” which takes up half of the billboard to guide audience’s sight. In this case, the text must be conspicuous enough for the audience on first sight. Same requirement also fit to the image. Color composition can also play roles of cueing the interaction in the OIA. The Economist once made an OIA with a big light bulb hanging on a solid red background. The light bulb was installed with motion sensor that once someone walked under it, the light bulb will be suddenly lit up. The color composition was concise and terse, laying stress on the key part of the ad——the light bulb.
More complex directions combine the three elements together. Like the OIA below, the interface direction is constituted by a two parts. One is the text and camera icon which direct the audience to touch the icon to start taking selfies. The countdown number on the lower left also limit the time which the audience should be ready. The other is the image and color composition which draws out the area where the audience should be put themselves in. If don’t follow the above directions constituted by the three elements of advertising, the interaction will be failed.
The truth is that these directions reduce interacity by exclude other options. The audiences are provided with only direction, which become the predictable external inputs. In real interactive machine design, designers also have to direct users’ operation, yet not to this extent. In open systems, inputs are completely uncontrollable, so the machine can learn and update itself through interaction. So we can assume OIA which use closed systems and removes the unpredictable agents can never update themselves. By directing the process, the OIA give audiences the only option, and by doing so assure every interaction is exactly and without accident the same.
4.2.4 Treats as Feedback
Unlike traditional advertisements, some OIA designers specially design treats as feedback to audiences’ option. In this case, when audiences complete the whole operations directed by the designers, some treats will be offered as feedbacks. The treats can be virtual like a personal designed selfie like I presented before, or real like the one below. Also they can be related to the brand they are promoting, or not. Those automatically give-away actions are coded as part of the algorithm so will continue until the stocks are empty. As the OIA below, the no-sugar Coca-Cola are distributed as treats if the audience shout out “YES” to the voice-activator. In this case, the shouting as directed can be seen as the external inputs, the outputs or feedbacks are designed as giving away a bottle of drinks as they are promoting. The treats are real, and related to the brand.
Some traditional OIA also use these strategies, to give away some treats as feed backs. However compared to the digital OIA, those traditional OIA can either arrange staffs standing aside to control the scene and create a promotion environment, or leaving the OIA alone working their magic by themselves. In this case, the treats are not the feedback to audience’s interaction. In these traditional OIA, the feedback can also be either related to the brand like the La Place Restaurant traditional OIA which give away fresh fruits, or related to the timing or place like the Wilkinson Razor traditional OIA which give away roses in Valentine’s Day. However in this case, the traditional OIA can only give away real treats to achieve the purpose of publicity.
4.3 Open Systems in OIA
Despite the closed interactive system discussed above, some advertisers do use open system in OIA interfaces. It should be granted that the use of open system can’t guarantee the ad to be more interactive nor more effective than the ad using closed system. However, with the technology developing so fast, and people are always fond of the new and tired of the old, soon the OIA with closed system will be at every corner. Without the novelty and freshness, how long will the OIA stand its advantages? So the use of open system in OIA is a promising attempt. Who knows, one day, you might be able to chat with advertisements on street.
Wegner describe the interactive-identity machines (IIMs) as interactive machines which output their inputs immediately without transforming. In article he stressed: “Though IIMs are not inherently intelligent, they can behave intelligently by replacing intelligent inputs from the environment.” In the field of OIA, many designers decided to take advantage of the intelligent but simple feature of IIM, transforming the environment directly and achieve interactive results. Normally, this kind of OIA install video player on site, and broadcast live scene from another site. Audiences are filmed through camera and footage directly sent back to the place where the video are filmed. Two groups of agents and two places are connected by a screen, whose only function is transfer real-time images. Like the OIA showed below, the bus stop billboard is actually a huge screen, broadcasting real-time images of an old grandpa. Audiences can interact with the OIA interfaces, but are in fact interacting with the man behind camera.
Other examples like Talking Tunas, Mattel’s Draw Something, etc. IIMs like this make OIA seem more interactive than the one with closed systems. They take in inputs from the unpredictable outside world which makes themselves have richer behaviour than the closed systems.
4.3.2 Touch-screen Video Games
Another kind of OIA with open systems is those implant games within. The modes of use in these OIA verify with the kind of games they implant, but normally are simple and similar. With interaction like a video game, it rely on audience’s previous experiences of game common sense to some extend. Also the game are normally designed as touch-screen games, so the sensitivity is salient. In this case, transparency and intuition are two principles that matters.
Game OIA remediate the real touch-screen video games yet are placed in public. But the interacity varies according to different ads. In the Surf Life Saving OIA, audience can only touch limited area and there’s only two modes as on and off in these icons. It’s easy to see that behind the interfaces are still algorithms and the systems are not open enough. However, by providing more than one choice, these ads simulate the early time videogames and henceforth create a more interactive experiences than those who leave the audience countable options. There are OIA like Adobe EchoSign Digital Sign-off Game that can provide different levels of difficulties; There are OIA like San Jose Earthquakes Shooting Game that allow audiences to choose every directions they want; There are OIA like Pepsi Puzzle Game which allow the audience to move the pieces wherever they want. They also add gravity effect in this game which make it more like the interactive pad. Etc.
OIA with open systems are receiving higher involvement of interaction than those with closed system. They are designed to cope with the unpredictable external world. And the principles for intuition and transparency are more valued than the closed systems. Besides, the “shareable” and “treats as feedbacks” features are also followed.
I’m not assuming that open systems are more effective than closed ones. Just as new media didn’t eliminate traditional media, OIA with closed systems can be more effective and provide impressive interaction no less than the ones with open systems. So I believe with the more and more technology applied in the OIA, there will always be novelty on street which keeps hijacking attentions, while those OIA can be either designed with open or closed systems.
4.4 Semiotic and Culture Practice
Advertising is a field that has close relationships with semiotic and culture, so every piece of work shall also been seen as a practice in this field and follow its rules. The reason of this design matter is that designers should always have a deep understanding of the references in the symbolic world to avoid taboos and to design OIA that can convey information properly.
In Peirce’s interpretant level table, people come into the perceptual information first and then recognize them as patterns. In most cases, traditional OIA which can’t attract fully attention will stayed in this layer because they only work as background and are not specifically adsorbed. But OIA are designed to have fully engagement with its audiences. As metamedia, OIA probably contain every channel (language, music components, image components, film/video) in the table for audiences. The three basic elements in the advertising which I listed before can be viewed as signifying features of these media. The audience can view the text as part and understand what to do next, and can also combine the text and the images as whole to understand how to perform a better interaction. Furthermore, most advertisements are coded within the environment of culture. Like the Light Bulb ad of The Economist presented above, the whole composition of a solid background and an impending light bulb don’t refer to simple meanings like “the light is turned on”. In fact, by putting it above a person’s head, and lighting it up have a semantic meanings of “inspired” or “bright”. Furthermore, the cultural encyclopedia not only help with the designers to create novel ideas within shared symbolic societies, but also help them to avoid taboos. Like the Axe’s Falling Angel Campaign can not be placed in the Muslim Society not because people there can’t understand it, but because in their shared cultural encyclopedia, it’s inappropriate.
In this paper I mainly discussed one kind of OIA which have digital interfaces and interact with human. Although most of the OIA listed above are designed with closed systems, to audiences’ mind they are interactive enough and have influence towards their affection, cognition and behaviour. Scholars have discussed what principles an interface of interactive machine should follow, however OIA have its own values.
As described, the value of intuition and transparency is not mandatory due to the need of producing surprise and suspense in OIA. However in those which implant games within, the interface should still be designed as intuitive and transparent. Secondly, OIA have advanced conditions to be designed with an option of share and with treats as feedbacks. Also, because of the closed system, most of the OIA can only produce simple options for the audience to operate. So designers should be able to direct process on the interface. Finally, as a cultural and semiotic practice, OIA designers should also follow the values of proper symbol and culture pattern.
Arbab, Farhad. “Computing and Interaction.” In Interactive Computation, 9–23. Springer, Berlin, Heidelberg, 2006.
Bergstrom, Jennifer Romano, and Andrew Schall. “What Is Eye Tracking?” In Eye Tracking in User Experience Design. Morgan Kaufmann, 2014.
Bierma, Nathan. “‘Minority Report’ Stirs Questions over What Advertising Will Be in the Future.” Knight Ridder Tribune Business News; Washington. June 25, 2002.
Duchowski, Andrew T. “Eye Tracking Techniques.” In Eye Tracking Methodology, 49–57. Springer, Cham, 2017.
Goldin, Dina Q., Scott A. Smolka, Paul C. Attie, and Elaine L. Sonderegger. “Turing Machines, Transition Systems, and Interaction.” Information and Computation, Special Issue Commemorating the 50th Birthday Anniversary of Paris C. Kanellakis, 194, no. 2 (November 1, 2004): 101–28.
Leeuwen, Jan van, and Jiří Wiedermann. “Beyond the Turing Limit: Evolving Interactive Systems.” In SOFSEM 2001: Theory and Practice of Informatics, 90–109. Lecture Notes in Computer Science. Springer, Berlin, Heidelberg, 2001.
Liu, Yuping, and L. J. Shrum. “A Dual-Process Model of Interactivity Effects.” Journal of Advertising 38, no. 2 (2009): 53–68. http://www.jstor.org/stable/27749635.
Wang, Jing. “Characteristics and Application of Interactive Outdoor Advertising Design Creativity in the Context of New Media.” Hubei Institute of Fine Art, 2016.
Wegner, Peter. “Why Interaction Is More Powerful Than Algorithms.” Commun. ACM 40, no. 5 (May 1997): 80–91.
Abstract: This article is an analysis of individual cultural identity in semiotic perspective. Discussions are around how cultural symbols and cultural stories have impacts on individual identity. It is also discussed under contemporary context, including how modern media affect the communication of cultural symbols and stories, their effects on cultural identity and how science and technology change the formation of individual identity.
Cultural identity is an important topic in the field of sociology, politics, religions, cultural studies, public psychology, and many others. Although more or less different from each other, the definition of cultural identity usually covers the characteristic of the individual and the whole culturally identical group he lives in. Many fruitful researches have been done about how cultural identity is established and how it affects people’s activities in a community from different angles (Korte, 2007), including comparative culture studying, political, social psychological and others. Here in this article, I would use some semiotic concepts and theories to discuss the construction of cultural identity.
【1】Identity from cultural symbols and stories
When biologists Francis Crick and James Watson first decoded the correct double-helix model of DNA in 1953, they might not anticipate what unprecedented change it brought about, in the way people viewed the nature and the whole universe. In following decades, DNA researches have turned into thousands of applications in both academic field and people’s daily life. Recently, a company called 23andme, whose slogan says “we are reinventing the way you see your ancestry—through science”, starts quite a welcoming business. Through lab analysis of the saliva sample, clients can get a report telling their family history, ancestry’s timeline and trajectory, and “DNA relatives” from the genetic data contained in their DNA.
Looks like just a fancy, high-tech business.
However, it’s way more than that. Science, here to be exact the biology, offers human beings consciousness, and even rights, to build identity by their own, instead of accepting unconsciously what is said and authorized by others. This technology, along with other modern methods, is so remarkable that it can reform the way people form their identity, which hasn’t changed for thousands of years since the beginning of human civilization.
In contrast, before such “high-tech” stuff occur, people define themselves, and the group they stay in, by collective memory.
This is definitely not a new concept. First theorized and investigated by French scholar Halbwachs (Halbwachs, 1958), collective memory has been a phrase commonly referred in social and cultural studies. Generally, based on the fact that humans are social beings, collective memory is defined as a mechanism, by which a group of people acquire some certain mutual views, melted and remolded every time passed to the following generation. The mutual views here include ideology, sense of value, lifestyle and so many other basic aspects of the foundation of a community, and identity of each individual is one of the most vital components.
Then, traditionally, what is meant, or what is covered under, by saying identity?
First, every identity originates from some cultural symbols. For example, the Chinese nation regards itself as “descendants of Dragon” for a long history, an animal usually featured as a long nimble body, countless feet, horns on the head, and fierce lion-like faces. Clearly, dragon doesn’t exist physically in ancient China, or any corner of the nature, but comes totally from imagination. People picture it so vividly, and everyone else bathing in the same culture also buy it and never feel that it makes no sense.
Because such dragon is a culture symbol. Every component of the dragon’s appearance maps to, thus standing for, an existed meaningful concept already accepted by members in the same culture. Therefore, as the creature with mixed appearance, dragon functions as a symbol of many good qualities people value in their culture – lion’s bravery, snake’s swiftness, horn-y animal’s strength. More than the combination of all its components, it also becomes a spirit vehicle, dropping old and carrying new meanings with the time changing. When the empire values Taoism (a traditional philosophy of “doing nothing” in life and less control over citizens in politics), dragon can even be interpreted and appreciated as a symbol staying far away secular struggles, keeping a detached state of mind all the time, and its fierce side will no longer be emphasized. As a result, people living in different times of ancient China may even hold different images of dragon — “what does it mean” in turn reshapes “how does it look like”.
In this way, the mythical, weird-looking animal dragon is constructed into a cultural symbol through efforts by generations of people. A very small group of them contribute in, with all kinds of purposes, updating and spreading the meaning carried by this icon (and its matching written character as long as the writing system was invented), and more people just attend unconsciously in the process of acknowledging, memorizing, then passing on such meanings and their container, namely cultural symbols.
Besides dragon, porcelain, tea, Chinese knit, bamboo, plum blossom, dumplings and many others are all cultural symbols holding its unique meanings and qualities. Being moral, being modest, being exquisite, staying harmony, staying united with your family – these are what such objects eventually imply as a result of long-term mutual construction. Accumulated and recognized by different generations, such objects are then abstracted into symbols, leaving behind pragmatic or maybe aesthetic functions.
This process assists to establish the identity of each individual under the same cultural context. People visualize qualities they value through such symbols and externalize the ideal state implied by the whole group of people in a certain era. These symbols then serve as reminders and spirit models for the young generations, as well as unvoiced answers to questions like “what kind of people we should be like”.
To some extent, such process builds a portion of cultural identity. Identity can’t be only in the form of abstract concepts, or it will be hard to express or last long enough. Instead, identity requires some tangible and flexible vehicles, so people can easily understand and remember, also remove or add contents in accord with the historic context. When Chinese people say they are “descendants of dragon”, they are using all good qualities constructed on this fiction animal from former history, to annotate, also concentrate the explanation about who they are. Cultural symbols play an important role in people’s cultural identity.
Besides cultural symbols, cultural stories are another source of identity.
From myths of ancestor’s origin to heroic legends, almost all nations have own stories as validation of their unique greatness. These stories usually come from oral rhythms or songs, refined and processed through a long history until the invention of writing system, then carved on tablets or written on paper.
However, it’s not the end. In early 20th century, researches of Chinese ancient myths uncovered how nation’s early stories were “accumulated”. (Gu, 1926, P80). In terms of ancient Chinese records, researches pointed out that the later the record was, the earlier the story was said to happen, as well as the greater and more detailed the plot was.
Similar as, if not more than, cultural symbols, cultural stories are also constructed and memorized among a continued cultural community. The relationship between cultural symbols and cultural stories are not simply affiliated. Sometimes cultural symbols will occur in stories, as critical bricks accumulating into blocks with all bunches of meanings already accumulated on themselves. While sometimes, stories also generate symbols, by concentrating the context and content into simple visualized elements.
Under the same cultural context, stories about early legends usually have common elements. Ancient Chinese great figures are always conceived when a young girl meet a mythical animal, and then born with surroundings of thunders and lightings in the sky or flames in the house. Such common elements, plots and ways of narratives in cultural stories form a unique pattern, which people in the same cultural community will follow when generating new “legend stories” with all kinds of purposes.
As cultural symbols do, such patterns of cultural stories function as tangible and flexible vehicles, yet carrying collective meanings in a combined group and multiple levels. Thunders and lightings imply the philosophic theory of numinous association between the “heaven” and the world. Flame is a sign of the “fire element”, one of the five that claimed as ingredients of everything in the universe. As a result, such cultural stories and their patterns delineate the frame of people’s views of external world and views of their own positions in this world. Cultural identity is shaped when people start to answer the questions like “where does our world come from”, or “where are we come from”. Identity comes clearer in the process of repeating and enriching stories.
Only very few number of cultural stories can be passed on through history, but patterns of stories usually survive. People use similar patterns to carry stories, refine them by features and values of a specific era, with the ultimate aim to convince themselves and following generations of who they are.
Furthermore, in cultural stories there’re usually enemies. Most heroic behaviors are about killing Others, and the origin of a community comes with a glorious victory over an opposed group. Not to mention that the pattern of cultural stories makes it especially easy to fill in any enemies necessary for the reality. In this way, people who share the same cultural stories will have a clear sense of who they are not, besides who they are. Cultural stories then help establish identity by excluding others.
Given that above, when we say identity, cultural symbols and cultural stories are two implications cannot be avoided. They are both constructed. All people in the same cultural context attend in this process. In symbols and stories, they render, thus visualize, values and traditions of their times and nation. People take symbols and stories as the extended storage of answers to questions like “who we are”, “who we are not”, or “what kind of people we should be like”. Cultural stories also have an important function as excluding Others. Consequently, identity is shaped in contents of symbols and stories and the process of constructing contents, and then passed on generation to generation.
All these compose collective memories.
【2】 identity from modern media and new methods
Back to that fancy “23andme” we mentioned in the very beginning. This technology is so unprecedented because it subverts the tradition way how people get their cultural identity. That long-term construction walks away, instead the genetic data, which is absolutely concrete, objective and unchangeable, takes over the authority to exclusively explain all questions about who you are and how does your family come to this place. Identity here is obtained from a new, modern, “high-tech” source. Humans, as social beings since the first day of civilization, never have such an experience that their positions in this world, in this space-time, have already lied in their own body, and no greater one else is required to tell them the whole story.
It doesn’t mean that all cultural symbols and cultural stories don’t make sense any more. They still carry values and the traditions, show mutual interests and beliefs of a group of people, but have less functions than ever on shaping and explaining people’s identity by limiting them in this frame. Collective memory requires great efforts and exclusive faith, but now there’re more options.
Identity find a more tangible and flexible vehicle in our times. Evolution theory explains how you come here, heredity explains why you are like this and what you can probably be like in the future, anatomy and neuroscience explains how do you physically and mentally function, and all these lead to a reasonable, objective, structurally complicated yet logically simple result. These modern methods not only offer new contents, but the fact that identity can come from other ways.
Besides modern scientific methods, modern media also contribute in changing people’s way to think of themselves.
Originally proposed by Shannon in middle 20th century, information theory studies the process of information being communicated through electrical channel. One fundamental topic is random lossy data during the communication. The theory claims that redundancy is required when encoding the original data to achieve more accurate outcome. When information like cultural symbols and stories are communicated in today’s context of globalization, it also experiences such a process of encoding, decoding, and data losing.
With the help of modern media, it’s much easier to convey cultural symbols and it happens more often than ever. Symbols, as an association of the icon pattern (sometimes also the sound or even taste pattern) with meanings constructed onto, usually more or less inversely de-associate during such process of communication. It is imaginable, since context and the long history of collective memories related to certain cultural symbol are more or less lost when communicated. For example, dragon, the honorable symbol for people in Chinese culture, once raises controversy when touched by other cultures for its fierce icon. Misinterpretations happen when symbols conveyed to a group of people don’t have collective memories, at the same time contexts are also lost, about constructed meanings and the process of constructing.
Modern media enables communication happen extremely fast and often, which makes it also common that symbols are just left as icons yet contexts are not accurate or not enough, so the process of correctly representing meanings is hard to realize. As a result, new contents pop into those incomplete symbols, then new connections are built between the same icon and not exactly the same meanings. The new contents popping into usually rely on exited cultural symbols among the group of people on the receiving side of this communication. Besides the ancient Chinese symbol dragon’s fierce icon, ominous and brutal meanings constructed on this icon in western culture are also the reason for its misreading. It is common that such existed contents pop into the incomplete symbol, joining its icon, lead to an unexpected interpretation.
Therefore, cultural symbols leave others a different impression way from original contents. With frequent communications following, such difference is sent back, building another level on that misinterpreted symbol among its original cultural community. “How others think of us” then reflects upon “how we think of us”. “Negative” impression of the dragon leads to a further clarification or exclusion— “it doesn’t mean……it should be……”—or emphasis on certain values constructed in the past but missed in the process of communication— “in fact, we people are like……”.
In this sense, incomplete communication and misinterpretation cause a reflection on individual cultural identity. In addition, the globalization and gradually frequent cross-culture contacts in the past few centuries also smashed down the sense of “uniqueness” of the self culture. People realized (might quite unwillingly) that its symbols, its stories, were no longer the only orthodox in the universe. In other cultures, the symbol Dragon experienced another way of construction, which valued different things, told stories in completely different patterns and narratives.
It should be emphasized that such decentralization happens by various means, and the conflict in cultural symbols and stories is only one of them. As a result, collective memories on certain symbols may be forced to narrow down, focusing on something more specific and special. For instance, by emphasizing the Green tea, ancient Chinese intellectuals reinforced their identity as being in a civilized, ethical and knowledgeable class, which was traditionally obtained through just the “tea” symbol yet blurred for the popularity and traditions of Black tea in other cultures became known on ancient Chinese land with increasing ocean trades. Such inner-adaption of the cultural symbols’ construction is driven by an unconscious need for unique cultural identity, and certainly in turn has impacts on the way people view themselves.
Modern media contribute in frequent and mass communication of cultural symbols and stories. They not only function as the medium, but sometimes reshape the information flowing through.
In early times, writing system and paper were the main medium to convey cultural symbols. People from outside could investigate the culture, then wrote down the introduction of a certain symbol in their own languages, and record the whole history of how meanings were constructed through generations. As a type of media, books are expected to be formal, knowledgeable, overall and logical when conveying information, in both the content and the way of narratives. So for Others, if they read a book about Dragon, they will very likely to be informed of the context and the process of how meanings are constructed behind this cultural symbol.
But in modern society, more media are designed to be in favor of images, videos or other more visualized forms, which is also preferred by the audience. They are expected to be vivid, impressive, attention-catching, dynamic yet superficial and segmented. Under this circumstance, the result of communication of cultural symbols like Dragon will be different for sure. Through the image-based medium, it’s very likely that the context is incomplete, which is required to fully generate accurate meanings. People watch a lovely icon of dragon on the screen, probably with lines of annotation aside. These are all they have to piece together into the symbol Dragon.
The affordance of different type of medium cause different degrees of the communication of cultural symbols and stories. It isn’t necessarily true that one type of medium does always better than others. But it’s true that the symbol, with an icon and the content constructed onto, is partly lost or changed in this process. Incomplete decoding, over decoding or misinterpretation may also emerge. All these may have effects on cultural identity as discussions before.
As described, in the long history of human civilization, cultural symbols and cultural stories are closely related to the formation of cultural identity. For one thing, symbols and stories discussed here are both constructed by a group of people sharing the same cultural context. They put qualities and traditions they value in symbols and stories, take them as a reminder and illustration for themselves and others who are identified with the same group, also a visualized model for younger generations to follow. For the other, cultural stories also serve the functions to exclude Others by showing the uniqueness of the self.
All cultural symbols and stories are dynamic. Every generation attends in the process of constructing meanings for the purpose of emphasizing the priority of a certain class, enhancing relationship among members, motivating fights against enemies, etc. All things happening in this long-term, continuing construction compose collective memories of a community.
All these have impacts on individual cultural identity. People identity themselves by reinforcing certain qualities and tradition, by answering questions like “who are we”, “where do we come from” or “what we should be like”, also by constructing an imagined community and excluding others.
In modern times, however, things become more complicated. Frequent communications between cultures usually lead to some data loss of cultural symbols and stories. The icon and meanings constructed on the icon are sometimes disconnected or missing, which makes the representing of meanings in the receiving side go beyond original intention. As a result, new meanings may pop into the original icon, combined together as different cultural symbols even though they look similar. Frequent contacts then make people realize their symbols and stories are no longer unique. Also, affordance of different types of modern media will cause different degrees of the communications of cultural symbols and stories.
Modern medium and communications result in changes in cultural symbols and stories, which further result in changes in individual cultural identity. People see their reflections through others’ eyes, reinforce their own identity by clarifying or specifying their original symbols and stories. Of course, some symbols and stories may be disappeared in this process.
What’s more, new options are also provided. Science and technology enable people to view themselves in completely new ways. Identity is no longer obtained from collective memories, but lies stably and objectively in everyone’s body. Cultural symbols and stories are experiencing a new way of construction in modern times.
French, B. M. (2012). The Semiotics of Collective Memories. Annual Review of Anthropology, 41(1), 337–353. https://doi.org/10.1146/annurev-anthro-081309-145936
Gentz, N., Kramer, S., & ProQuest (Firm) (Eds.). (2006). Globalization, cultural identities, and media representations. Albany: State University of New York Press.