Abstract

In this paper, I provide an overview of smartpen technology, using the Livescribe Echo Smartpen as my primary example, in order to trace the conceptual development of this technology as a tool for sign formation and abstraction. Using the conceptual frameworks of C.S. Pierce, Andy Clark and David Chalmers, I demonstrate the ways in which the smartpen and smartpaper interface enables users to perform meaning-making through advanced forms of cognitive offloading. Ultimately, analyzing smartpen technology through these frameworks provides a deeper understanding of the unique semiotic functions afforded by the development of pen and paper into digital technologies.

Introduction

The inscription of knowledge for functions ranging from personal use to general communication to overall cultural progression has long been achieved through the artifacts of pen and paper. As an interface, these tools allow for useful extensions of human cognitive functions, particularly in regard to acting as external storage devices for recalling and sharing meaning. However, digital and computational technologies, such as the PC, tablet, or smartphone, which facilitate dynamic digital information processing, have in many ways perhaps surpassed the pen as tools for higher level meaning-making and abstraction. However, smartpen technology, which integrates digital and computational capabilities into its interface, gives users the ability to transform handwritten text into digital text for dynamic information processing. Thus, through an exploration of the smartpen and its functions, I will demonstrate the ways in which smartpen technology provides a technological interface for cognitive offloading and meaning-making that advances the semiotic capabilities of the culturally ingrained artifacts of pen and paper.

Disambiguation of Terms

There are multiple models of smartpens currently available, however, the Livescribe Echo Smartpen incorporates the most conceptually advanced technology at this time. As a result, I will use it as my primary example in this paper. The Livescribe Echo Smartpen is a computing system that, combined with “dot-enabled paper” and Echo Desktop software, which can be run on any PC or smart device, comprises a broader interface system architecture. For the purposes of this paper, I will use the following terminology to refer more broadly to these items and the concepts they represent using the following terms: “smartpen”, “smartpaper”, and “desktop software.”

Image Source: Livescribe Echo Gallery

De-Blackboxing Smartpen Technology

Like any ballpoint pen, the smartpen is just over six inches in length and contains a ballpoint ink cartridge. However, this is where the similarities between the two end. What the smartpen adds is an infrared camera, an OLED (organic light emitting diode) display, embedded microphone and speaker, audio jack, USB port, and, of course, a power button. These components are supported by internal flash memory and an ARM 9 processor and powered by a rechargeable lithium battery (Echo Smartpen Tech Specs). While the smartpen can be used to write standard text on any surface, its digital recording and computing functions are enabled when used in conjunction with smartpaper, which to the naked eye appears as a standard sheet of paper, but in actuality contains an intricate pattern of “microdots” across its surface. Finally, the system architecture of the smartpen is completed by a companion software program, which can be run on any PC or smart device (About Livescribe Dot Paper).

Image Source: Livescribe Echo Tech Specs

Within this system architecture and with these interface components the smartpen allows the user to do much more than simply write. When powered “on,” the infrared camera, allows the user to instantly record handwritten notes, scripts, or designs taken on smartpaper. The script or designs captured by this camera can be automatically synched with audio captured by the smartpen’s integrated microphone, which can then be played back through either the embedded speaker or through headphones attached to the standard audio jack (Echo Smartpen Tech Specs).

As mentioned, the smartpen works in conjunction with smartpaper, which comes in a variety of aesthetic and functional styles, but is unique in that its surface is completely covered with “microdots” (100 micrometers in diameter) positioned across an invisible grid (About Livescribe Dot Paper). The positioning of these microdots, approximately 0.3 mm between each dot, forms a larger pattern (See image). Each microdot represents a specific location within the pattern that is “activated” when contacted by the ink in the smartpen, allowing the infrared camera within the pen to capture and record exact patterns. This pattern recognition can occur simultaneously while recording audio in order to sync the recorded audio with exact points of text.

Image Source: About Livescribe Dot Paper

Of course, the recognition of a pattern on smartpaper and recorded audio would not be possible without the ability to digitally encode the content (Denning & Bell 470). The smartpen contains 2 GB of flash memory and an ARM 9 processor, powered by a rechargeable lithium battery. In addition to storing user-produced content, the smartpen’s memory houses the smartpen’s software for both encoding data, such as content recognition algorithms (Marggraff et. al), and various other bundled smartpen applications, such as a scientific and simple calculator, audio replay commands, translation software, and so on.

Image Source: Livescribe Echo Gallery

In terms of connectivity, there are several options for storing and organizing content collected by the smartpen. If the user is positioned near a Bluetooth-enabled smart device, the content collected by the smartpen can be transferred wirelessly and simultaneously, as the user inputs it on to the smartpaper. Alternatively, the content can be stored on the smartpen itself within its memory until the user is able to connect to Bluetooth and make the wireless transfer. Lastly, the smartpen’s USB port allows the user to dock the smartpen and connect to a PC in order to make a direct transfer.

Once the data from the smartpen has been transferred to a PC or smart device. The desktop software enables the user to perform multiple functions with the digitized text, images, and audio. Primarily, the software allows the user to reorganize individual sheets of paper into notebooks and reorder sheets within a notebook. In its digital form, the handwritten text also becomes searchable, allowing users to easily locate text within notebooks or even across notebooks. Furthermore, this software allows the user to transcribe the digital handwritten text into standard computer text for word processing. Sheets and notebooks, along with their synched audio, can easily be exported as PDFs and shared (Echo Smartpen Tech Specs).

Fundamental Concepts

In order to understand the advanced potential that the smartpen and smartpaper interface contain as a cognitive technology, I will rely on the theoretical contributions of C.S. Pierce, Andy Clark and David Chalmers. In particular, I will refer to Pierce’s concept of semiosis and the triadic nature of signs as a basis for understanding how meaning and abstractions are made (Irvine Grammar 17). In conjunction with Pierce’s theories, I will use the concept of “the extended mind” as developed by Andy Clark and David Chalmers to describe how the sign formation process, as a cognitive function, is extended into tools and artifacts (Clark & Chalmers).

Pierce’s Theory of Sign Formation

A sign is something by knowing which we know something more.

C.S. Pierce (Irvine Semiotics 16)                                                 

By “semiosis” I mean… an action, or influence, which is, or involves, a cooperation of three subjects, such as a sign, its object, and its interpretant…

C.S. Pierce (Irvine Semiotics 21)

To understand a sign, which perhaps may be simplified as something to which meaning is attributed and through which meaning is understood, one must understand how a sign is formed. According to C.S. Pierce, semiosis is the process through which meaning is made and interpreted within a sign. Furthermore, this process of meaning-making is acknowledged through a three-part relationship. In other words, for the meaning of any sign to be understood, there must be: 1) a physical manifestation of the sign to be interpreted, whether visual, audible, tactile, etc.; 2) an abstract concept or secondary physical object, which the physically manifested sign represents; and finally 3) an interpretant or meaning that is produced from the relationship between the physically manifested sign and the concept or object it represents. (Irvine Semiotics 14-5)

By explaining this three-part relationship within the process of semiosis, Pierce reveals three fundamental concepts of meaning-making. First, the relationship between the physical manifestation of the sign to be interpreted and the abstract concept or secondary physical object reveal the interdependence of signs within larger sign systems in order to create or understand meaning. Second, as the interpretant is produced through the relationship between the physically manifested sign and the object or concept it represents, rather than contained within either of these entities, we are able to understand that the interpretant itself is a sign. Third, and perhaps most importantly, in order for meaning to be understood, a sign’s interpretant must be acknowledged. Thus, there must be an active agent outside of this system capable of both observing the physical manifestation of the sign and drawing its connection to the object or concept it represents in order to understand the meaning that the interpretant or relationship between the two conveys.

The Extended Mind Theory

Bearing in mind Pierce’s concept of semiosis, the most logical active agent capable of acknowledging an interpretant as a sign’s meaning is our human cognitive system. Though the specific process of human cognition remains an unobservable mystery (Barrett), we are able to observe the process of perceiving the physical manifestations of signs (i.e. viewing physical images or objects, listening to sounds, feeling materials, etc.), as well as the utilization of the meaning made with these signs (i.e. verbal, physical, emotional, etc. responses).

However, the work of Andy Clark and David Chalmers in The Extended Mind offers a unique alternative interpretation of this process. In essence, Clark and Chalmers assert that the cognitive functions of the human mind are not restricted to the mind itself, but rather through the use of tools and artifacts, the human mind can be “extended.” For example, take the primary cognitive function of memory. When an individual writes down a telephone number instead of memorizing it, the individual is, in essence, using the paper as an external memory in which to store information rather than retaining it within his own memory.

This example reveals two notable implications of Clark and Chalmers’ theory. First, the process of extending cognitive functions to an artifact or tool is an implicitly “external”, “active,” and productive process. In other words, not only can cognitive functions occur in an environment outside of one’s mind through artificial cognitive agents, but they can produce meaning that can then be used or applied within other cognitive agents, either human or artificial. Second, this example reveals that the idea of larger cognitive or system architectures and coupling. In other words, through the extension of human cognitive functions, systems of interdependency are created between cognitive agents that enable more efficient and ideally effective cognitive processes. Within these systems, the human cognitive agent couples with an artificial cognitive agent through which meaning can then be produced and used (Clark & Chalmers 7-9).

The Connection between These Concepts

The significance of Clark and Chalmers’ theories in the context of Pierce’s concepts of semiosis and sign formation is that when these processes are extended from the human mind into the artifacts and technologies that we use, we are able to create extremely powerful tools and networks of meaning-making. As Clark and Chalmers explain, these networks can be as simple as the coupling of one’s mind to a standard pen and paper interface (Clark & Chalmers 12-5). However, when coupled with more complex artifacts or more powerful technologies, the cognitive outputs become more complex and powerful as well, allowing for higher levels of meaning abstraction. With this in mind, we can begin to understand how significant the development of computing technologies as information processing systems has been for the progression of human thought.

In fact, even in the earliest conceptions of computer processing systems, such as Vannevar Bush’s “memex,” we find that much of the developmental inspiration of these systems stems from a need to facilitate storage and communication of information in ways that would facilitate higher levels of meaning-making (Bush). Bush’s vision provided the foundation for much of the information processing technologies that we have today. Other early computer designers progressed Bush’s vision even further. For example, Douglas Engelbart was responsible for revolutionary designs such as “hypertext” as a method for connecting multiple layers of digital content (Englebart). Donald Sutherland developed the Sketchpad, which provided an early example of design-based programming, through the use of a “light pen” (Sutherland). Also pivotal was Alan Kay, whose vision included mobilizing the personal computer interface into a singular portable device and the introduction of software applications within a user-programmed environment (Manovich 57).

These concepts, beginning with Pierce’s philosophical foundation and Clark and Chalmers’ explanation of how artifacts and technology facilitate higher levels of meaning making through cognitive extension, provide the basis I will use to investigate the smartpen as a cognitive tool for meaning-making and abstraction.

The Cognitive Advantage of the Smartpen & Smartpaper Interface

Smartpen technology presents a modular system architecture consisting of three primary components: the smartpen itself, smartpaper, and the desktop software (Baldwin & Clark 63). Both the smartpen and the smartpaper contain mechanical interfaces that can be manipulated by a user to produce analog data (Denning 470). For example, the ink of the smartpen is standard ballpoint ink which can be used to write standard text on a standard piece of paper, while the smartpaper can be used as a standard notebook if written on with standard ink. The digital interface of the smartpen and smartpaper is created when the powered smartpen is used to input information on the smartpaper by bringing the ink cartridge in direct contact with the microdots contained on the smartpaper grid. The interface of the system software on the other hand offers a standard set of functions that depend on the device on which it is run, such as a PC, tablet, or even smartphone.

Like any artifact, the smartpen and smartpaper interface is a semiotic tool that facilitates the expansion of sign creation and interpretation. According to C.S. Pierce there are three fundamental types of signs: icons, indices, symbols. Iconic signs are those that can be interpreted through representations of likeness. Indices are signs that point to other signs, such as an arrow that directs the indicate directions or a car horn that redirects the attention of the individual who hears the horn. Most importantly, however, indices reveal the positioning of signs within a larger system of meaning. Implicit within indication of the direction that one must go, are the other directions that one must not go. In addition, the sound of the horn reveals that there must be some active source making the sound. Thus, in recognizing an indexical sign, there are both clear and implicit connections to other signs. Finally, symbols are signs to which the correlation between the sign itself and the object it represents is attributed rather than contained or apparent within the sign itself (Irvine Semiotics 18-9).

As an interface, the smartpen and smartpaper system is designed to facilitate indexical sign formation. In other words, the microdot grid that the smartpaper contains functions as a larger meaning system in which any added marks or script inherently become signs that are indicated by specific locations within the grid. Thus, the grid itself acts as a set of indexical signs through which users can develop additional signs of any type, whether iconic, indexical or symbolic. What is unique, however, is the degree of control and precision over the digital formation of signs that this particular interface gives to users. Rather than relying on predeveloped icons or symbols of a standardized PC or smart device interface, the user can design their own signs to represent meaning. This allows the user to work with signs that are more clear and easily interpretable by the user, which in turn may facilitate more powerful forms of cognitive offloading. In other words, if the user is able to work with signs that they can understand more intuitively, this relieves the user of the cognitive burden of learning new signs and then recalling their meaning.

Furthermore, through the modular components of the smartpen system architecture, we can begin to breakdown the various layers of meaning that exist within each interface (Irvine Powerpoint Slide #69). As any standard pen and paper interface, the smartpen and smartpaper interface is at its base a medium for abstracting the sign systems of handwritten icons or symbols. Through the addition of the smartpen’s camera and microphone, however, this interface is transformed into a medium for also abstracting the sign systems of sounds and images. This layer of abstraction is formed through the ability to “program” functions into handwritten text, creating a unique form of hyperlinked text. Specifically, the process of capturing and synching handwritten text and recorded audio, activates a specific location on the microdot grid. In doing so, it programs a command function for the smartpen’s camera. As a result, whenever the smartpen “taps” the handwritten text, it initiates the audio playback function of the pen. In addition to these user-created programs, there are certain preprogrammed command functions that the smartpen contains as well. For example, users can control the smartpen’s utility functions by drawing a small cross to use as arrow indicators.

The processing capabilities of the smartpen combined with the microdot pattern of the smartpaper grid, elevate the medium of this interface even further, allowing users to add layers of abstraction within the sign systems of time and space. To explain, as the user activates microdots on the grid of the smartpaper, the information that is collected is organized in a sequential manner and “bookmarked” by the system processor (Pettersson & Ericson). Consequently, this sequential organization and bookmarking process creates sign instances that are catalogued by the specific time in which they were captured. Additionally, the microdot pattern contained within the smartpaper grid establishes a spatial sign system for identifying specific instances of location.

Of course, transferring notes from the smartpen and smartpaper interface to the desktop software interface allows for even more layers of meaning abstraction. Particularly notable, however, is the degree to which this transfer blurs the lines of separation between the artifacts of pen and paper and digital word processing. Even though the smartpen and smartpaper interface on its own is digital, the ability to automatically sync notes to the desktop software as users write, allows users to see instant encoding of analog information into its digital form. Of course, this provides users a more transparent understanding of the combined interfaces (Murray 61). Within the desktop software, the user can then apply their tacit understanding of working within a desktop application to manipulate the digitally encoded information further. For example, the user can catalogue and rearrange sheets of digital information and even search through the text using the media-independent searching function of “Control F” (Manovich 122).

The Smartpen Grows Signs

Perhaps even more interesting is the advanced way in which smartpen technology effectively offloads the cognitive meaning-making process. In other words, by combining two sets of signs, one visual (the written script or design) and one audible (the recorded sound), a specific correlation between the two signs can be drawn to interpret meaning. In doing so, we observe the individual textual sign and audible signs merge into each other to form a new more complex sign for further, higher orders of abstraction. (Irvine Grammar 15)

Thus, as Pierce states “the essential function of a sign is to render inefficient relations efficient” (Irvine Semiotics 16). By combining standard text signs with audible signs, the Smartpen allows users to offload the cognitive function of remembering or recalling the contextual information about the written script or designs. This allows users to make more efficient use of various sign types, whether iconic, indexical or symbolic, in order to take notes. More specifically, this allows the individual to store metainformation, or information about the nature of information (Floridi 31), in the form of an audible object sign. As explained, if a user draws a diagram in order to depict a process that is being described audibly, the smartpen allows them to record and sync the process description to the diagram. In such an instance, the user has created a visual sign whose object, or the concept that the visual sign represents, is stored as metainformation that can be accessed by the user in order to aid in interpreting the sign.

Still, perhaps one of the most useful aspects of the Smartpen’s meaning-making capabilities lies in its human distributive cognitive functions, or the ability to distribute cognitive activity across human minds (Zhang & Patel). Through the smartpen’s desktop software, users can share information collected by the smartpen through various remediations of its original form, such as PDFs or emails (Livescribe(™)). This allows individuals the opportunity to share both written text and audio, but also acts as a form of individual significance parsing. For example, when the smart notes are shared, the recipient of the notes is able to listen to the recorded audio, while viewing the text. This allows the user to acknowledge not only the interpretants that are developed by the actual audio or text, but also make inferences about the interpretants that the individual who shared the notes originally acknowledged. Specifically, it allows the recipient to understand whatever hierarchy or organization the original user might have given to various aspects of this information.

This parsing process is in essence a step-by-step revelation of Pierce’s assertion that “all thought is in signs” and therefore “requires a time” in which each sign is actually interpreted, and which by interpreting creates a new sign. Essentially, the Smartpen creates a map of the “time” in which the audible sign is interpreted into a textual sign, creating a more complex sign that differs from simply the audible reading of a text. Rather than just restating the textual sign out loud, by translating it into the audible sign system, the smartpen integrates the meaning of these two sign systems, but adds the contextual meaning of a sign with the interpretation and context of the secondary sign (Irvine Semiotics 16).

Refining Smartpen Technology

Despite presenting unique opportunities for sign formation, the smartpen system architecture is not without its own usability issues. Primarily, as the smartpen and smartpaper interface is built on the culturally ingrained conventions of the standard pen and paper interface, the user is able to apply a relatively high degree of tacit knowledge in their mechanical or analogue usage of the smartpen and smartpaper interface (Murray 61). However, to a certain degree this tacit knowledge also inhibits usability by masking the digital affordances of the interface. For example, the smartpen and smartpaper both resemble a standard pen and paper, except for certain design cues, such as the power button on the smartpen or the control panel at the bottom of the page (See image), that indicate their digital capabilities. However, outside of the intuitive knowledge to use the pen as an instrument to make marks on the paper, to perform the digital functions of the pen requires a certain degree of learned user competency. For example, the user must learn to first turn on the pen prior to writing and then must learn the functions of each of the buttons on the smartpaper itself. Even more obscure, however, is the knowledge that the text itself contains additional information that can be accessed through tapping the text.

Image Source: Livescribe Faq: Using The Home Button

Of course, learning the functions of smartpaper requires a relatively simple learning curve. However, in order to ensure that the user properly initiates the digital components of the smartpen and smartpaper interface, the design of the smartpen itself could be improved. Namely, by adopting a design that incorporates a forcing function, or a design property that uses “the intrinsic properties of the representation to force a specific behavior” (Donald 34), that requires the user to power on the smartpen in order to write. Perhaps, such a change would be a simple as adopting the retractable pen design, which forces the user to press a button in order to eject the tip of the ink pen. The smartpen could incorporate this design feature within the process of powering on the smartpen itself.

Furthermore, the body of the smartpen, which is similar in length to a standard pen, is much wider in circumference than a standard pen. The smartpen’s larger circumference is currently an unavoidable and necessary property of its design, as it results from the components that make the smartpen capable of digital mediation: the camera, microphone, processor, memory, etc. The smartpen’s size, of course, may present a relative degree of discomfort for some users in terms of writing with the pen. More importantly, however, it inevitably forces the user to adapt their grip on the smartpen, which may reduce the degree of precision that the user can achieve in controlling their handwriting or designs. While this issue does not necessarily affect the functional usability of the smartpen, it could diminish the pen’s creative and design affordances.

Perhaps the most significant design issue with smartpen technology, however, is connected to its larger system network. While as two individual interfaces, the smartpen and smartpaper and the Echo desktop software, can function independently with their own unique sets of affordances, the modular nature of this architecture is not entirely seamless in terms of integration and efficiency. For example, the process of converting handwritten text into digital text is extensive: information must be input into the notebook, then transferred to the software application, and finally converted within the software. As a result, in cases where the user simply wants digital text, the smartpen and smartpaper interface is drastically more inefficient than simply writing within a PC or smart device word processing application.

Longterm Potential

Despite these issues in design, smartpen technology represents unique and valuable opportunities in sign formation and advanced cognitive offloading. In addition to successfully merging select functions of existing technologies in order to create unprecedented forms of mediation, smartpen technology’s greatest potential perhaps lies in the amount of mobility it could allow users. Through the development of various technologies, such as the laptop PC, tablets and smartphones, we can understand the importance and demand for portable electronic devices that allow for advanced cognitive functions (Manovich 102-11). Unlike a PC or even tablet, the smartpen is extremely portable and its internal storage allows users to store information within the smartpen itself. Furthermore, unlike smartphones, which rely on keyboards that do not facilitate comfortable or practical use for extensive information input, the smartpen provides a mobile information processing device that is both more ergonomic and practical for larger inputs of information.

Still, in order for smartpen technology to achieve its potential, at least two things would need to precede widespread adoption of this technology. Specifically, the current cost of the smartpen itself would need to become less inhibitive. However, as innovation with both smartpen and smartpaper technology continues and competing forms of the smartpen and smartpaper interface become available, the costs for this technology should decrease. More importantly, however, the standardization of smartpaper, on which the smartpen’s digital capabilities depend, would need to become more widespread. Of course, this would require widespread acknowledgment of the benefits of smartpen technology in terms of its cognitive offloading capabilities in order to increase demand for this technology.

Thus, assuming that smartpen technology is embraced, it could alter the way that we understand and use the artifacts of pen and paper. The line between analog and digital will be further blurred, granting individuals new perspectives for how they might use these artifacts as tools for meaning-making and, consequently, new forms of cultural creativity and expression. When smartpen technology is viewed through a semiotic perspective of this type, we can understand the unique opportunities that pen and paper as digital technologies represent for creating, preserving, and sharing meaning and knowledge.

Works Referenced

“About Livescribe Dot Paper.” 2016. Concept. Accessed December 13. http://www.livescribe.com/en-us/faq/online_help/Maps/Common/CRef_SP_About_c_about-livescribe-dot-paper.html.

Andersen, Peter Bogh. 2001. “What Semiotics Can and Can’t Do for HCI.” Knowledge-Based Systems 14: 419–24.

Atuma, S. S., K. Lundström, and J. Lindquist. 1975. “The Electrochemical Determination of Vitamin A. Part II. Further Voltammetric Determination of Vitamin A and Initial Work on the Determination of Vitamin D in the Presence of Vitamin A.” The Analyst 100 (1196): 827–34.

Baldwin, Carliss Y., and Kim B. Clark. 2000. Design Rules. Vol. 1. The Power of Modularity. Cambridge, Mass: MIT Press.

Barrett, John C. 2013. “The Archaeology of Mind: It’s Not What You Think.” Cambridge Archeological Journal 23 (No. 01): 1–17.

Bush, Vannevar. 1945. “As We May Think.” The Atlantic, July. http://www.theatlantic.com/magazine/archive/1945/07/as-we-may-think/303881/.

Clark, Andy, and David Chalmers. 1998. “The Extended Mind.” Analysis, Oxford University Press 58 (1): 7–19.

Denning, Peter J., and Tim Bell. n.d. “The Information Paradox.” American Scientist 100: 470–77.

Donald, Norman. 1991. “Cognitive Artifacts.” In Designing Interaction: Psychology at the Human-Computer Interface, 17–38. Cambridge University Press.

“Echo Desktop.” 2016. Accessed December 13. https://www.livescribe.com/en-us/smartpen/echo/echo_desktop.html.

“Echo Smartpen Tech Specs.” 2016. Accessed December 13. https://www.livescribe.com/en-us/smartpen/echo/echo_desktop.html.

Engelbart, Douglas. “Augmenting Human Intellect: A Conceptual Framework.” In The New Media Reader, edited by Noah Wardrip-Fruin and Nick Montfort, 93–108. Cambridge, MA: MIT Press, 2003.

Evans, David. Introduction to Computing: Explorations in Language, Logic, and Machines. August 19, 2011 edition. CreateSpace Independent Publishing Platform, Creative Commons Open Access: http://computingbook.org/.

Floridi, Luciano. 2010. Information: A Very Short Introduction. New York: Oxford University Press.

Irvine, Martin. 2016. Semiotics, Symbolic Cognition, and Technology Key Writings. Compiled and edited with commentary by Martin Irvine. Communication, Culture & Technology Program, Georgetown University.

———. 2016. The Grammar of Meaning Systems: Sign Systems, Symbolic Cognition, and Semiotics. Compiled and Edited with commentary by Martin Irvine. Communication, Culture & Technology Program, Georgetown University.

———. 2016. “Semiotics Foundations, 2: Media, Mediation, Interface, Metamedium.” PowerPoint Presentation.

Jackendoff, Ray. 2009. Foundations of Language: Brain, Meaning, Grammar, Evolution. Reprint. Oxford: Oxford Univ. Press.

Jayaram, H. N., D. A. Cooney, H. A. Milman, E. R. Homan, W. M. King, and E. J. Cragoe. 1975. “Ethacrynic Acid–an Inhibitor of L-Asparagine Synthetase.” Biochemical Pharmacology 24 (19): 1787–92.

Johnson, Jeff. 2014. Designing with The Mind in Mind: Simple Guide to Understanding User Interface Design Guidelines. Second edition. Amsterdam ; Boston: Elsevier, Morgan Kaufmann.

“Livescribe(™) Connect(™) Makes Handwritten and Spoken Information Easily Shareable with Facebook, Evernote(R), Google(TM) Docs and Email – all from Paper Livescribe Introduces the Affordable GBP99 2GB Echo Smartpen Starter Pack.” 2011. PR Newswire Europe Including UK Disclose. http://proxy.library.georgetown.edu/login?url=http://search.proquest.com/docview/896962663?accountid=11091.

“Livescribe Echo Gallery.” 2016. Accessed December 14. https://www.livescribe.com/en-us/smartpen/echo/photo.html

Manovich, Lev. 2013. Software Takes Command: Extending the Language of New Media. International Texts in Critical Media Aesthetics. New York ; London: Bloomsbury.

Marggraff, J., E. Leverett, T.L. Edgecomb, and A.S. Pesic. 2013. Grouping Variable Media Inputs to Reflect a User Session: US 8446297 B2. Google Patents. https://www.google.ch/patents/US8446297.

Moggridge, Bill. 2007. Designing Interactions. Cambridge, Mass: MIT Press.

Murray, Janet H. 2011. Inventing the Medium: Principles of Interaction Design as a Cultural Practice. Cambridge, Massachusetts: The MIT Press. http://site.ebrary.com/lib/alltitles/docDetail.action?docID=10520612.

Naone, Erica. 2016a. “Computing on Paper.” MIT Technology Review. Accessed December 7. https://www.technologyreview.com/s/409190/computing-on-paper/.

———. 2016b. “Taking Apart the Livescribe Pulse.” MIT Technology Review. Accessed December 7. https://www.technologyreview.com/s/409948/taking-apart-the-livescribe-pulse/.

Pettersson, M.P., and P. Ericson. 2007. Coding Pattern: US 7175095 B2. Google Patents. https://www.google.com/patents/US7175095.

Sutherland, Ivan. “Sketchpad: A Man-Machine Graphical Communication System.” In The New Media Reader, edited by Noah Wardrip-Fruin and Nick Montfort, 109–126. Cambridge, MA: MIT Press, 2003.

Zhang, Jiajie, and Vimla L. Patel. 2006. “Distributed Cognition, Representation, and Affordance.” Pragmatics & Cognition 14 (No. 2): 333–41.

 

Abstract

Surrealist art is one of the most elusive genres to define. It is rendered realistically, but represents unrealistic, dreamlike states. As André Breton maintained in Manifestoes of Surrealism, surrealist art is based on the “play of thought” and “omnipotence of dream” (Breton 26). This representation complicates how we make meaning. I will use the Peircean semiotic model combined with Ray Jackendoff’s Parallel Architecture to explore feature extraction within paintings under the genre of surrealist art.

The process of feature extraction is dependent on culture and socialization. According to Dr. Irvine, “Peirce saw that one of the most important affordances of human symbolic cognition is precisely that which is combined in the dual necessary “platform” of semiosis: symbolic thought and expression necessarily require material-perceptible sign systems functioning in necessarily social, collective, and intersubjective communities of meaning-making (Irvine, “A Student’s Annotable Peirce” 3). In other words, semiosis is dependent not only on an interpreter’s ability to think symbolically, but it depends on the social context he or she uses to assign meaning, in this case, to the pieces of the paintings. It is important to acknowledge the tension between the individual nature of interpretation and the social context in which it is based in order to explore our semiotic understanding from a more well-informed perspective. This exploration will begin with the question: How does the combination of culturally implied meaning and feature extraction impact our understanding of surrealist art? How does a de-blackboxing process occur?

To achieve a deeper understanding of the de-blackboxing process that occurs when unpacking the surrealist tradition, I will focus on two paintings: René Magritte’s The Human Condition, (or La Condition Humaine,) (1933) and Salvador Dali’s Melting Watch (1954). These paintings are instances of the surrealist genre where viewers draw meaning by simultaneously understanding and subverting reality. From these paintings, I will explore the dialogic nature of these cultural artefacts in the context of the continuous historical, social and artistic network in which they reside.

Introduction

The research question stated above prompts an understanding of Charles Sanders Peirce’s semiotic model and Ray Jackendoff’s Parallel Architecture as they apply to other meaning systems besides language. These models and ideas will serve as a foundation to explore my research question: Peirce’s triadic model includes the representamen, interpretant and object.

1. The representamen is the perceptible sign to which we associate meaning. In art, this would mean the elements of a painting (color, brush strokes, subject of image, etc.).
2. The interpretant is the relationship between the viewer (or cognitive agent) who makes associations and the representamen. In art, this would be the recognition of the connection between elements of a painting.
3. The object relates to the meaning associated with the representamen. In art, the emotional and/or social response generated by style and composition of a work.

Jackendoff’s Parallel Architecture can be leveraged beginning with this explanation: To comprehend language, Jackendoff gives us parallel architecture as a way to understand the relationship between semantics, syntax and phonology. These elements only produce sentences because they happen simultaneously (Jackendoff 126). Further, different combinations of these elements impact the construction and meaning of sentences, which we understand in the context of our environment/community. This idea will be extrapolated by focusing on the combinatorial power of various sign and symbol systems.

To understand how unpacking art is both similar and different than the process we use to understand language, I will include a historical and conceptual recovery, where I will use Jackendoff and Peirce’s ideas to make discoveries about surrealist art. Further, I will investigate what constitutes an appropriate interpretive possibility within the meaning system the paintings reside in (as well as the viewers who interpret them).

The historical perspective will include an examination into the meanings of these works in the dialogic network that precedes the artists as well as the interpreters in today’s community. The conceptual recovery will explore how signs and symbols are determined by possible meanings, and further, clusters of meanings within a genre. By presenting a simultaneously historical, contextual and conceptual perspective of surrealist art, the fluid, ongoing nature of sign formation can be more effectively understood. From here, we can see how signs gain symbolic significance when examining specific surrealist works as a whole. It is the combination, or marriage, of elements within a painting that allows us to enact meaning.

Main body of paper

In its simplest terms, the surrealist art movement was just that: a movement away from traditional bourgeois styles of art toward a newly subversive genre that “probed the structure of consciousness itself” (Hopkins 20). The movement was officially born in 1924 and grew to become a global phenomenon before its demise in the 1940s (Hopkins 15). In that time, artists such as a Salvador Dalí, Joan Miró, and André Masson and René Magritte used various approaches to transcribe “dreams” using “considerable conscious deliberation” (Hopkins 37). Their motivation to create realistically unrealistic paintings stemmed a belief that modern art could “forge a new relationship with its audience,” (Hopkins 21). This relationship changed the social experience viewers had when interpreting surrealist works through a “two-pronged attack on bourgeois social conventions and the aestheticism of an earlier Modernism” (Smith and Wilde 405).

In his work Dada and Surrealism: A Very Short Introduction, Hopkins articulates “surrealist artists such as Dalí or Magritte are frequently denigrated as ‘literary’” (Hopkins 82). Although a less currently relevant critique, Hopkins aligns their work with poetry. The word poetic suggests that these artists had the ability to appreciate complex emotional or aesthetic themes. Coupled with their sensibility, the word literary brings me to my unpacking of The Human Condition and Melting Watch in terms of Peirce and Jackendoff.

“Everything we see hides another thing, we always want to see what is hidden by what we see.” -René Magritte

With Peirce’s model as a foundation, we can see dense clusters of tokens and types within The Human Condition that impact our understanding of the relationship between normal and extraordinary.

Source: http://www.nga.gov/content/ngaweb/Collection/art-object-page.70170.html

We begin by recognizing the painting as a cultural artefact of the surrealist movement, which allows us to recognize and extract the “Magritte” features and establish multiple chains or levels of Interpretants (Irvine, “A Student’s Annotable Peirce” 6). By conceptually labeling the painting as a piece of this larger genre, we look at it through the surrealist lens. In this way, the painting itself is a “complex Interpretant of many other paintings and types and styles of painting” (Irvine, “A Student’s Annotable Peirce” 9). This is what Peirce would refer to as an “immediate interpretant,” and is thus, the first layer of meaning (Irvine, “A Student’s Annotable Peirce” 6).

This interpretation begins we see this painting in its entirety, as one token, or instance of a type. When we break down the elements within the painting, we see a window with a view of a tree and forest area framed by curtains. These icons, in Peirce’s terms, are: the window itself, curtains, tree, tree line/forest, dirt path, and the blue sky and clouds (indicating that the scene looks from the perspective of inside to outside).

   

          

These aspects of the painting are defined as icons because they are not necessarily symbolic. They are images that resemble or imitate an object. Our ability to recognize these icons connects to our pre-established meaning environment. As Dr. Irvine explains, much of what exists around thought and meaning is culturally implied. We reference our “encyclopedia” of “conceptual/symbolic cultural shared” knowledge (Irvine, “Semiotics, Symbolic Cognition and, Technology” 32). Through this action, we imagine a window, window dressings and a landscape as we have been taught to recognize them. Thus, these icons are representamen in Peirce’s terms.

However, once the eye moves to the three legs of the easel, our interpretation is complicated.

The view outside the window is interrupted by a few uncharacteristic icons: The clip at the top of the easel interrupts the image of the sky.

The sides of the easel break the seamlessness of the landscape as well as the line of the curtain.

                 

Therefore, what we first identify as icons is problematized by the fact that they stretch onto multiple plains (the painting itself and the painting within the painting). Viewers must understand how the icons are simultaneously as they appear, and not as they appear. For example, what looks like one, single dirt path that extends across the image is actually two separate renderings. This subtle shift from a seemingly normal landscape to something entirely different reflects two themes associated with Magritte: repetition and “masquerading as ordinary” (Magritte et al. 20). This landscape appears to resemble a common, familiar outdoor scene, as does the painting on the easel. However, that the painting does not necessarily reflect what the viewer cannot see. Therefore, the themes of repetition and replication are complicated, since viewers cannot actually determine what exists behind the painting on the easel. We cannot determine the painting on the easel as an accurate representation and it cannot fill in the gap of what we, as viewers, cannot see when looking out the window. In his works, Magritte used “normalness” to invoke memorability (Magritte et al. 20). Once these elements outlined above are identified as dissonant, and in this way not ordinary, we can begin to dissect the painting as something other than a landscape through a window.

The Human Condition becomes symbolic when we see that there is a painting within a painting. The painting itself represents a cultural artefact in the surrealist genre, and it functions as a singular piece of art completed by Magritte. Within this image is another painting. We know it is a painting because of the way it is depicted on the easel and canvas. Viewer expectations are subverted due to the realistic representation of the painting and the assumption that the painting reflects the view that is blocked by the canvas. In this way, reality is questioned because viewer assumption is based on a false premise. The icons situated in Magritte’s work are “real” if the painting on the easel iconically represents that reality. In keeping with Magritte’s stylistic features, “His paintings from this period are invitations to look closely and to pay careful, critical attention to what is seen, both in each image and, when turned away from them, in the surrounding world (Magritte et al. 21). Thus, the viewer sees one depiction as real and the other as a representation. The painting itself appears as real, while the painting within it appears as a replication.

This view, however, is not possible when looking at the surrealist work as a whole. Both the painting and the painting inside are a part of the same work, suggesting that all of the elements work in parallel to contribute to a unified sense of meaning of the whole image. In his work Foundations of Language, Ray Jackendoff articulates the “multiple parallel sources of combinatoriality, each of which creates its own characteristic type of structure” (Jackendoff 107). His explanation centers around the properties of language (phonology, syntax, and semantics), but if we extrapolate his understanding of the combinatorial power of one symbolic-cognitive system to the interface of surrealist art, we can see how separate elements of the painting come together to form one whole image, or instance of surrealist art. When discussing the linguistic structure, Jackendoff explains it as a “collection of independent but linked levels of structure” (Jackendoff 131). This explanation connects to Peirce, who articulated that sign/ symbol structures are stacks of features developing in parallel as a result of logical sequences (Irvine, Presentation for discussion: “Cognition, Symbols, Meaning” Slide 53).

In this way, the icons represented in The Human Condition, link together to form multiple combinations of meaning. These icons establish the connection between certain syntactic constituents with conceptual structures (Jackendoff 131). For example, the painting on the easel is only identified as a replication of a landscape because of the way it is portrayed. It is sitting on an easel, painted on a canvas, inviting viewers to think that the painter used the portion of the scene that we cannot explicitly see as his or her subject matter. All of the separate pieces of the image come together simultaneously when viewers make the connection that the portion of the window scene that they cannot see is not necessarily what is represented in the painting on the easel. Thus, we can understand the work as a meta-painting. All at once, The Human Condition becomes one whole image as well as a series of separate renderings.

“Those who do not want to imitate anything, produce nothing.” -Salvador Dalí

Much like Magritte’s work challenges what is ordinary, Salvador Dalí’s Melting Watch challenges how we conceive time and reality. He does this by presenting the concept of time in a way that is unfamiliar to us. According to Hopkins in Dada and Surrealism: A Very Short Introduction, Dali’s paintings discredited reality (Hopkins 100). Melting Watch discredits reality by showing how time is warped in a dream setting. This type of work falls into the surrealist genre, and it should be noted that this conceptual label is an immediate interpretant. By assigning this label, we can begin to understand how the components of the painting impact the syntactical and conceptual interfaces of surrealist art.

Source: http://www.dalipaintings.com/melting-watch.jsp

In keeping with the surrealist genre, Dalí uses the distorted watch to symbolize how time passes while we are dreaming (MoMA). In this way, the watch acts to subvert our understanding of time. To come to this conclusion, though, we have to first identify that the object in the picture as a clock, then we have to analyze how it is being represented differently than a normal clock, so that we can understand that Dalí is actively portraying the construct of time as arbitrary and useless.

Therefore, while we do recognize the object as a watch, we have to do so by acknowledging it is less watch-like. This process, when described by Dr. Irvine, involves “Meanings, intentions, or values [which] are not properties of — or “baked into” — any substrate or medium used to produce perceptible, interpretable forms” (Irvine, “Introduction to Technical Theory of Information” 5). Meaning does not come from the medium of art, but it is enacted by viewers who use their collectively understood material sign structures in the context in which they reside to interpret and understand what they see.

We can label the watch as an “icon” because it “resembles or imitates” an object, but there are characteristics that do not match up with our mental image, creating a tension between what we see and our previously established reference to a clock (Irvine, “Semiotics, Symbolic Cognition, and Technology” 31). We see the object is circular, has two hands to denote the hours and minutes, and has numbers spaced out evenly. However, its distorted, exploding state suggests it is more of a “hypoicon,” specifically a metaphor, which denotes the representative character of a sign by representing a parallelism in something else (“Semiotic Elements and Classes of Signs” 7). In this case, the shape informs us that the watch symbolizes something other than time as we understand it. It does not represent it as structured and powerful.

Other icons include the moth, fly, water, numbers on the clock and numbers separated from the clock and the mountains in the lower right hand corner of the image.

        

In connection with Peirce’s theory of semiotics, signs and symbols are replicable expressions that are only recognized to have meaning because they are understood (Irvine, “Introduction to the Technical Theory of Information” 6). Viewers understand the separate pieces of this painting as replicable objects that they have seen, or experienced before, which enables them to identify the objects. This is the first layer of meaning.

Once these objects are examined in the context of Melting Watch, their status becomes both iconic and symbolic. To further interpret the theme of time, we can look at the icons as they relate to the clock. To demonstrate this process, I will look specifically at the moth and the number five. The presence of a moth in close proximity to the clock could indicate that it represents something other than just an insect.

This leads to the question: is the moth connected to the watch? Is it connected to how we conceptualize time? This realization bridges the gap between iconic and symbolic significance. Moths are a nocturnal species, which indicate nighttime and connect to a manipulation of time during a dream. They are also attracted to light. The shadows in the image suggest that there is some source of light. There are many connotations associated with moths, which open up a wide range of possible meanings depending on how viewers combine aspects of the image. Moths can signify vulnerability, death, faith, transformation and sensibility (Leith 56). These possibilities alone offer multiple interpretations of Dalí’s work and relate back to the specifically “Dalí” features of this surrealist work, which centered on “dream painting” and subversion of reality (Hopkins 40).

Similarly, the number five, which is separated from the clock as an object, questions what is real with regard to time.

Frozen in time, it no longer marks the hours or minutes, but exists as an object in opposition to the structure of time. To interpret the number in this way, viewers have to acknowledge the way we tell time and the role that the numbers on a watch play in framing our conception of time. To connect this discovery to Peirce’s explanation of semiosis, the number five is first identified as an icon because it resembles an object, then as an index in how it is used in relation to keeping time, then as a symbol in how it is positioned away from the clock, suggesting its disassociation with the traditional understanding of time.

The process of sign formation is fluid, though. That is, to understand why the number five is separated from the watch is to understand its relation to clocks, time and reality. Jackendoff explores this idea when he says “The parallel constraint-based architecture is logically non-directional: one can start with any piece of structure in any component and pass along logical pathways provided by the constraints to construct a coherent larger structure around it” (Jackendoff 198). The sign systems work concurrently in a parallel architecture in which all of the layers are engaged simultaneously in the sign processes of any one of images individually. Further, as Dr. Irvine writes, “creating combined symbolic expressions that require engaging two or more sign systems in parallel…” involves “visual representations with many “vocabularies” of image-making…” (Irvine, “Introduction to Linguistics and Symbolic Systems” 16). In other words, when viewers look upon Melting Watch, they are drawing upon collective memory to engage in a contextual and conceptual way, creating a parallel between symbol systems.

Conclusion

 Surrealism as an artistic movement challenged how viewers interact with and understand subject matter using various sign and symbol systems through feature extraction. Surrealist art provided a new opportunity for viewers to use their imaginations to actively subvert what they believed they knew about art, culture and most generally, about the themes of time and space. By reinforcing this narrative through feature extraction, we build upon the culturally constructed meaning system.

These meaning systems are part of complex and deep ideologies and value systems that we activate in order to produce interpretant expressions for the meaning, in this case, of two surrealist paintings (Irvine, “A Student’s Annotable Peirce” 8). Surrealist paintings can act as an interface to a larger system of meanings and values. From here, feature extraction is used to identify the perceptible features that appear in the images, in the way they are presented (e.g., in a museum) and the way they are described. These features are situated in the context of historical and conceptual interpretation associated with the paintings. Both The Human Condition and Melting Watch are a part of a cultural meaning system framed by institutional contexts.

It is important to note that meaning comes when we, as cognitive agents, enact it. This process happens on a few levels and can be understood by bringing together Peirce’s semiotic model and Jackendoff’s Parallel Architecture. The layers established within and around each surrealist painting are pieces of a greater network of sign systems. The sign systems work concurrently in a parallel architecture in which all of the layers are engaged simultaneously in the sign processes of any one of images individually. This is because the layers of abstraction and representation are in an evolutionary ecosystem of sign systems. As previously noted, the surrealist movement of art declined in the 1940s. Viewer understanding of The Human Condition and Melting Watch is impacted by the current community of interpretation in which they reside. These communities offer spaces for discovery of the meaning processes behind surrealist art.

Further, it should be noted that while this paper concentrated on feature extraction in a historical and conceptual context, it did not focus on the construction of meaning within the physical space at a museum. There are opportunities for further research when exploring museums as systems of categorization, memorialization and classification.

Even still, these surrealist works serve as examples of how dreamlike features prevent viewers from rationally understanding what a picture represents. To interpret the dense clusters of signs and symbols within these paintings, it is important to acknowledge the combinatorial power of the elements as well as the fluid, ongoing nature of interpretation and sign formation. Especially with regard to surrealist, and therefore subversive, imagery, understanding how each sign works both separately and in conjunction with the rest of the image is crucial to de-black boxing the function of illusionistic representational paintings. Combinatorial power is relevant in how it reveals the significance of realistically depicted, yet simultaneously dreamlike renderings presented by surrealist artists.

Works Consulted

Ades, D., & Vancouver Art Gallery. The colour of my dreams: The Surrealist revolution in art. Vancouver: Vancouver Art Gallery. 2011. Print.

Breton, André. Manifestos of Surrealism. Trans. Richard Seaver and Helen R. Lane. N.p.: U Michigan P., 1969. Print.

Clark, Andy and Chalmers, David. “The Extended Mind.” Analysis 58, no. 1 (January 1, 1998): 7–19.

Clark, Andy, Supersizing the Mind: Embodiment, Action, and Cognitive Extension (New York, NY: Oxford University Press, USA, 2008).

Fanés, F. (2007). Salvador Dalí: The construction of the image, 1925-1930. New Haven: Yale University Press.

Hopkins, David. Dada and Surrealism: A Very Short Introduction. Oxford; New York: Oxford University Press, 2004.

Irvine, Martin, “A Student’s Annotable Peirce: Primary Texts On Signs and Symbolic Thought With Transcriptions of Unpublished Papers from Peirce’s Manuscripts”

Irvine, Martin, Presentation for discussion (conclusion): “Cognition, Symbols, Meaning”

Irvine, Martin, “Introduction to Linguistics and Symbolic Systems: Key Concepts” (intro essay).

Irvine, Martin, “Introduction to the Technical Theory of Information” (and using Information Theory + Semiotics)

Irvine, Martin, Selections from: Semiotics, Symbolic Cognition, and Technology: A Reader of Key Texts

Irvine, Martin, The Grammar of Meaning Making: Sign Systems, Symbolic Cognition, and Semiotics. Communication, Culture & Technology Program, Georgetown University.

Jackendoff, Ray, Foundations of Language: Brain, Meaning, Grammar, Evolution. New York, NY: Oxford University Press, USA, 2003.

Leith, James A., and George Whalley. Symbols in Life and Art. Kingston, Ont.: Published for the Royal Society of Canada by McGill-Queen’s UP, 1987. Print.

Luhmann, Niklas. Art as a Social System. Trans. Eva M. Knodt. Stanford, CA: Stanford UP, 2000. Print.

Magritte, R., In Umland, A., D’Alessandro, S., Museum of Modern Art (New York, N.Y.), Menil Collection (Houston, Tex.), & Art Institute of Chicago. (2013). Magritte: The mystery of the ordinary, 1926-1938. New York: Museum of Modern Art.

“MoMA Learning.” MoMA. The Museum of Modern Art, n.d. Web. 14 Dec. 2016.

Pinker, Steven. Big Think. Steven Pinker: Linguistics as a Window to Understanding the Brain. Accessed September 19, 2016. https://www.youtube.com/watch?v=Q-B_ONJIEcE.

“Semiotic Elements and Classes of Signs.” Wikipedia. Wikimedia Foundation, n.d. Web. 13 Dec. 2016.

Shanes, Eric, Salvador Dalí, Inc Ebrary, and Proquest (Firm). The Life and Masterworks of Salvador Dalí. Vol. Rev. and Updat 2nd;1. Aufl.;Rev. and Updat 2nd;2nd;. New York: Parkstone International, 2010;2012;. Web. 5 Dec. 2016.

Smith, Paul, and Carolyn Wilde, eds. A Companion to Art Theory. Oxford: Blackwell, 2002. Print.

“The Human Condition, 1933 by Rene Magritte.” The Human Condition, 1933 by Rene Magritte. ReneMagritte.org, 2009. Web. 12 Dec. 2016.

Zaslavskii, O. B. (2005). Language as an underlying idea in Salvador Dali’s works. Word & Image, 21(1), 90-102.

Value and the Meaning System

Abstract

This paper aims to demonstrate the indexical nature of interface through a case study on the dual-function of Library of Congress (LOC), it is simultaneously an museum and a library. The paper illustrates the differences between an artifact and an interface and their relation to value. The main goal of this paper is to understand the significance of reproduction, especially digitalization, in preserving an artifact’s interface to its meaning system, and, therefore, protecting the integrity of the collective human experience, or in other words, universal cultural memory.

 

One of the biggest complaints people often bring up these days is that people do not read anymore because technological advancement has taken away people’s motivation for reading. This statement is, however, not factual. The Pew Research Center conducted a survey in 2014, their findings suggest that Millennials are reading more books than the over-30 crowd.[1] Why? It turns out that the Internet has helped to cultivate a reading habit among Millennials through the widening access to information.

But can digital books replace physical books? Personally, I enjoy the textual of a physical book and somehow find a digital book less valuable and less personal. There are many reasons for it, such as I can doodle on a physical book, not have to worry about charging my tablet, and, most importantly, the tangible quality of a physical book makes my copy unique and valuable.

This train of logic leads to the thesis of my paper. What makes something valuable? The most obvious answer is scarcity. This is why rare metals are expensive. But what about culture? How do you determine the value of a cultural product? Is it determined by how well it is made or how many copies of it still exist in the world?

To answer this, we must contemplate the significance of reproduction of artifacts. Morris Eaves, who has studied William Blake extensively, states that an interdisciplinary approach from fine art and technology engenders a new perspective on the relationship between Blake’s term “machine” and art. He maintains that,

The perfect system of reproduction would reproduce anything perfectly. Because there is no perfect system, the reproductions will always be imperfect. The technological gap between the original and the reproduction is what interests us, because we can safely guess that the difference that occurs in the gap is significant and affects the artists’ originals not only now but also in the future and affects as well the audience, its understanding of the work, judgment of the work, artistic expectations, and so on.[2]

The gap Eaves mentioned here is also what Walter Benjamin laments in his response to the rise of photography. Benjamin goes a step further warning the ramification of digital reproduction,

The situations into which the product of mechanical reproduction can be brought may not touch the actual work of art, yet the quality of its presence is always depreciated. This holds not only for the art work but also, for instance, for a landscape which passes in review before the spectator in a movie. In the case of the art object, a most sensitive nucleus – namely, its authenticity – is interfered with whereas no natural object is vulnerable on that score. The authenticity of a thing is the essence of all that is transmissible from its beginning, ranging from its substantive duration to its testimony to the history which it has experienced. Since the historical testimony rests on the authenticity, the former, too, is jeopardized by reproduction when substantive duration ceases to matter. And what is really jeopardized when the historical testimony is affected is the authority of the object.[3]

Walter Benjamin maintains that the loss in authenticity in art caused by reproduction will eventually jeopardize the original artifact’s value. A valid argument indeed, yet it lacks certain productivity. Lack of productivity in terms of Benjamin never offers a feasible solution to the issue he sees. Artifacts are mediums of universal cultural memory. They help us understand the collective human experience. If an artifact’s interface is forever lost, the meaning system that it associates with will be lost as well. André Malraux offers different take on this issue, he claims that reproduction, though the quality of such is not as favored as the original, can grant access to those who would not have such access otherwise. Putting together a collection of art works is like building a “museum without wall”[4]. Reproduction takes arts out from their esoteric environment and do so ensures their presence in universal cultural memory. Malraux maintains that,

Nowadays an art student can examine color reproductions of most of the world’s great paintings, can make acquaintance with a host of second-rank pictures, archaic arts, Indian, Chinese and Pre-Columbian sculpture of the best periods, Romanesque frescoes, Negro and ‘folk’ art, a fair quantity of Byzantine art… Hitherto the connoisseur duly visited the Louvre and some subsidiary galleries, and memorized what he saw, as best he could. We, however, have far more great works available to refresh our memories than those which even the greatest of museums could bring together. For a “Museum Without walls” (musée imaginaire) is coming into being, and (now that the plastic arts have produced their printing press) it will carry infinitely farther that revelation of the world of art, limited perforce, which the “real” museums offer us within our walls. [5]

This is a video about how Library of Congress (LOC) preserve and extrapolate infarction from old and damaged artifacts using technology.

The speaker in video, Sarah Werner, maintains that digitization ensures the integrity of cultural memory.[6] The work Werner presents validates Malraux’s claims, digital reproduction, though will always be getting better, preserves artifact and its interface to its meaning system.

So back to my thesis. We are facing a dilemma between damage to the value of original artworks due to reproduction and the benefit of valuable access gifted by reproduction. So how do we determine value in any meaning system?

In order to answer this, my paper will divide into three sections;

1. Index’s relationship to value
   • Why index cannot retain value
2. Artifact and Interface
   • Artifact retains value
   • Interface as index
   • Interface’s relationship to value
3. The necessity for digital reproduction for it is a means to protect the value of universal cultural memory

Index and Value

Listed below are two indices of books. Index A is an opinion piece on “what are the most influential books ever?” This list is compiled by “an expert panel of academic book-sellers, librarians, and publishers, and members of the public were asked to vote online for their top 20.”[7] Index B is where you can locate the most influential book, On the Origin of Species by Charles Darwin, using Library of Congress Classification (LCC), which is adopted by the majority of research and academic libraries in the U.S.

Index A ranks these books according to their assigned value whereas index B maps out the location of the number one book in the LCC system. At first glance, two indices appear quite different. The main difference is in purpose. Yet from a semiotic perspective, these two serve the exact same function according to Charles Sanders Peirce’s definition of an index, “a guidepost, which points down the road to be taken, or a relative pronoun, which is placed just after the name of the thing intended to be denoted, or a vocative exclamation, as ‘Hi! there,’ which acts upon the nerves of the person addressed and forces his attention.”[8] The two indices have no physical property of their own, only serve as pointers to unveil the underlying meaning systems. In other words, these two indices are signifiers that signify different connotations, and the difference between the two only lies within what they signify, the signified.

Pierce’s semiotic theory is undoubtedly more productive when considering the relationship between the signifier and the signified, yet Ferdinand de Saussure’s idea on the arbitrary relationship between the two is helpful when we consider the index and the underlying meaning system it indicates. It is true that index A seems more subjective comparing to index B and, therefore, it is more arbitrary. In some way, that is true. Yet I will cite Martin Irvine’s definition of arbitrary so to proceed my argument,

The term (arbitrary) should not imply that the choice of the signifier is left entirely to the speaker (… individual does not have the power to change a sign in any way once it has become established in the linguistic community); I mean that it is unmotivated, i.e. arbitrary in that it actually has no natural connection with the signified.[9]

This argument remains valid outside linguistic as well. Now going back to the two indices, it is clear that relationship between index A and its meaning system is just as arbitrary as that of index B. Though Index A aims for objectivity since it is compiled through a variety of measures conducted by both professionals and the general public to ensure the accuracy of the ranking, subjectivity is still the fundamental constituent. Nevertheless, the indexical ranking is still unmotivated in the linguistic sense mentioned above, simply a reflection of the outcome.

Index B is also arbitrary even though it is more “objective.” This image shows how LCC classifies different books,

Yes, index B is arguably more objective in the way that it treats any given book equally, categorically assigning a random value to one type of book, but the process itself is arbitrary as there is no natural relation between letter and the category.

Value, in these instances, is either a qualitative or quantitative manifestation of a “subjective” or “objective” evaluation. In short, subjectivity and objectivity have nothing to do with the arbitrary nature of signs.

Furthermore, as stated earlier that since indices do not have the physical property to contain value, therefore they do not have value yet retain the ability to signify such.

Artifact and Interface

Then again, what is the difference between LLC and my personal way of categorizing book? The answer is standardization via institution. Institution justifies an arbitrary instance of signifier/signified correlation into an established standard. Bruno Latour’s idea will be very helpful for examining the case here. We delegate the task of categorizing books to LOC, and through standardization, LOC conditions us to understand its classification while freeing us from the labor of using different systems in different places, hence forming a complete circle of meaning system[10].

This, however, is not the only job LOC has. On their official website, one can easily spot the two major functions (ignore the ask a librarian function here):

Beside setting standards for book cataloging, LOC is the institution to organize and protect books and doing so, ensuring the integrity of the collective human experience we call the cultural memory[11]. The duality of its function here presents an interesting dialectical dichotomy, LOC as an artifact and LOC as an interface. But before I delve into why it is significant for me to dissociate artifact from interface, I shall first define these two terms in the context of my writing and second admittedly state that these two concepts can be considered as a single instance.

So what is an artifact? Irvine eloquently explains that, an artifact is “something humanly constructed by design.”[12]

Artifacts are the fundamental makeup of the collective human experience, or universal cultural memory. Books, arts, and music are all instances of human artifacts and more importantly vehicles for cultural memory. Furthermore, institutions, like museum, are also instances of artifact, or physical symbol systems[13]. Regardless what the specific type of instance it is, any of these examples shows to possess a physical quality, or in other words, tangible. But artifacts are not just physical manifestation, according to Michael Cole summarized by Irvine, “Artifacts are simultaneously ideal and material. They coordinate human beings with the world and one another in a way that combines the properties of tools and symbols.”[14] [15] The physical quality of an artifact constitutes the “material” description, whereas the concept of interface provides structure for it.

Herbert A. Simon maintains that,

An artifact can be thought of as a meeting point–an “interface” in today’s terms–between an “inner” environment, the substance and organization of the artifact itself, and an ”outer” environment, the surroundings in which it operates. If the inner environment is appropriate to the outer environment, or vice versa, the artifact will serve its intended purpose[16].

Accordingly, there is no a definite division between an interface and an artifact as the former is the ideal manifestation of the latter. However, I argue it is productive to separate the two when considering where value is distributed. For instance, this is a blog documented by a teacher about her students’ field trip. When they visited the LOC, she posted a picture of the Gutenberg Bible and wrote,

[17]

A simple analysis here reveals that there are several layers of meaning here.

1. She values it for its technological significance in the printing history as students were taught in school.
2. She took the photo because she wanted to document the filed trip.
3. She understands the historical, spiritual, and religious significance of The Gutenberg Bible as it is being featured in the LOC.

So here is the question, she did not take photo of the Gutenberg Bible because she wants to read it. Gutenberg Bible as the interface is missing in her photo. But to the teacher, there are still enough reasons for her to document her and her students’ close proximity to this artifact. It is the same reason that people want to take selfies with famous paintings in museums, they attribute value to the artifact for its material property not for its interface.

Researcher who are interested in the artifact’s interface might find value in what artifact’s interface mediates. But even in this case, value does not come from the interface, but what the interface indicates.

Case Study

This comparison reveals dual functions of LOC, it is simultaneously a museum and a library.

In order to further explore this idea, I decide to evaluate the distribution of value using LOC as an example and its dual functions as a museum and library will explains this perfectly. Instead of using words, I shall employ annotated system diagrams below to illustrates my thought processes.

If how we perceive value derives from various meaning systems, then the indexical nature of interface would indicate its ability to call upon value and it inability to retain such value. Interfaces are, after all, instances of extended cognition, which takes away physical constraints and maximize our cognitive process.

I will cite Irvine here again to illustrate this point,

So, “meanings” are never “in” any artefact as a physical or material thing; meanings and values emerge in the correspondences in a cultural meaning system that uses clusters of signs in recognizable patterns.[18]

Meaning and value are closely associated with each other. If the interface is jeopardized, the artifact’s meaning system will be lost forever. Through preservation and digital reproduction, the interface of such is protected, hence ensuring the access to the artifact’s meaning system. Benjamin, Malraux, and others’ concerns about technology are valid for the original is eternal better yet Malraux’s notion of musée imaginaire ensures the value of the cultural memory. This video here shows not only books but other instances of artworks can be preserved through digital reproduction. It is true that the value of the original will be decreased once they have been reproduced, yet technology ensures the integrity of its intended purpose, which is invaluable.

[1] Zickuhr, Kathryn, and Lee Rainie. “Younger Americans and Public Libraries.” Pew Research Center: Internet, Science & Tech, September 10, 2014. http://www.pewinternet.org/2014/09/10/younger-americans-and-public-libraries/.

[2] Eaves, Morris. “Blake and the Artistic Machine: An Essay in Decorum and Technology.” PMLA 92, no. 5 (1977): 903–27. doi:10.2307/461845.

[3] Walter Benjamin, The Work of Art in the Age of Its Technological Reproducibility, and Other Writings on Media. Edited by Michael W. Jennings and et al. Cambridge, MA: Harvard University Press, 2008.

[4] Malraux, André. The Voices of Silence. French, Les Voix du Silence, 1951. Translated by Stuart Gilbert. Garden City, NY; repr. Princeton: Doubleday; Princeton Univ. Press, 1953.

[5] Malraux. The Voices of Silence.

[6] CBS Sunday Morning. America’s Film Heritage Preserved at the Library of Congress. Accessed December 10, 2016. https://www.youtube.com/watch?v=SEhMxp2os3M.

[7] Thomson, Stéphanie. “The 20 Most Influential Books in History.” World Economic Forum, November 15, 2015. http://weforum.org/agenda/2015/11/the-20-most-influential-books-in-history/.

[8] Martin Irvine, “The Grammar of Meaning Making: Signs, Symbolic Cognition, and Semiotics.”

[9] Martin Irvine, “Key Writings on Signs, Symbols, Symbolic Cognition, Cognitive Artefacts, and Technology.”

[10] Latour, B. [as Jim Johnson] (1988). “Mixing Humans and Nonhumans Together: The Sociology of a Door-Closer” Social Problems, Vol. 35, No. 3, Special Issue: The Sociology of Science and Technology. (Jun., 1988), pp. 298-310.

[11] Dietz, S., Besser, H., Borda, A. and Lévy, P. (2004) Virtual Museums (of Canada): The Next Genera on, Canadian Heritage Information Network. Available from: <h p://besser.tsoa.nyu.edu/howard/Papers/vm_tng.doc.> [Accessed 9 March 2015].

[12] Martin Irvine, “Working with Semiotic Concepts and Methods: From Peirce to Computer Interfaces.”

[13] Herbert A. Simon, “The Sciences of the Artificial.” Cambridge, MA: MIT Press, 1996.

[14] Irvine, “Key Writings.”

[15] Michael Cole, “Cultural Psychology: A Once and Future Discipline.” 1996.

[16] Irvine, “Key Writings.”

[17] Finnerty, Valerie. “2016 LMS Washington, DC Trip: Day 4!” 2016 LMS Washington, DC Trip, May 26, 2016. http://2016lmsdctrip.blogspot.com/2016/05/day-4.html.

[18] Irvine, “Key Writings.”

Abstract

Music is the art of sound. It has been along with humans for a very long time and conveys rich meanings among people and through time. Music is considered as a “universal language of mankind” because of its affective power across the boundaries of languages. The power of music roots in its symbolic systems. This paper discusses the semiotics of music in the theoretical framework developed by C. S. Peirce. Then, I examine the similarities and differences between language and music. It is evident that music and language share the same brain area. However, they differ a lot, especially in formal structure. I also analyze how music serves as a cognitive artifact that is distributed among members of social groups. Next, I briefly review the history of music digitalization and see great potential in computer-generated music. I explore the parallel between humans and computers ways of pattern recognition in music and consider this comparison as a useful direction for future studies both in human music cognition and computer science.

1. Introduction

Music is an art form that uses sound as a medium to transfer meanings. We all have our experience about music. Why does music have so much power? The reason lies in the symbolic systems of music. C. S. Peirce, the co-founder of semiotics provided us with theoretical tools to analyze signs, also useful for examining music. In addition, as the most researched branch in semiotics, linguistics has the potential to provide music with useful methods and frameworks; though there are many differences between music and language. Ray Jackendoff and others wrote detailed papers and books in which they looked into the parallels and nonparallels between music and language from an interdisciplinary perspective. This paper also discusses how digitalization impacts the music industry and the great potential in computer composition.

2. Music as a symbolic system

Music has been along with us since the very dawn of human civilization. The oldest musical instrument so far is thought to be a bone flute forty thousand years’ old[1]. Its function remains unknown but probably for nothing more than religious, military, or entertainment purposes.

We all listen to some kinds of music in our lives. Some people listen to music as background when they are walking or driving, while others are immersed in music in concerts or clubs. In addition, music is ubiquitous. Churches, restaurants, yoga clubs, shopping malls, and other places all play different music to build different atmospheres. The secret lies in the rich meanings of music that can be understood by humans. As a symbolic species, we humans can give meaning to and take meaning from all perceptible signals, be it visual, olfactory, tactile or acoustic. Music is an art of acoustics, definitely a symbolic system.

Emotional responses can be evoked by some music, while others tell you stories. Some music could even provide you with religious ecstasy. These meanings are all conveyed through musical signs.

2.1 Peircean Signs in music

C.S. Peirce was known as a co-founder of semiotics, and he developed many valuable concepts and methods in order to study signs. As he put it, “we think only in signs.” Everything could be a sign as long as it is interpreted by someone[2]. Peirce’s theories of semiotics are useful for analyzing all kind of symbolic systems, music included.

Peirce’s model of signs consists of three components—representamen, object, and interpretant.

• The representamen, also called the “sign” or “sign vehicle” by some scholars, is the form of the sign. In music, the representamen can be many things—the music itself, a movement, a melody, a beat, a genre, a music score, a performance, a recording, a sound effect, the environment of the listener, the stage design, the clothes the performers are wearing, or even a mistake. As long as it is interpreted as something else rather than itself, it could be seen as a musical sign. For example, American ethnomusicologist Thomas Turino once analyzed how musical meanings were built in Jimi Hendrix’s Woodstock performance of “The Star-Spangled Banner.” Many things were considered by him as signs, including Hendrix’s wearing of a seemingly contradictory tuxedo and tennis sneakers, his use of a loud electric guitar with “feedback and distortion,” and the sound effects of airplanes and siren[4]. These signs were interpreted in a meaning-rich context—a specific concert (Woodstock) and a certain historical period with a unique international situation and ideological trend, which had a huge impact on people’s understanding of Hendrix’s music.
• An object is something to that the sign refers, mainly in the form of an abstract concept. For example, a song called “A Morning of the Slag Ravine” by Joe Hisaishi from Hayao Miyazaki’s film Laputa: Castle in the Sky is a sign for the morning, therefore it’s object is the abstract concept of “morning.” People who are familiar with Hayao Miyazaki’s films would think of the morning sun emerging from the eastern horizon when they hear the song even without visual cues.

A Morning of the Slag Ravine by Joe Hisaishi

• An interpretant is a sense in the observer’s mind where the representamen and object are brought together.

In addition, Peirce developed three modes of signs, which represent three kinds of relationship between signifiers and signified[1]:

• Icon is a mode for resemblance, such as a portrait. For example, at the beginning of Garth Brooks’s The Thunder Rolls, there is a sound effect imitating the sound of thunder, which is an iconic sign.

Garth Brooks – The Thunder Rolls

• Index is a mode that is “mediated by some physical or temporal connection between sign and objects[3].” For example, ripples on the surface of the water are indexical to the wind. An example in music is the famous beginning of Beethoven’s Symphony No.5 in C minor. It sounds like someone is knocking at the door, but it does not completely imitate the knocking sound, as the beginning of The Thunder Rolls imitating thunder sound. The sound of door knocking is an index for someone at the door. The beginning of Symphony No.5 indicates fate knocking at the door.

• Symbol is a mode in which the sign and object are connected by social convention, such as language and number. In this mode, the relationship between sign and object is arbitrary and must be “agreed upon and learned[2].” According to Thomas Turino, unlike language, most musical signs function as icons and indices, but there also exists many musical symbols[4]. For instance, in his book Signs of Music: A Guide to Musical Semiotics, Finnish musicologist, and semiologist Eero Tarasti gave an example of J. S. Bach’s Fugue in C sharp minor from Book I of the Well-Tempered Clavier, as shown below. For listeners of the Baroque period, this subject is a symbol of “cross and thus the Christ,” which got quoted a lot in other music with the same symbolic meaning[5]. The relationship between this melody and its sign was built based on historically religious convention. Most people in the twenty-first century no longer recognize it as a symbol for Christ. Another example is the beginning of Beethoven’s Symphony No.5 I mentioned before. It became so famous that it was adopted and quoted a lot in other genres of music as a symbol for victory, partially because of its analogue to the Morse code for the letter V—“dit-dit-dit-dah” (another symbolic sign). During the World War II, BBC even used these four notes as the beginning of its broadcasting programs[6].

Musicians themselves could become symbols, too. A perfect example is Sixto Rodriguez, the American singer depicted in the 85^th Academy Award Winner film Searching for Sugar Man. He remained unknown in his home country but had earned significant fame in Australia, Botswana, New Zealand, Zimbabwe, and especially South Africa, where he became a symbol for anti-Apartheid activities and influenced many musicians protesting against the government[7]. Another example is Jian Cui, the first and the most famous rock star in the mainland of China. He was called the Godfather of Chinese rock and roll. His songs and performances teem with music signs. For instance, he was well-known for covering his eyes with a piece of red cloth in his performances. And he also was very good at combining western rock music with Chinese traditional instruments like suona horn, building an unexpected conflicting but subtly harmonious atmosphere. These behaviors were interpreted as signs of a rebel against traditional values and political realities. “I covered my eyes with a red cloth to symbolize my feelings,” he said in an article for Time in 1999[8]. He was considered in China as a symbol of rock spirit and freedom.

Another noticeable symbolic system in music is the musical marks and symbols we use to notate music in scores. There are a lot of musical notations based on historical and cultural conventions that should be learned in order to understand. The most widely used method today is five-line staff that originated from Europe. However, in ancient China, people used a totally different music notation called “Gongchepu” (工尺谱). Unlike five-line staff that mostly uses non-word characters, Gongchepu uses Chinese characters and punctuations to notate music, and it was written from top to bottom and then from right to left, just like ancient Chinese writings on bamboo slips, another demonstration for the arbitrariness of symbols.

However, as Daniel Chandler pointed out in his Semiotics: The Basics, the three Peircean modes are not mutually exclusive. That is to say, a sign could be any combination of the three. For example, as I mentioned before, the beginning of Beethoven’s Symphony No.5 is an index for someone knocking at the door, as well as a symbol for victory.

Another effect characterizing the interaction among different musical signs is the “semantic snowballing” proposed by Thomas Turino[4]. He suggested that music can simultaneously comprise many signs which interact with one another and mix together as time goes on, like snowballs. Moreover, musical semiotics has a chain effect, in which the object of a sign becomes the representamen of another sign and so on[4]. I find this snowballing and chain reaction happening not only inside the music domain but also between music and other symbolic systems. For example, in the Chinese Gongchepu notation, there are two symbols for beats—“板” (bǎn) notated as “。” for strong beats and “眼” (yǎn) notated as “、” for weak beats. Gradually, the two characters of “板” and ”眼” became symbols for beats and many idioms were developed based on them. For instance, the idiom of “一板一眼” that literally means strictly following the standard beat is used as an adjective to describe scrupulousness and stiffness. Likewise, the English word “offbeat” that originally means not following the beat also became a synonym for “unconventional.” Thus, music symbols snowball into linguistic symbols.

An interesting theory stressed by Thomas Turino in his Signs of Imagination, Identity and Experience is the identity-building function of musical indexical signs in “high-context” communication, as Edward Hall called it. A shared music-related experience among members in an intimate social group could serve as a source of the affective power. In this way, great meanings are stored in and transmitted through music[4]. For example, a couple who watched Titanic as they were dating for the first time might consider the song My Heart Will Go On as an index for their relationship in their later lives.

Similarly, I personally find that game music has an extraordinarily strong indexical power. Many people find the background music of Super Mario Bros and Contra evokes an intense reminiscent mood. The reason, I think, lies in the repetitive strengthening of the immersive game experience. As Koji Kondo, the composer for the soundtrack of Super Mario Bros put it, he had two goals for his music: “to convey an unambiguous sonic image of the game world,” and “to enhance the emotional and physical experience of the gamer[9].” This emotional enhancement is so strong that sometimes they got upgraded to the level of symbol. The quality of the synthesized tones in early game music (a “qualisign” in Peirce’s theoretical frame[10]) is considered as the token for a music genre called “chiptune” or “8-bit music,”which greatly impacts later electronic dance music[11].

2.2. A linguistic perspective of music

In human symbolic systems, the most sophisticated and characteristic one is language. Language is like the jewel in the crown of human cognition. According to American linguist Ray Jackendoff in his book Foundations of Language, human language is really unique and much more complex than other communication systems such as the sound of whales and birds because human utterance could transmit unlimited information with unlimited and arbitrary forms from limited rules and lexicon[12]. Likewise, as another equally sophisticated symbolic system, music shares many features with language. In addition, music seems more competent in some ways such as emotional arousal or affect enhancement[10]. It is even able to function across the boundaries of languages. It is evident that music can induce universal emotion-related responses[13]. Therefore, many people consider music as another kind of language, even a “universal language of mankind[14].”

But strictly speaking, how similar is music to language? Can linguistic methods be used to approach music? How could a linguistic perspective help us understand music? Many types of research have been conducted with this theme.

In my Music as a Language, one of my weekly essays of the course of Semiotics and Cognitive Technologies, I discussed some similarities between music and language. For example, they both consist of sequences of basic elements of sound such as the phoneme in language. They both have structural rules, for example, syntax in language and chord progression in music. In addition, people from different areas tend to develop their own dialect and grammar both in language and music[15]. However, I didn’t examine them in detail and didn’t inspect their differences either.

In his Parallels and Nonparallels between Language and Music, Ray Jackendoff discussed his detailed observation of music and language in many aspects including general capacities, ecological functions, and formal structures[16]. He emphasized that language and music differ in their functions in human life. As he put it, language can be put to both propositional and affective use while music can only convey affective content, though sometimes the distinction between them blurs, for instance, in poetry.

In particular, Jackendoff reiterated the generative theory of tonal music (GTTM) proposed by himself and music theorist Fred Lerdahl[17]. He considers metrical grid as a capacity shared by music and language in the rhythmic domain, but saw no credible analogue in the use of pitch space, even in tone languages such as Chinese and many West African languages. So he concluded that the capacity of the use of linguistic pitch is entirely different from that in music.

However, other pieces of literature provide evidence pointing to the other way. In his Musicophilia: tales of music and the brain, British neuroscientist Oliver Sacks scrutinized the intriguing correlations between musical absolute pitch (AP) and linguistic background[18]. He specifically mentioned the research conducted by Diana Deutsch, a cognitive psychologist at the University of California, San Diego, and colleagues. Deutsch observed, “native speakers of tone languages – Mandarin and Vietnamese – were found to display a remarkably precise and stable form of absolute pitch in enunciating words[19].” By further detailed comparative study on AP in two populations of first-year music students—one at the Central Conservatory of Music in Beijing and the other at the Eastman School of Music in New York, Deutsch found the percentage of students possessing AP in Beijing was way higher than American students—60% vs. 14% in the group in which students began their music training at the age of 4 and 5, 55% vs. 6% in the group of age 6 and 7, and 42% vs. 0% in age of 8 and 9 group[20]. It’s clear evidence that the capacity of musical pitch correlates with that of linguistic pitch.

Back to Jackendoff’s theory, despite the structural rules such as bars and chord progressions, he didn’t think music has a counterpart to the linguistic syntactical structure as complex and strict as in language. Even GTTM’s prolongational structure suggested by himself that has a similar “recursive headed hierarchy” like language was not considered as a common ground between music and language. However, partially inspired by the hierarchical structure of actions in robotics, Jackendoff suggested that complex actions that integrate “many subactions stored in long-term memory” has the potential to serve as a candidate for a “more general, evolutionarily older function” shared by language and music since they are evidently all implemented in the Broca’s area of the brain[21].

Following this action-related road, Rie Asano and Cedric Boeckx took “the grammar of action” into account and developed a more general syntactical framework in terms of “action-related components,”in which they suggested the difference between the syntax of music and language boils down to their different goals in hierarchical plans for actions[22].

I conclude that linguistic methods might provide us with useful perspectives to understand music because they share many cognitive capacities. But we have to remember, music and language are two different symbolic systems with their own features. Just as philosopher Susanne Langer rationally put it, symbolic systems other than languages don’t have “vocabulary of units with independent meanings” and their laws are entirely different from the linguistic syntax that governs language. Therefore, we should not blindly apply linguistic principles and methods on other media such as photography, painting, and music[2].

2.3. The signs of genres

Despite so many differences between music and language, they both tend to develop many dialects or genres as we call them in music. Basically, a music genre is a conventional category of music that shares some recognizable features and patterns. Those features and patterns are musical signs too.

The signs that we use to determine genres vary a lot. Sometimes a genre is recognized through the music instruments, or more precisely, the sound quality used by musicians. For example, heavy electric guitar (sometimes distorted) may indicate rock music, while a song using acoustic guitars probably is country music.

Chord progressions sometimes serve as signs for genres. For example, the 12-bar blues chord progression is considered as a sign for blues music.

Scale sometimes has the power to determine a genre as well, especially for traditional music because most modern music uses a diatonic scale. For example, we can easily recognize the Japanese style in the famous piece of Sakura Sakura largely because the unique pentatonic Japanese scale—major second, minor second, major third, minor second, and major third (for example, the notes A, B, C, E, F, and up to A)[23]. Similarly, Chinese music, Indian Raga music, Arabic music, jazz, and blues also have their identifiable scales.

Sometimes the quality of the vocal also allows us to identify the genre. For example, the deep vocal of Amy Winehouse was very soul and jazzy, while the quick rapping vocal of Eminem indicates rap music.

However, digitalization allows for a fusion of music genres. Today, we often hear more than one genre-specific signs in one song. For example, Karen Mok, a Chinese singer from Hong Kong released a jazz album called “Somewhere I Belong” in 2013, in which she adapted twelve songs of different genres into jazz. One of the songs is While My Guitar Gently Weeps originally by the Beatles, in which Mok used guzheng, a Chinese traditional plucked musical string instrument with over 2500 years of history to play it in a jazz style, producing a really creative style of world music.

Karen Mok / While My Guitar Gently Weeps

3. Music as a cognitive artifact

In his Cognitive Artifacts, Donald Norman defined cognitive artifact as “an artificial device designed to maintain, display, or operate upon information in order to serve a representational function[24].” In this view, just like language, music is definitely a cognitive artifact.

First of all, music has many cognitive functions. When we offload cognitive efforts onto music, the performance of the whole system is improved. The reason partially lies in the affective power of music. For example, religious music helps gather people together to form a community with a transcendent purpose without much verbal persuasion. Love songs enhance lovers’ emotion, positively or negatively. Music could even serve as a political weapon. Rodriguez’s music I mentioned before is an example of this.

Second, music can transmit information and emotion through space and time. Gloomy Sunday transmits a gloomy mood to many people in different parts of the world and even was blamed for several suicides according to some urban legend. Johnny B Goode stores a story from the 1950s inside it and still passes on the information to people in the twenty-first century.

In addition, music plays an important role in every culture. It helps build a collective identity. Even four-month-old infants can recognize and prefer the music of their own culture, according to research[25].

Therefore, music becomes a perfect example for distributed cognition since it could distribute across the members of social groups, coordinate between internal and external structure, and distribute through time[26], as we analyzed above.

4. Digitalization of music

From the 1950s, musicians started to use electronic instruments to record, produce, transmit, and store music. Fourier Transform was used to transform acoustic vibration into digital signals[27]. For example, in the 1970s, Alan Kay’s Smalltalk language was used to create programs that captured tones played on the keyboard and then produced editable music scores accordingly with different colors representing different timbres[28].

At first, computers could only complete limited tasks. The range of computer synthesized sounds was restricted. As the computing power got stronger and stronger thanks to Moore’s Law, computers became meta-media and revolutionized the production and distribution of music with new hardware and software tools.

Gradually, computers cannot only imitate existing instruments with increasing precision but also create brand new sound effects and combinations that never existed before. For example, a vocoder is a kind of machine designed to record, compress, and digitalize human voices into editable formats that could be stored and manipulated in unprecedented ways. There is no doubt that digitalization opens up unlimited new possibilities for musical creation. Musicians are provided with a nearly infinite repository of materials. For example, in their song Contact from the album “Random Access Memories,” French electronic music duo Daft Punk used an audio sample from the Apollo 17 mission in which NASA astronaut Eugene Cernan was talking about something strange outside their spaceship. They also used a sample from another song called “We Ride Tonight” by Australian rock band The Sherbs. However, the combination of existing materials creates a novel musical experience.

In this digitalization process, new genres are emerging, such as electronic dance music and ambient music. An interesting demonstration of the power of digitalization is in using software, someone slowed Justin Bieber’s fast-tempo song by eight times, so that it sounds very ethereal like Sigur Rós but nothing like Justin Bieber himself.

Justin Bieber’s U Smile 800% Slower

As I discussed before, music is full of signs, which, in their essence, are recognizable patterns. Computers are good at pattern recognition and matching. One consequence is that computers are more and more capable of identifying patterns that we previously thought could only be recognized by humans. For example, some subtle signs of genres and some musicians’ personal styles can be recognized by computers. Using machine learning algorithms, computers are even competent in “creating” music in certain styles or genres.

One example is David Cope, a composer, and scientist at the University of California, Santa Cruz, who writes programs and algorithms that analyze existing music and create new music in that style. In his patent US 7696426 B2, he described his software Emmy’s logical framework, which contains pattern matching, segmentation step, hierarchical analysis step, non-linear recombination step, and then result in the output. His software takes many factors into accounts, such as pitch, duration, channel number, and dynamics. As he put it, “style is inherent in recurrent patterns of the relationships between the musical events, in more than one work.” Following this logic, based on probability principles, his software is able to capture and rank recurrent patterns as signatures for styles and create new pieces of music[29]. The following video is one of Emmy’s works, which styles in Bach.

Bach-style chorale by musical intelligence computer program created by David Cope. 

Another example is a Beatles-styled song called “Daddy’s Car” composed by an Artificial Intelligence software at SONY CSL Research Lab[30]. Today, some software can produce jazz music too.

Daddy’s Car: a song composed by Artificial Intelligence – in the style of the Beatles

Computers cannot feel music like us. They cannot feel the affection in music, sway to the music, and develop their own preferences, either. They can only break music into 0s and 1s and look for patterns in it. However, I think, the ways they learn and create music are not necessarily different from us. They both involve a process of information storing, retrieving, and pattern matching. They both need to draw patterns from perceivable signals first, then store them in long-term memories, rank them based on how recurrent they are, and match new patterns with those stored memories, although they differ a lot in details. This potential parallel might be a future research direction in order to understand music cognition and develop music-related computer algorithms.

5. Conclusion

Music is a symbolic system that can be approached through C. S. Pierce’s semiotics framework. Despite its similarities with language, music has its unique structure that has no counterpart in linguistics. So it is necessary to develop its own theoretical framework. Music also serves as a cognitive artifact that distributes through time and among people. The digitalization of music creates unlimited possibilities for the music industry, among which, the most noticeable achievement today is the computer-generated music produced by algorithms. I conclude that the computer and the human brain share similar models for musical sign recognition.

---

References

[1] Massey, Reginald, and Jamila Massey. The Music of India. Abhinav Publications, 1996.

[2] Chandler, Daniel. Semiotics: The Basics. 2nd ed. Basics (Routledge (Firm)). London ; New York: Routledge, 2007.

[3] Deacon, Terrence William. The Symbolic Species: The Co-Evolution of Language and the Brain. 1st ed. New York: W.W. Norton, 1997.

[4] Turino, Thomas. “Signs of Imagination, Identity, and Experience: A Peircian Semiotic Theory for Music.” Ethnomusicology 43, no. 2 (Spring 1999): 221–55.

[5] Tarasti, Eero. Approaches to Applied Semiotics [AAS] : Signs of Music : A Guide to Musical Semiotics. Berlin/Boston, DE: De Gruyter Mouton, 2002. http://site.ebrary.com/lib/alltitles/docDetail.action?docID=10755238.

[6] MacDonald, James. “British Open ‘V’ Nerve War; Churchill Spurs Resistance.” The New York Times, July 20, 1941. http://www.nytimes.com/learning/general/onthisday/big/0719.html#article.

[7] Bartholomew-Strydom, Craig, and Stephen Segerman. Sugar Man: The Life, Death, and Resurrection of Sixto Rodriguez. London: Bantam Press, an imprint of Transworld Publishers, 2015.

[8] JIAN, CUI. “Rock ‘N’ Roll.” Time, September 27, 1999. http://content.time.com/time/world/article/0,8599,2054475,00.html#ixzz2VFsiqp88.

[9] Schartmann, Andrew. Koji Kondo’s Super Mario Bros. Soundtrack. New York: Bloomsbury Academic, 2015.

[10] Parmentier, Richard J. Signs in Society: Studies in Semiotic Anthropology. Advances in Semiotics. Bloomington: Indiana University Press, 1994.

[11] Driscoll, Kevin, and Joshua Diaz. “Endless Loop: A Brief History of Chiptunes.” Transformative Works and Cultures 2, no. 0 (February 17, 2009). http://journal.transformativeworks.org/index.php/twc/article/view/96.

[12] Jackendoff, Ray. Foundations of Language: Brain, Meaning, Grammar, Evolution. OUP Oxford, 2002.

[13] Egermann, Hauke, Nathalie Fernando, Lorraine Chuen, and Stephen McAdams. “Music Induces Universal Emotion-Related Psychophysiological Responses: Comparing Canadian Listeners to Congolese Pygmies.” Frontiers in Psychology 5 (2015). doi:10.3389/fpsyg.2014.01341.

[14] Colin, Newton. “IN the Words of American Poet Henry Wadsworth Longfellow, ‘music Is the Universal Language of Mankind.’” Sunday Mail (Adelaide), May 16, 2004.

[15] Wang, Jieshu. “Music as a Language – Jieshu Wang | CCTP711: Semiotics and Cognitive Technology.” Accessed December 13, 2016. https://blogs.commons.georgetown.edu/cctp-711-fall2016/2016/09/22/music-as-a-language-jieshu-wang/.

[16] Jackendoff, Ray. “Parallels and Nonparallels Between Language and Music.” Music Perception 26, no. 3 (February 2009): 195–204.

[17] Jackendoff, Ray. A Generative Theory of Tonal Music. MIT Press, 1985.

[18] Sacks, Oliver W. Musicophilia: Tales of Music and the Brain. 1st ed. New York: Alfred A. Knopf, 2007.

[19] Henthorn, Trevor, Mark Dolson, and Diana Deutsch. “Absolute Pitch, Speech, and Tone Language: Some Experiments and a Proposed Framework.” Music Perception 21, no. 3 (Spring 2004): 339–56.

[20] Deutsch, Diana, Trevor Henthorn, Elizabeth Marvin, and HongShuai Xu. “Absolute Pitch among American and Chinese Conservatory Students: Prevalence Differences, and Evidence for a Speech-Related Critical Period.” The Journal of the Acoustical Society of America 119, no. 2 (January 31, 2006): 719–22. doi:10.1121/1.2151799.

[21] Patel, Aniruddh D. “Language, Music, Syntax and the Brain.” Nature Neuroscience 6, no. 7 (July 2003): 674–81. doi:10.1038/nn1082.

[22] Asano, Rie, and Cedric Boeckx. “Syntax in Language and Music: What Is the Right Level of Comparison?” Frontiers in Psychology 6 (2015). doi:10.3389/fpsyg.2015.00942.

[23] Harich-Schneider, Eta. A History of Japanese Music. London: Oxford University Press, 1973.

[24] Norman, Donald A. “Cognitive Artifacts.” In Designing Interaction, 17–23. New York: Cambridge University Press, 1991.

[25] Soley, Gaye, and Erin E. Hannon. “Infants Prefer the Musical Meter of Their Own Culture: A Cross-Cultural Comparison.” Developmental Psychology 46, no. 1 (n.d.): 286–92.

[26] Hollan, James, Edwin Hutchins, and David Kirsh. “Distributed Cognition: Toward a New Foundation for Human-Computer Interaction Research.” ACM Trans. Comput.-Hum. Interact. 7, no. 2 (June 2000): 174–196. doi:10.1145/353485.353487.

[27] Chagas, Paulo C. Unsayable Music: Six Reflections on Musical Semiotics, Electroacoustic and Digital Music. Leuven University Press, 2014. http://www.jstor.org.proxy.library.georgetown.edu/stable/j.ctt9qf0qh.

[28] Kay, Alan, and Adele Goldberg. “Personal Dynamic Media.” Edited by Noah Wardrip-Fruin and Nick Montfort. Computer 10, no. 3 (March 1977): 31–41.

[29] Recombinant music composition algorithm and method of using the same. Accessed December 14, 2016. http://www.google.com/patents/US7696426.

[30] “AI Makes Pop Music in Different Music Styles.” Flow Machines, September 19, 2016. http://www.flow-machines.com/ai-makes-pop-music/.

Hi Dr. Irvine,

I am posting some notes and a general outline for what I am thinking. I expanded a bit on some of the points where I thought that the connections I’m drawing might not be immediately clear. I’ve also included a running bibliography.  Hopefully, you’ll be able to find some method in the madness. Thank you for your willingness to take a look at this. Any feedback you might have is definitely welcome!

Best,
Rebecca

Abstract – 

Working idea...The concept of the smartpen and paper interface present a provide a more dynamic interface for end-user tailorability/mobility and a potential parardigm shift in the hybridization of analogue and digital interface.

In this paper I will provide an overview of Smartpen technology, using the Livescribe Echo as a primary of the example, in order to trace the conceptual development of this technology as an important progression of symbolic/computational mediation. While this technology is currently in a relatively rudimentary state, I will argue that its refinement and integration could lead to more advanced forms of symbolic abstraction and cognitive offloading. Most importantly, I will make a case for the potential paradigmatic shift that technology of this type may hold for the ways in which we use and understand the culturally ingrained artifacts of pen and paper.

Technical Overview/Definition of Smartpen/paper

1. Hardware
   1. Pen
   2. Infrared Camera & Audio Recording
   3. Bluetooth
   4. Paper Grid 
2. Software
   1. Bundle
   2. Apps

 

https://www.livescribe.com/en-us/media/pdf/Livescribe_Tech_Specifications.pdf

Conceptual Overview

1. Sign formation (Pierce -Concepts /Anderson – Application)
   1. Iconic significance – the Smartpen allows for digital remediation of icons developed/identified by the user.
   2. Indexical Significance – The Smartpen software allows for unique restructuring/sequencing of handwritten notes. It also allows users to search for words in handwritten notes.
   3. Symbolic significance – The Smartpen personalizes notes by digitizing things such as handwriting, which carries symbolic significance
2. Mobilization – (Moggridge pg. 191) – The Smartpen is more portable than a laptop or tablet. It has built in storage that allows users to use it when not connected to Bluetooth.
3. Remediation/Meaning Stacks- (Irvine) – The Smartpen syncs handwritten notes to the computer which can then be transcribed into digital text. It also “instantly” hyperlinks or programs text on paper to perform command functions. For example, you can perform functions on the pen by drawing a small cross to use as arrow indicators. Also, text with which you have recorded audio will begin audio playback when tapped by the point of the pen on the piece of paper! 
   1. Signs grow (Irvine/Pierce?)
   2. abstraction? (Evans)
   3. recursion? (Evans)
4. Sutherland’s sketchpad (Sutherland) – 
5. Metamedium (Manovich)
   1. Smartpen as a simulation of prior media extended with new properties (Manovich 110)
   2. Hybridization (Manovich)
   3. Softwarization of Pen and Paper
      1. Encoding/Digitization – (Floridi?)
      2. Hypertext (Englebart)

HCI

1. Issues (Murray/product reviews)
   1. Size of pen
   2. Lack of integration
2. Suggestions

Longterm Trajectory

1. Symbolic potential
   1. Personalization and control of images/text
   2. Alan kay’s vision
2. Smart Textbooks?
3. Integration/Tablet PCs – Moggridge (pg 198)
4. Paradigm Shift for pen and paper

Running Bibliography

• Andersen, Peter Bogh. 2001. “What Semiotics Can and Can’t Do for HCI.” Knowledge-Based Systems 14: 419–24.
• Donald, Norman. 1991. “Cognitive Artifacts.” In Designing Interaction: Psychology at the Human-Computer Interface, 17–38. Cambridge University Press.
• Engelbart, Douglas. “Augmenting Human Intellect: A Conceptual Framework.” In The New Media Reader, edited by Noah Wardrip-Fruin and Nick Montfort, 93–108. Cambridge, MA: MIT Press, 2003.
• Evans, David. Introduction to Computing: Explorations in Language, Logic, and Machines. August 19, 2011 edition. CreateSpace Independent Publishing Platform, Creative Commons Open Access: http://computingbook.org/.
• Floridi, Luciano. Information: A Very Short Introduction. New York: Oxford University Press, 2010.
• Irvine, Martin (2016). Semiotics, Symbolic Cognition, and Technology Key Writings. Compiled and edited with commentary by Martin Irvine. Communication, Culture & Technology Program, Georgetown University.
• Irvine, Martin. 2016. “The Museum and Artworks as Interfaces: Metamedia Interfaces from Velázquez to the Google Art Project.” PowerPoint Presentation.
• Johnson, Jeff. 2014. Designing with The Mind in Mind: Simple Guide to Understanding User Interface Design Guidelines. Second edition. Amsterdam ; Boston: Elsevier, Morgan Kaufmann.
• “Livescribe(TM) Connect(TM) Makes Handwritten and Spoken Information Easily Shareable with Facebook, Evernote(R), Google(TM) Docs and Email – all from Paper Livescribe Introduces the Affordable GBP99 2GB Echo Smartpen Starter Pack.” 2011.PR Newswire Europe Including UK Disclose. http://proxy.library.georgetown.edu/login?url=http://search.proquest.com/docview/896962663?accountid=11091.
• Manovich, Lev. 2013. Software Takes Command: Extending the Language of New Media. International Texts in Critical Media Aesthetics. New York ; London: Bloomsbury.
• Moggridge, Bill. 2007. Designing Interactions. Cambridge, Mass: MIT Press.
• Murray, Janet H. 2011. Inventing the Medium: Principles of Interaction Design as a Cultural Practice. Cambridge, Massachusetts: The MIT Press. http://site.ebrary.com/lib/alltitles/docDetail.action?docID=10520612.
• Sutherland, Ivan. “Sketchpad: A Man-Machine Graphical Communication System.” In The New Media Reader, edited by Noah Wardrip-Fruin and Nick Montfort, 109–126. Cambridge, MA: MIT Press, 2003.