Computers are powerful tools, and today we cannot think to complete our daily tasks without using them. But how did we come to use programming languages in computers? What are the fundamental concepts that lead to the idea of artificial languages, and how does that connect to our use of natural language? This paper discusses the adoption of sign-theoretic perspective on knowledge representation, the application that lies under the human-computer interaction and the fundamental devices of recognition. This paper suggests the study of computer systems as sign systems, on a semiotics perspective, based on Pierce’s triadic model and looks at an application such as the General Problem Solver, an early work in computerized knowledge representation.
Being a computer science student, I found out how powerful the art of programming and coding can be, but what fascinates me is the representation of all these different programming languages that we use to program. In digital computers, the user’s input is transmitted as electrical pulses, only having two states, on or off, respectively represented as either a 1 or a 0, and this sequence of 0’s and 1’s represents the “computer’s language”. But how did something so easy in concept, just having two states, becomes something so fundamental? Of course we have to take a step back and think about all the principles and interactions that lead us to this idea, the most important of them being the human computer interaction and the meaning behind different signs and symbols.
Computer Systems as Sign Systems
This is where I turn to semiotics, the study of sign and symbols, to understand the meaning behind these representations. Andersen presents semiotics as a framework for understanding and designing computer systems as sign systems. There are different semiotics methods that can be applied to different levels of computer systems, but I will focus on a particular perspective, one that looks at computer systems as targets of interpretations. I am interested to look at programming and programming languages as a process of sign creation, and the semiotic approach behind it. It is interesting to view computer systems as signs and symbols, whose main function is to be perceived and interpreted by a group of users. Anderson suggests that when you think of computer systems through the lenses on semiotics, they are not ordinary machines anymore. Rather, they are symbolic machines constructed and controlled by means of signs. The interface of a system is an example of a computer based sign, and using this system means that it involves the interpretation and manipulation of text and pictures. And underneath the interface, there are other signs that we see. The system itself is specified by a program text or a language, which on its own is a sign. Then, the actual execution of the program requires a compiler, which its main function is to transform code written in one programming language, into another programming language. That makes the compiler a sign. If we continue with this approach, passing through different layers of the system, we will encounter more signs, from the operating system to assembly code, to machine code.
There are many kinds of semiotics theories when in comes to defining the concept of a computer-based sign and computer system.
1. The Generative paradigm
This paradigm was founded by Noam Chomsky in 1957. This generative grammar is focused on the individual language user, not on the social process of communication. This paradigm looks at a language based on a rule-defined set of sentences. Halliday explains why this is not a good approach:
“A language is not a well-defined system, and cannot be equated with “the set of grammatical sentences”, whether that set is conceived as finite or infinite. Hence a language cannot be interpreted by rules defining such a set. A language is a semiotic system…-what I have often called a “meaning potential”…Linguistics is about how people exchange meanings by “languaging.” (Halliday 1985).
2. The Logic Paradigm
This paradigm was first founded by Frege, but it is counted as a linguistic theory with the logical grammars of Richard Montague (1976). This paradigm consists in translating natural language sentences into logical formulas on which rules of inference can operate, and that’s why this theory has become an integrated part of computer science. One way to construct such system is to represent knowledge in terms of logical statements, to translate queries into logical formulas and to let the system try to prove the query from the knowledge based. Now days we can link this theory to the idea of a “neural network”. By that I mean you program and built a system to achieve a certain goal, let’s say you write a program that goes through different image files and selects the images where dogs appear. You can feed data to a system, the more the merrier, and train to find the images that you are looking for. But the problem with this kind of approach is that it is not a linguistic approach, and it does not describe a linguistic behavior, rather a factual one. If we use logic and facts, we defeat the purpose of understanding the sign representation.
3. The Object-Oriented paradigm
There is a relation between the concept of object-oriented programming and semiotics, where the system is seen as a model of the application domain, and we see the components of classes and objects and their interactions in the domain. These concepts are also characteristics for a semantic analysis that go back to Aristotle, and his idea of the hierarchy of classes, also known as the Porphyrian tree. (Eco, 1984)
4. Pierce’s triadic process
If we are talking about computers and semiotics there is without doubt one person that has to be mentioned for his incredible work and that is Charles Sanders Peirce (1839-1914). As Irvine mentions in his paper, Peirce, is without question the most important American philosopher of the mid-19th and early-20th century. His background is very interdisciplinary. Peirce was a scientist, mathematician, cartographer, linguist and a philosopher of language and signs. He commented on George Boole’s work on “the algebra of logic” and Charles Babbage’s models for “reasoning machines”. Both of these concepts are fundamental for logic used today in computing systems.
Pierce’s process on meaning-making, reasoning and knowledge is a generative process, also known as a triadic experience, and is based on human sign systems and all different levels of symbolic representation and interpretation. This process in explained through Martin Irvine’s evaluation, “A Sign, or Representamen, is a First which stands in such a genuine triadic relation to a Second, called its Object [an Object of thought], as to be capable of determining a Third, called its Interpretant, to assume the same triadic relation to its Object in which it stands itself to the same Object,” (Irvine).
Although many fundamental computer science principles apply binary states, Peirce discovered that the human social-cognitive use of signs and symbols is a process that can never be binary, it’s never either science and facts or arts and representations. Rather, the process of understanding symbols and signs is a process that covers everything from language and math to scientific instruments, images and cultural expressions.
The Peircean semiosis
As Irvine suggest, the Peircean semiotic tradition provides an open model for investigating the foundations of symbolic thought and the necessary structures of signs at many levels of analysis:
- the generative meaning-making principles in sign systems (individually and combined with other systems like in music and cinema/video),
- the nature of communication and information representation in interpretable patterns of perceptible symbols,
- the function of physical and material structures of sign systems (right down into the electronics of digital media and computing architectures),
- the symbolic foundations of cognition, learning, and knowledge,
- how a detailed semiotic model reveals the essential ways that art forms and cultural genres are unified with scientific thought and designs for technologies by means of different ways we use symbolic thought for forming abstractions, concepts, and patterns of representation,
- the dialogic, intersubjective conditions of meaning and values in the many lived contexts and situations of communities and societies.
Now that we have gone through different paradigm models and semiotics theories in helping us understand computers as systems, and after we are introduced to Pierce’s process we are going to take a closer look at the field of computational semiotics and its applications today.
Computational semiotics is an interdisciplinary field that draws on research in logic, mathematics, computation, natural language studies, cognitive sciences and semiotics properties. A common theme across these different disciplines is the adoption of sign-theoretic perspective on knowledge representation. Many of its application lie in the field of human-computer interaction and fundamental devices of recognition.
Tanaka-Ishii in her book “Semiotics of Programming” makes the point that computer languages have their own type of interpretative system, external to the interpretative system of natural languages. That is because human beings do not think in machine language, and all computer language expressions are meant for interpretation on machines. Computer languages are the only existing large-scale sign system with an explicit, fully characterized interpreter external to human interpretative system. Therefore, the application of semiotics to computer languages can contribute to the fundamental theory of semiotics. Of course that computation is different from human interpretation, and interpretation of artificial languages differ from that of natural languages, but understanding the semiotic problems in programming languages leads to considering the problems of signs.
Many of the concepts, principles of computer programming have derived from technological needs, without explicitly stating the context of human thought. An example of this idea is the paradigm of object-oriented programming which we saw earlier.
Let’s take a look at a simple program written in Java that calculates the area of a triangle given three sides:
At a glance, this is a simple code for people whose field of study is Computer Science, but even if that is not the case, you can understand the code. Why?
Because the mathematical principles and formula for finding the area of a triangle still apply.
Following Heron’s formula to get the area of a triangle when knowing the lengths of all three sides we have:
Let a,b,c be the lengths of the sides of a triangle. The area is given by:
Area = √p(p−a) (p−b) (p−c)
where p is half the perimeter, or a+b+c/2
Even you may not know how to code, just by looking at the code and reading the words, you can guess what the program does. The same formula from above is applied in our code, where the user is prompted to enter the three sides and the area of the triangle is calculated. So, this explains how powerful the meaning behind symbols and signs is, and how important they are, especially in a field like computing.
Let’s take a look at an early work in computerized knowledge representation.
General Problem Solver by Allen Newell, J. C. Shaw and Herbert A. Simon.
Earliest work in computerized knowledge representation was focused on general problem solvers such as the General Problem Solver, a system developed by Allen Newell, J.C. Shaw and Herbert A. Simon created in 1959. Any problem that can be expressed as a set of well-formed formulas (WFFs) or Horn clauses, and that constitute a directed graph with one or more sources (viz., axioms) and sinks (viz., desired conclusions), can be solved, in principle, by GPS. Proofs in the predicate logic and Euclidean geometry problem spaces are prime examples of the domain the applicability of GPS. It was based on Simon and Newell’s theoretical work on logic machines. GPS was the first computer program which separated its knowledge of problems (rules represented as input data) from its strategy of how to solve problems (a generic solver engine).
The major features of the program are:
- The recursive nature of its problem-solving activity.
- The separation of problem content from problem-solving techniques as a way of increasing the generality of the program.
- The two general problem-solving techniques that now constitute its repertoire: means-ends analysis, and planning.
- The memory and program organization used to mechanize the program(this will be noted only briefly, since there will be no space to describe the computer language (IPL’S) used to code GPS-I.
GPS as the authors explains, grew out of an earlier computer programmer, the Logic Theorist, which discovered proofs to theorems in the sentential calculus of Whiteheand and Russell. It is closely tied to the subject matter of symbolic logic. The Logic Theory lead to the idea behind GPS, which is the simulation in the problem-solving behavior of human subjects in the psychological laboratory. The human data were obtained by asking college sophomores to solve problems in symbolic logic, or as we know it “thinking out loud” as much as possible while they worked.
The structure of GPS
GPS operates on problems that can be formulated in terms on objects and operators. As the authors explains, an operator is something that can be applied to certain objects to produce different objects. The objects can be characterized by the features they possess, and by the differences that can be observed between pairs of objects. Operators may be restricted to apply to only certain kinds of objects, and there may be operators that are applied to several objects as inputs, producing one or more objects as output. (Simon)
Using this idea, a computer program can be described as a problem to be solved in these terms. So, the objects are possible contents of the computer memory, the operators are computer instructions that alter the memory content. A program is a sequence of operators that transform one state of memory into another, the programming problem is to find such a sequence when certain features of the initial and terminal states are specified.
In order for the GPS to operate within a task environment it needs several main components:
- A vocabulary for talking about the environment under which it operates.
- A vocabulary dealing with the organization of the problem-solving process.
- A set of programs defining the term of the problem-solving vocabulary by terms in the vocabulary for describing the task environment.
- A set of programs applying the terms of the task-environment vocabulary to a particular environment such as: symbolic logic, trigonometry, algebra, integral calculus.
Let’s take a look at the executive organization of the GPS. With each goal type, a set of methods is associated in achieving that goal.
GPS solved simple problems such as the Towers of Hanoi, a famous mathematical puzzle.
The GPS paradigm eventually evolved into the Soar architecture for artificial intelligence.
When looking at computers from a semiotics perspective, we can understand the meanings behind the system and the different layers that make the system. From the interface of a system to machine code, we can find signs and symbols all the way down. It is interesting to see the correlation and dependencies between an artificial language, such as the different programming languages that we use, and the usage of our natural language. Humans as cognitive beings have the advantages of having a symbolic faculty and many sign systems. That helps us to make mental relations between perceptions and thought, and using advance technology today we talk about virtual reality and augmented reality.
Looking at Pierce’s model on meaning-making, we can interpret different levels of symbolic representation, and understand that the world we live in, the technology we use can never be binary, rather it is a dynamic environment with many interrelated processes.
When looking at computers as systems, we have to keep in mind the technical structure of the system, its design and implementation and its function. But, we also have to look at the semiotics process, and when looking at the theoretical framework of a system, we should also look at concepts for analyzing signs and symbols, where users interpret in their context of work. From the semiotic approach, interfaces of a system are not separated from its functionality.
By approaching our digital and computational life from the semiotic design view, and by understanding that semiotics, in its standard conceptualization, is a tool of analysis, we can see that we live in a media continuum that is always already hybrid and mixed, and that everything computational and digital is designed to facilitate our core human symbolic-cognitive capabilities. By using semiotics as a tool of analysis we can turn computers into a rich medium for human-computer interaction and communication.
Andersen, Peter B. “Computer Semiotics.” Scandinavian Journal of Information Systems, vol. 4, no. 1, 1992
Andersen, P.B. . A Theory of Computer Semiotics, Cambridge University Press, 1991
Clark, Andy and David Chalmers. “The Extended Mind.” Analysis 58, no. 1, January 1, 1998
De Souza, C.S., The Semiotic Engineering of Human-Computer Interaction, MIT Press, Cambridge, MA, 2005
Eco, U., A theory of Semiotics. The MacMillian Press, London, 1977
Gudwin, R.; Queiroz J. (eds) – Semiotics and Intelligent Systems Development – Idea Group Publishing, Hershey PA, USA, 2006
Halliday, M. A. K., Language as Social Semiotic. The Social Interpretation of Language and Meaning, Edward Arnold, London 1978
Halliday, M. A. K., System and Function in Language, Oxford University Press, Oxford, 1976
Hollan, James, Edwin Hutchins, and David Kirsh. “Distributed Cognition: Toward a New Foundation for Human-computer Interaction Research.” ACM Transactions, Computer-Human Interaction 7, no. 2, June 2000
Hugo, J. “The Semiotics of Control Room Situation Awareness”, Fourth International Cyberspace Conference on Ergonomics, Virtual Conference, 15 Sep – 15 Oct 2005
Irvine, Martin, “The Grammar of Meaning Making: Signs, Symbolic Cognition, and Semiotics.”
Irvine, Martin, “Introduction to Linguistics and Symbolic Systems: Key Concepts”
Mili, A., Desharnais, J., Mili, F., with Frappier, M., Computer Program Construction, Oxford University Press, New York, NY, 1994
Newell, A., A guide to the general problem-solver program GPS-2-2. RAND Corporation, Santa Monica, California. Technical Report No. RM-3337-PR, 1963
Newell, A.; Shaw, J.C.; Simon, H.A., Report on a general problem-solving program. Proceedings of the International Conference on Information Processing, 1959
Norvig, Peter, Paradigms of Artificial Intelligence Programming: Case Studies in Common Lisp. San Francisco, California: Morgan Kaufmann. pp. 109–149. ISBN 1-55860-191-0, 1992
Peirce, Charles S. From “Semiotics, Symbolic Cognition, and Technology: A Reader of Key Texts,” collected and edited by Martin Irvine.
Ray Jackendoff, Foundations of Language: Brain, Meaning, Grammar, Evolution. New York, NY: Oxford University Press, USA, 2003
Renfrew, Colin. “Mind and Matter: Cognitive Archaeology and External Symbolic Storage.” In Cognition and Material Culture: The Archaeology of Symbolic Storage, edited by Colin Renfrew, 1-6. Cambridge, UK: McDonald Institute for Archaeological Research, 1999.
Rieger, Burghard B, A Systems Theoretical View on Computational Semiotics. Modeling text understanding as meaning constitution by SCIPS, in: Proceedings of the Joint IEEE Conference on the Science and Technology of Intelligent Systems (ISIC/CIRA/ISAS-98), Piscataway, NJ (IEEE/Omnipress), 1998
Sowa, J, Knowledge representation: logical, philosophical and computational foundations, Brooks/Cole Publishing Co. Pacific Grove, CA, USA, 2000
Tanaka-Ishii, K. “Semiotics of Programming”, Cambridge University Press, 2010
Wing, Jeannette “Computational Thinking.” Communications of the ACM 49, no. 3, March 2006