Author Archives: Matt Lindia

My Television is a Computer: Design Thinking in Apple TV’s Interface

Abstract:

            For the past decade or so, television has been undergoing a shift  from a broadcast medium to a digitally networked one. And while such a shift entails certain technical advances and the designs therein, it also indicates a need for redesigning the user interface of the television. Unsurprisingly, Apple has engaged with the design of digital television interfaces from the very beginning, primarily through their Apple TV product. This paper takes the Apple TV – and the constituent parts which are involved in its human-computer interaction – as a case study in the design problems posed in applying computational media to television. Namely, it first considers the Apple Remote, and the challenges of using a remote as an input device for a computational machine, as opposed to a keyboard and mouse. Secondly, it considers the two graphical user interfaces designed for the Apple TV, the Front Row interface and the tiled grid. This section deals with the design principles common for 10-foot user interfaces while also dealing with the ways in which these two interfaces accomplished different purposes within the sociotechnical system built around Apple TV.

Introduction

In Apple’s never-ending quest to achieve unparalleled dominance in every segment of the media market, their forays into television proved multiple and varied in success. Even laying aside attempts which largely failed, such as Macintosh TV and the Apple Interactive Television Box, the history of Apple TV – or iTV as it was initially called – is convoluted and filled with eccentricities. The Apple TV as a product, initially introduced in 2007, underwent a number of iterations, even to the point that branding aside, the Apple TV which one might purchase today, in 2019, is designed to accomplish very different tasks than the one released in 2007. These iterations mirror trends in consumer computing and the rise of streaming video services. Even given these technical evolutionary stages, however, one persistent design problem (and its solution) separates out Apple TV as indicative of digital television and distinct from other technologies in the late 2000’s through present day: how to design a computer as a TV. From the perspective of 2019, this question may seem quaint, or even banal. After all, the answer seems straightforward enough, simply connect the TV to a laptop through an HDMI cord, or another technology to post-date Apple TV, Google Chromecast. But neither of these solutions actually solves the problem that Apple was tackling. Instead, they create systems where the television screen acts as a mirror for the PC or laptop monitor. For the user to manipulate the program, they turn to their laptop to do so. In other words, the television is a glorified monitor, and not a unique computing machine with both input devices and interfaces designed for the medium of the TV. The problem of TV interface design is one which defines the cultural moment of 2007 to 2009 where the role of the television decidedly changed from that of a broadcast medium to a medium which took full advantages of the affordances of networked computing (Braun 2013). These affordances, which according to Simon and Rose (2010) had been incorporated in limited ways with television technologies since the 1990’s, required distinct sorts of design choices for the expanded application of computational affordances in the last half of the 2000’s: designing the television as a system of information and designing the television as a user interface. The first goal of this paper, then, is to explore design choices which informed technical components of Apple TV employed in several iterations: data synchronization and data streaming. This section will also discuss the consequences of those choices in terms of design and user experience, as well as reasons for the ultimate triumph of streaming over synchronization. Second, this paper will seek to understand the design principles behind Apple TV as a computer designed for the television and according to its affordances and constraints; which is to say, without textual input or an X/Y pointing device (i.e. a mouse). This section will discuss the Apple remote as an input device and the principles of 10-foot user-interfaces at large, as well as the specific design of the Apple TV interface.

Computing TV

            It takes no great technologist to determine that the inputting mechanism for a television significantly differs from the inputting mechanism for a computer. To understand the exact interface problems which the Apple TV ultimately addressed, however, one must first understand the gap in the market Apple was attempting to fill through this product: on-demand television. To be fair, pay-per-view television was developed in the United States as early as the 1950’s (Smith 2001). Pre-digital on-demand television involved calling in to the pay-per-view provider by telephone in order to request the desired program. With the advent of digitally transmitted television in the 1990’s, as well as the boom in consumer computing in the late 20th century, the constraints of broadcast and cable television for providing on-demand video wavered in the face of new digital affordances. For Apple in 2007, during the incunabula stage of the Apple TV, great strides were being made toward designing an interface wherein users could take advantage of iTunes’ ability to offer video on demand not on their laptop, but on their television (Chamberlain 2010). In a certain sense, with digital television technologies taking the position of market dominance previously held by cable, Apple needed a way to compete with the affordance of digital TV to provide on-demand video within its own technical system. Apple had been providing on demand video rentals and purchases themselves for about 4 years, at the time, through their innovative iTunes platform, but the platform suffered severe limitations due to some of the design choices of other Apple products at the time. In other words, before the release of the iPhone in 2007, Apple’s only networked devices – that is, the only devices with the capability to access the Internet – were its laptop and desktop products. In those early days, the constraints of the iPod required a level of dedication from the user in order to purchase or consume on-demand music or video. Anyone who owned an iPod in its earliest iterations surely remembers having to buy a song or movie on the desktop or laptop iTunes interface, next plugging the iPod into the computer’s USB port, waiting for the iPod to sync, and finally dragging the music file from the iTunes library to the iPod device so as to signal the initiation of copying the song or movie file to the iPod from the personal computer. Only once this process was completed could the user consume their “on-demand” product away from their personal computer.

Now, this account of the iTunes store only has to do with the design problems of Apple TV insofar as it sets the scene by describing the state of Apple’s dealings with on-demand video at the time of the Apple TV’s release. Particularly, this early approach to using Internet-based marketplaces, accessed on personal computers and not media players themselves, explains Apple’s decision to design the first generation of Apple TV effectively as a storage and playback unit for media files bought on the iTunes store. Just as iPod users bought content from their personal computer, and then connected their iPod to the computer in order to copy files to the media player, first generation Apple TV users originally had no other option than to purchase media on a personal computer (Cheng 2007). Oddly, this unfortunate legacy from iPods and other Apple media players was not a technical constraint – as even the first Apple TV’s were networked computers with Internet access, which Apple capitalized on in promoting access to YouTube from the Apple TV – but instead a simple instance of technological “lock-in,” where a technological practice persists, not because it offers any advantages, but because it is an established method for doing something (David 1985). Fortunately for the user, this inconvenience lasted less than a year, as a software update in January 2008 provided users with the opportunity to access the iTunes store directly from the Apple TV interface, removing the inconvenient need for a personal computer to mediate this access.

Of course, this Apple TV differed significantly from the ones on the sterile white shelves of Apple stores today if not in its technical capacity, then in its technical function. In other words, even though Apple TV employed data streaming in order to play YouTube videos, it primarily relied on the technique of data synchronization for playing videos purchased from its own store (Pegoraro 2007). Unlike streaming, which involves sending media files from router to router as Internet packets, which are dissembled upon sending and reassembled upon receiving, data synchronization involves the data set on one device being exactly copied, or mirrored, on another. And while data synchronization offered some advantages, such as ensuring that a user’s iTunes library on a personal computer might match their library on their Apple TV, its major drawback, from a design perspective, was that it required the files to be downloaded to the Apple TV itself. In some respects, this constraint only becomes evident looking through the rearview mirror: the design of the fifth generation Apple TV, with only 32 and 64 GB’s of storage relies on its capabilities to stream music and video, and therefore to not require much space by way of internal storage. Comparatively, the first generation Apple TV’s basic model boasted 164 GB of storage. To be fair, the first generation Apple TV offered capabilities for both streaming and synchronization, however, synchronization was the default mode, as streaming was foreign to many users (Pegoraro 2007). Regardless, only after streaming made data synchronization largely obsolete would models with less storage space appear more tenable. The second generation of Apple TV, however, left behind the relic of data synchronization in favor of streaming for all time-based media.

Input and Interface Design for Apple TV

            Even with the technical capacity to use personal and consumer computing products as a gateway to content for the Apple TV, which a traditional television could then display on its screen, Apple still needed to overcome significant design challenges in terms of the input and interface of this media player. Particularly considering the emergence of on-demand movies and media on digital cable in the 1990’s and 2000’s, Apple needed a human-computer interface which simultaneously appealed to the brand’s design sensibilities, as well as provided users with straightforward navigation techniques where they could browse, locate, and access media easily and without frustration. In a word, Apple set out to design an interface and user input system which allowed for the user to take full advantage of the computational power of the Apple TV, while still perpetuating the user experience of using a television, and not a computer, which is to say, the primary input mechanism was a remote and not a mouse and keyboard. Because interfaces facilitate human-computer interaction, the following discussion of user-facing software and input devices often refer back to one another; the software must accommodate the constraints of the input device, and the input device should maximize the affordances of the software. At the same time, they can be more or less taken as individual modules in the combinatorial design of the Apple TV, and therefore I will address them as such. To be clear, by discussing the design of Apple TV’s interface, I am not only referring to the design of the Graphical User Interface (GUI), but the deep cultural history of the word as a meeting and the joining point for two disparate artefacts (Irvine n.d.). Specifically, Daniel Chamberlain’s definition of interface helpfully charts out the territory to be covered throughout this paper. He writes, “In a material sense we can think of those interfaces as consisting of three parts—a physical means of interacting with a screen-based display driven by dedicated software” (Chamberlain, p. 85, 2010, original emphases). Particularly, I am concerned with understanding the design of the GUI as interfacing the physical means of interacting (the Apple Remote) to the networked content (movies, music, etc.) on their screen-based display. The previous discussion took the dedicated software to task, albeit in limited scope, because the software, in the case of television’s adaptation of digital affordances, is not necessarily unique to the television medium, and therefore of secondary concern for this paper. In the following paragraphs, then, I consider the design problems and solutions of input and interface for Apple TV from both the design of the Apple Remote, as an input technology, as well as the Front Row interface and the subsequent tiled app interface of tvOS.

Apple Remote

Figure 1: First Generation Apple Remote

What’s a television without a remote? Ever since the television remote was invented in the latter half of the 20th century, its status as the ubiquitous mode for interacting with the television has remained unchallenged. For Apple to carry the experience of television watching through to Apple TV, they were wise to adopt this piece of hardware in their sociotechnical system – even if in the second generation and onward they enabled Apple TV with Bluetooth capabilities to connect with Bluetooth QWERTY keyboards for the convenience of those particularly fed up with the obstinance of “typing” with the Apple Remote. The Apple Remote was largely designed so as to allow a user to manipulate the Front Row interface – through which the user interacted with the Apple TV. Until a major design change in the fourth  generation of Apple Remote, its design visually borrowed from the iPod family, boasting only a minimalistic six buttons and the iconic “wheel” of the iPod (albeit this wheel did not function as a wheel and was simply four buttons positioned along the circumference of a circle). Because of their limited number, many of the buttons accomplished more than one feature, based on

Figure 2: Second Generation Apple Remote

whether the user was interacting with the Apple TV through the course of playing media or through navigating the menu. The fast-forward and rewind buttons, for example, doubled as left and right navigators, as the increase and decrease volume buttons doubled as up and down navigators. Through this technique, the design choice of positioning these buttons along the wheel proves more helpful than a simple perpetuation of Apple’s iconic brand and design philosophy. In other words, while the wheel did in fact preserve Apple’s visual aesthetic in the Apple Remote, it also allowed for these buttons to perform semiotic double duty. For example, if rewind/fast-forward and increase/decrease volume had been positioned in rows or columns, this implicit directionality of left/right and up/down would have been lost. For the buttons to perform double duty in the hypothetical column and row setup, they would need to be labeled as such so that they user might properly interpret their function. By positioning them along the wheel, the designers at Apple assume that their users have already been conditioned to understand the significance of directionality and design their hardware accordingly. Bruno Latour (under the pseudonym Jim Johnson) calls this phenomenon pre-inscription – the information or learning which the user is assumed to have before interacting with a technology (Johnson 1988).

The obvious limitation of such a remote, however, were its limited affordances for inputting textual data, which inevitably created problems for any sort of search functions. For a technology which organized entire catalogs of films and music, this constraint was fairly significant. Even with the adaptation of connectivity to Bluetooth keyboards with the second generation, Apple had designed the product so as to assume that the Apple Remote would function as the primary input device – any choice to allow the user to use a keyboard would be an extra flourish and affordance. For this reason, Apple had to ensure that whatever graphical user interface it designed or used would be entirely navigable through the affordances of the Apple Remote. This meant not only foregoing the QWERTY keyboard, but also abandoning any sort of mouse or cursor. Fortunately for Apple, interfaces which solved this problem proved not to be design impossibilities. And while, as we will see, the first interface did not endure until the present day, it characterized much of the early user experience of Apple TV. Furthermore, many of the principles of which it consisted still can be identified in the design of Apple TV’s current interface.

Front Row Interface

Figure 3: Front Row Interface

While the Front Row interface was always conceived of by Apple as a multimedia player, it pre-dated the Apple TV by around two years. And even while the user might have originally interacted with the Front Row interface through the medium of a Macintosh computer, Front Row’s release coincided with the release of the first generation Apple Remote, as the two were designed for one another. The Front Row interface was Apple’s first attempt at developing “10-foot user interface” (10-foot UI). 10-foot UI’s emerged with the rise of smart TV’s and the need to not only account for the manipulation of icons and symbols with a remote device and not a mouse and keyboard, but also the increased distance of the user from the screen when watching television as opposed to working on a computer (the 10 feet in 10-foot UI reference this distance) (Lal 2013). This distance generated several constraints of transitioning computing technology to the medium of the television (the first being the use of the remote as an input device): screens needed to be treated as single entities, and not windows, icons needed to be larger, and the interface should make it clear to the user which icons or symbols with which they were interacting by highlighting or otherwise indicating the icon at hand (Lal 2013).

According to Michael Moyer (2009), through the course of their development, 10-foot UI’s solved the problem by employing one of two major methods, the first of which quickly proved inferior to the second. The more obvious, but less successful design of 10-foot UI’s involved developing browsers for the television screen. As alluded to above, this involved enlarging the search bar and other icons, so as to ensure its visibility from the couch. Ultimately, however, it failed to provide a way to input text without requiring either the extreme patience of selecting every letter individually out of the entire alphabet for textual input or requiring the connection of a Bluetooth keyboard. For this reason, the second common design for 10-foot UI’s – a widget-based design – remains the industry standard. The widget system involves sorting out each program or feature into separate icons, not dissimilar to the manner in which apps are presented on smartphones, which the user can then sort through in order to select their desired function. Particularly in a digital economy where many Internet-based services are not housed by companies which manufacture communication technologies themselves, the widget design seems to offer many advantages. For example, instead of relying on a browser to mediate access to Netflix, Hulu, YouTube, Spotify, or any other number of media companies, the widget offers the user direct access to the content therein. Perhaps the greatest testimonial to these advantages, however, can be found in the fact that many of our touchscreen technologies, such as smartphones and tablets, employ the design of widget-based interfaces, at least in part, even when interfaces which rely on searching or textual input more heavily are not constrained by the physical and technical limits of their input systems.

Apple’s Front Row interface, however, needed to solve the problem of accessing the growing number of media types available for purchase or general consumption on Apple’s platforms. This included music, movies, TV shows, podcasts, and pictures. For all intents and purposes, the “widgets” which a user could choose between acted as access points, not for different media services or companies, but to discrete libraries for different types of media housed within Apple’s platforms. In other words, a user could toggle between text reading “Music,” “Movies,” “TV Shows,” and so on, while the correlating icon cycles along with the highlighted text to the left, in order to access the respective libraries. Upon accessing said libraries, the user would go through a similar process to select the actual file that they intended to stream or to play.

Leaving the Front Row and the Constraints of Widgets

Figure 4: tvOS Interface with “tiled widgets”

When Apple retired the Front Row interface in 2011 in favor of its OS X Lion for the Macintosh and tvOS for Apple TV (released the following year), it also abandoned the text-based widgets which defined the previous system. Instead, Apple opted to display software options through tiles of widget icons, with which the user interacted in essentially the same manner as he or she had grown accustomed to while using Front Row (i.e. using the remote to move vertically or horizontally, highlighting an icon to select along the way).

While this interface design afforded no new technical capacities for the Apple TV it signaled an important shift in the economics the product. As mentioned above, the Front Row interface was basically designed so that Apple users could organize and access a diverse number of media types all of which (or at least most of which) were under Apple’s umbrella. Of course, Apple could have simply kept the Front Row interface and added the products from new developers to the list of text which users could scroll through (i.e. Netflix, Spotify, and so on), but by choosing to design the interface with tiled widgets, which looked so similar to app icons on the ubiquitous iPhone, Apple subtly indicated a shift in their thinking about the Apple TV as a product. In other words, Apple TV was no longer a product which enabled consumption of media purchased through Apple’s platforms in the living room but was now a platform which facilitated the consumption of all digital and streaming content. From a developer’s point of view, this opened up the Apple TV from being an in-house Apple device to being one where products from a diverse number of developers could promote and distribute their products. However, this new focus of the Apple TV as a product did nothing to de-black box it from a consumer perspective. Meaning that even as Apple TV expanded the accessibility of its interface to non-Apple products which used Internet protocols to stream media, it still severely limited any access to the Internet writ large, particularly as opposed to browser designs for 10-foot UI’s. The interface constrains the user from accessing any aspects of the web other than those expressly designed to be accessed by the apps represented by the widget icons.

Conclusion

Apple TV proves an interesting case study as a smart TV technology not only because it serves as an archetypical example of the design problems all smart TV’s encounter – namely, the problem of distance and the problem of input – but also because of the interesting transition in this product’s specific history wherein it shifts from being designed as a product to mediate access to in-house Apple media, to being a meta-medium which facilitates access to non-Apple software and products. By considering the design problems encountered and overcome by the Apple TV, one discovers new answers to the old question, “why is this technology designed this way and not another way?” Particularly, Apple TV makes evident a unique symbiosis of the input technology and the interface design, where the interface accounts for the constraints of the input device.

Perhaps it is worth noting that the Apple TV product design only represents a single approach to reconciling television and personal (entertainment) computing, of which several product designs remain popular. Of course, there are other products on the market which solve these design problems in essentially the same way as Apple TV, such as Roku set-top boxes and the Amazon Fire TV Cube. Other products, however, like Google Chromecast, entirely sidestep design problems introduced by 10-foot UI and remote control input, and simply connects a personal computer to a TV, whereby the computer functionally acts as an input device and no new interfaces need to be designed. Comparatively, many companies now manufacture TVs which themselves possess computational power and can connect to the Internet. For smart TV products such as these, the design not only needs to accommodate for the limited user input of a remote and the 10-foot UI design, but also must consider how to seamlessly integrate the computational affordances of the smart TV with those of the cable TV system.

Considering the design of the Apple TV as primarily defined by input and interface problems, however, ultimately only considers a limited – albeit unique and important – set of principles guiding the design of this product and others like it. For example, it only deals with the combinatorial and modular design of Apple TV in an indirect and incomplete sense. And while an analysis which takes combinatorial and modular design into greater account might contribute a study which more fully de-black boxes this product, it would not necessarily address the unique design problems which we have explored above. By better understanding the input/interface system, the unique qualities of computers designed for video consumption on television monitors reveals an interesting moment of designing computational media both according to its own affordances and constraints, but also according to the affordances and constraints of input devices.

Bibliography

Braun, J. (2013). Going over the Top: Online Television Distribution as Sociotechnical System. Communication, Culture and Critique, 6(3), 432–458. https://doi.org/10.1111/cccr.12015

Chamberlain, D. (2010). Television Interfaces. Journal of Popular Film and Television, 38(2), 84–88. https://doi.org/10.1080/01956051.2010.483354

Cheng, J. (2007, March 27). Apple TV: An in-depth review. Retrieved December 14, 2019, from Ars Technica website: https://arstechnica.com/apple/reviews/2007/03/apple-tv-1st-generation.ars

David, P. A. (1985). Clio and the Economics of QWERTY. The American Economic Review, 75(2), 332–337.

Irvine, M. (n.d.). Introduction to Affordances, Constraints, and Interfaces.

Johnson, J. (1988). Mixing Humans and Non-Humans Together. 35, 298–310.

Lal, R. (2013). Digital Design Essentials: 100 Ways to Design Better Desktop, Web, and Mobile Interfaces. Rockport Publishers.

Moyer, M. (2009). The Everything TV. Scientific American, 301(5), 74–79. Retrieved from JSTOR.

Pegoraro, R. (2007). Apple Tries to Bridge Computer Desk, Living Room. Retrieved from http://www.washingtonpost.com/wp-dyn/content/article/2007/03/28/AR2007032802156.html

Simon, R., & Rose, B. (2010). Mixed-Up Confusion: Coming to Terms with the Television Experience in the Twenty-First Century. Journal of Popular Film and Television, 38(2), 52–53. https://doi.org/10.1080/01956051.2010.482907

Smith, R. A. (2001). Play-by-Play: Radio, Television, and Big-Time College Sport. Baltimore: John Hopkins University Press.

Sterile and Generative Systems

The introduction and first chapter of Jonathon Zittrain’s book, The Future of the Internet and How to Stop It, deal with questions of the design of the Internet as a sociotechnical system, particularly, as it has transitioned from a largely generative system to a sterile system. For Zittrain, a generative system describes a system which invites contribution, innovation, and involvement from the user, whereas a sterile system limits the role of the user to participate in pre-made processes with limited ability to be meaningfully involved with the process of creating in the application.

In some ways, the generative/sterile dichotomy harkens back to Donald Norman’s complaint about the misuse of the word “affordances” in design communities. In other words, the difference between a sterile system and a generative one has nothing to do with the technical capacities of the machine (i.e. its real affordances), but the perceived affordances, or even the affordances accessible to the user via the interface. Even in these introductory chapters, Zittrain provides a nuanced view of sterile and generative systems which is not entirely biased towards one or the other (although, based on the word choices, sterile and generative, I can guess where he ultimately would prefer to see the Internet go). However, Zittrain does not hesitate to point out that the generative design of the early Internet system, when maintained at nation wide scale, opens up opportunities for “bad code” to spread, identity theft to occur, and the general rise of malicious hacking.

Sterile systems, on the other hand, limit (although not totally stamp out) these pernicious practices. When the user makes less choices with software, they are less likely to engage with a “bad” software. And yet, even within the sterile system in which most of us operate, we are becoming increasingly aware that sterile systems are not utopian playgrounds, entirely safe for data. Zittrain wrote in 2009, so how could he foresee the Cambridge Analytica scandal? And while Facebook’s selling of data may not be entirely connected to its black-boxed sociotechnical practices, the public inability to understand the full implications of such practices certainly is.

In some ways, it seems that the logical end of Zittrain’s argument is a choice. Would you rather engage in a sociotechnical system which is more open for everybody, which simultaneously makes users more susceptible to malware as well as opportunities for innovation? Or, would you rather engage in a system to which innovation is closed off except for specialists, but also provides a different sort of safety against harmful software? It might be nice if even a significant portion of people would be inclined to the former, but I believe most would lean toward the latter.

Works cited:

Donald A. Norman, “Affordance, Conventions, and Design.” Interactions 6, no. 3 (May 1999): 38-43.

Jonathan Zittrain, The Future of the Internet–And How to Stop It. New Haven, CT: Yale University Press, 2009.

On the Internet or On the Interface?

People have always used prepositions as metaphors for the way they interact with media. We read through books, talk on the phone, and look at pictures. And while none of these metaphors entirely capture the experience of interacting with these human artefacts, none miss the mark quite so bad as claiming to get on the Internet. However, even though the preposition fails to offer much help by way of an accurate description, just as much confusion arises from the object to which it refers: the Internet.

Of course, in some ways, we need some sort of nominal indicator so that we can refer to this network-of-networks in some workable way during casual conversation. Unfortunately, the tendency of this sort of conversation tends to regress into reifications of “the Internet” which further black-box this human artefact and only stand in the way of understanding what actually occurs when going “on the Internet.”

Essentially, when one claims to go on the Internet, he or she refers to accessing an interface (such as a web browser) designed to send and receive small bundles of data called internet packets according to pre-established protocol which are asynchronously sent across computing machines called routers. What the Internet-user in 2019 experiences as “getting on the Internet” is really the back-end of this complex (albeit almost instantaneous) process: the arrangement of packets received from routers by the interface. In a certain sense, it might be more accurate for the digital citizen to talk about getting on the interface instead of getting on the Internet. In other words, the Internet is not the thing that one uses, but the process of distributing information and data which become manipulatable through interface design.

Aside from de-black-boxing the online world for the average user, understanding the Internet as a process of distributing information across a network of computing technologies can help the scholar or designer understand more specifically their object of study. In other words, one can examine the design of an interface, the design of the data/information, the design of the network, the design of the protocol, or the design of the physical, computing machines. Not to mention studying the history, effects, politics, and economics of any of these things. To study the design, history, or effects of the Internet is a massive undertaking which would probably take several lifetimes. However, by understanding the Internet as a complex set of practices, media scholars can achieve rigorous and meaningful conclusions about the role of the Internet in the modern world by focusing on particular moments in this process.

Works Cited:

Martin Irvine, The Internet: Design Principles and Extensible Futures

Denning and Martell, Great Principles of Computing, Chap. 11, “Networking.”

The Affordance of X/Y Coordinates

From early cave paintings to ancient writings through television and computational media, humanity continually demonstrates a proclivity to abstract our symbolic capacities onto two-dimensional substrates. The application of this insight by Doug Engelbart and others laid the foundation for any number of advances in interface design, from the two-dimensional arrangement of pixels in graphical user interface, to the x/y coordinates that a technology like a mouse and cursor depends upon, finally to touchscreen design.

The principle, however, that computational interfaces in their current iteration consist of screens constructed of rows of “picture elements,” or pixels, which respond to programmed signals which indicate color, darkness, and position easily recedes from the forefront of conscious thought when using even the most basic functioning interfaces. The WordPress interface used for this course, for example, functions primarily as a text input program. However, as students write their weekly reflections, one can only assume that it is only upon the rarest occasion that they consider how every keystroke signals to the computer and the website interface, triggering a complex sequence of commands which finally result in “lighting up” the set of pixels correlated with the letter-form which they typed. Conversely, when one presses the backspace key, they do not really “erase” the previous character, they simply trigger the command which communicates that the correlating pixels ought to return to the previous state of the background color (white in this case). The same goes for functions like bold, italics, text alignments, and even hyperlinks. While other affordances may also be triggered by something like hyperlinks, the immediate, human-perceptible change (i.e. black text to blue, underlined text) is nothing more than a sequence of commands correlating to tiny elements of light arranged on an x/y axis.

The difficulty in interface design, to which I find myself continually returning, is that as people become more and more accustomed to the abstracted designs of an interface, the less and less they have to think about the black-boxed system of physical phenomena which ultimately underlay the entire process. Certainly, in cases like pixels, which only concerns the most basic, and fundamental level of an interface design, the stakes are relatively low. However, in many cases, designed interfaces which hide the physicality of computing only leads to confusion about the limits of computational media and promotes a perception that the digital is the same as the magical.

Works cited:

Martin Irvine, From Cognitive Interfaces to Interaction Designs with Touch Screens

Design, Remediation, and Semiotics

Through the course of this week’s readings, my mind kept returning to two distinct, but related, questions. First, does the intentionality of figures like Alan Kay, Doug Englebart, and others in designing computing systems to remediate “real world” commonplaces such as the desktop exempt them from the pattern of media theory, first observed by Marshall McLuhan, that “the content of the new medium is the old medium?” Second, are computational systems, and particularly commercial computational systems, unique from all other human artifacts in that the remediating action always involves symbols remediating other symbols and not only stylistic imitations (like Manovich describes with Gutenberg Bible’s imitating manuscripts or cinema imitating the theater)?

Regarding the first question, several of the readings for this week, including the Manovich and Moggridge excerpts discussed the process of making cultural computing viable through the use of iconic remediations like the desktop metaphor and the GUI display. These histories also emphasized the role of their inventors in providing these breakthroughs in human-centered design for computers. And while certainly these innovations ought not be trivialized or overlooked, they seem to carry with them a fact that is simultaneously  self-evident and ontologically limiting; that computers remediate the way they do because they were designed to do so. On a certain level, this is irrefutably true. The actualized remediations that the digital citizen interacts with every day owe the debt of their existence to Stu Card, Larry Tesler, Doug Englebart, and so on. But none of these figures invented remediation for computational media. In fact, the problem with formulating these innovations in such a way so as to equate them with remediation is that it obscures the fact that even before computers enlisted the help of icons for mainstream acceptance and use, computers were still remediating something. Isn’t the digital substrate itself a remediation of boolean logic? Doesn’t command line interface remediate the syllogism? The point I am trying to make is that it seems unfair to even unintentionally imply that these men initiated remediation in the history of computing. They simply initiated the version of remediation which is actualized and recognizable to the mainstream computer user.

This leads me to my second question. There are, as noted in the cases of boolean logic and command line interface, cases where existing phenomena seem to be remediated by a computer in the same way as, say, the theater was remediated by filmmaking. However, a greater number of remediated objects on the computer are required to first go through a process of being represented by other symbols and finally reconstructed (as is the case with icons, GUI, etc). This seems categorically different than what traditionally occurred in remediation. In the case of the cinema or the printed book, the cultural practices of the old medium influenced the cultural practices of the new medium, which could then continue these practices or change them as a society saw fit. In the case of digital iconicity, however, the old medium is semiotically constructed so as to communicate something to the user. In other words, its old cultural practices are insignificant and the representation of the old medium is only as significant as a device of communication to the user. Hence skeuomorphs like the floppy disk in the save icon, or a postage stamp for email. The differences between these two types of remediation, in our reading, seems implicit, but perhaps needs further working out.

Works Cited:

Lev Manovich (2013) Software Takes Command, New York, NY: Bloomsbury

Marshall McLuhan (1964) Understanding Media. Cambridge, MA: The MIT Press

Bill Moggridge, ed. (2007) Designing Interactions. Cambridge, MA: The MIT Press

Code: Writing for Humans and Machines

There’s a level of irony that even for our most abstract symbol systems, on the level of human interaction, language persists. In other words, even though a computer might abstract a line of Python code a level further into a series of 1’s and 0’s, the programming language has been designed to accommodate for the human proclivity to deal with information through computable symbols, which is to say, with words. Of course, there are a number of differences between natural language and computer code, but the extreme formalism of code seems the most immediately obvious.

Of course, any computer code needs to be designed in such a way that the human “writer” can input instructions to the machine without ambiguity. Furthermore, the code must be designed in such a way that accommodates for the material limits of computer design. In other words, the code must account for the key insight of information theory (that the message does not matter) and must account for the material process of reproducing a message at a different point, which, in our computing devices as they exist today, means being designed according to the limits of boolean logic.

Code exists in this liminal space: it must be meaningful to the coder and meaningless to the computer (in the information theory sense of meaninglessness). In this way, it must account for the semiotic possibilities of human cognitive faculties, as well as the material limits of the computer. Because of this bi-directionality, computer code differs from perhaps all other writing practices in the course of history in that, on some level, it needs to contain both symbols that say and symbols that do.

It has become the practice of some avant-garde artists, particularly in the earlier days of personal computing, to exploit this very affordance of programmable media by constructing highly formalized poems which can both be read by a human reader as well as run as an operational program. One such attempt at this bi-directional code/poetry is shown in an excerpt here by literary artist/theorist, John Cayley.

Even if the human side of this poem is awkward and constrained by the limits required for maintaining its status as working code, it proves an interesting point about the design of programmed languages: that they always have to account for the semiotic processes of both the man and the machine.

Works cited:

Cayley, John (2002). “The Code is Not the Text (Unless it is the Text)”. Electronic Book Review.

Wing, Jeanette (2006) “Computational Thinking.” Communications of the ACM 49, no. 3: 33–35.

Entropy and Meaning

Claude Shannon wrote that “the fundamental problem of communication is that of reproducing at one point either exactly or approximately the same message selected at another point.” Perhaps a more humanistic rephrasing would sound something like, assuming interpreting actors in two or more locations understand a perceptible artifact or series of perceptible artifacts to correlate to the same imperceptible and cognitively generated meanings, the fundamental problem of communication only requires reproducing, at one or more points, the perceptible artifact.

For Shannon, the journey of “information” from point A to point B does not require for the perceptible artifact to remain the same the entire time — it can be added to or taken away from, it can be reorganized and jumbled, sent in part or in whole — as long as by the time it arrives at its destination, the perceptible artifact returns to either “exactly or approximately” the same form as when it left its sender. Shannon’s theory, and the Information Theory which grows out of it, turn communication on its head; it only accounts for what is said (broadly speaking), not what is meant.

It would be easy here to disregard meaning and meaningfulness from Information Theory entirely. However, important principles rely on an assumption of meaningfulness as crucial parts of the process of reproducing a message at two different points. In particular, the insights gleaned from entropy and redundancy depend on the meaningfulness of a system either to remove redundancy for purposes of efficiency or to add it in order to ensure the integrity of a transmission.  In other words, Shannon realized that human meaning systems were patterned and therefore predictable. A highly patterned message, which tended away from randomness (i.e. exhibited low entropy) carries with it a greater degree of redundancy, and therefore can be probabilistically predicted with greater accuracy. On the other hand, a message with high entropy, or a great degree of randomness,  provides new information with every bit, making it more difficult to accurately predict.

In this way, for Shannon, determining entropy depends on meaningfulness, not in terms of the actual Peircean object of a bit of information (or what we tend to think of as “content”), but because by assuming meaningfulness, one can assume a pattern, or a level of redundancy, which means the system does not tend toward randomness, and the probabilistic likelihood of any bit of information can be discerned according to known bits of information. Of course, while an insight like this can provide remarkable understanding into modern technologies, such as autocorrect, speech-to-text software, and even machine learning, this assumed “meaningfulness” still disregards “meaning” itself. In other words, even if we program our machines to look for redundancies because our meaning systems are full of them, they still cannot know the Peircean “object” of the physical artifact, which, no matter the extent of mathematical fancy footwork, requires a cognitive agent to be truly understood.

References:

Martin Irvine, Introduction to the Technical Theory of Information

James Gleick, The Information: A History, a Theory, a Flood. (New York, NY: Pantheon, 2011).

Affordances: Potential and Actual

As commonplace as discourse on “affordances” has become in design theory and media theory has become, I deeply sympathize with Donald Norman’s attempts at correcting their rampant misuse and providing greater specificity as to what is meant by an affordance or a constraint. I disagree with Norman, however, when he implies at the end of his article, that discourse on affordances and constraints might be better suited if left to “physical objects, real knobs, sliders, and buttons” (p. 42). In fact, given Janet Murray’s charge to consider all computational media as a single medium, I think the application of affordances to digital media demonstrates the philosophical richness of the concept in a way that retrieves and updates the foundational concepts of potentiality and actuality.

Potentiality has to do with the possibility of an occurrence, whereas actuality has to do with the reality of an occurrence. In terms of affordances, JJ Gibson’s original concept shared much more similarities with potentiality than actuality. A physical book, for example, could afford anything which its properties would feasibly allow, including expected features such as reading, dog-earing, page turning, but also including absurdities such as throwing, sitting-on, or even eating. For Gibson, affordances have to do with the limits of potentiality for an object — what can be done with it, not what is done with it.

Concerned with design, Norman talks about perceived affordances in terms of communicating the proper use of an object through design choices, and not about the total scope of potentiality of an object. In other words, Norman talks about holding the user’s hand in leading them through to actualized affordances as useful for the user in distinguishing the most productive use of an object as opposed to the total scope of its potential affordances. For example, e-readers, like the early Kindle designs encourage the actualization of a “turning the page” affordance by the design of buttons with arrows pointing in either direction. This design operates as a sort of communication to the user that the device contains this potential affordance, for which it is designed to be actualized. For Norman, inserting these buttons did not “create the affordance.” The affordance existed independently of the buttons as a function of the computational design of the Kindle. The buttons merely work as a tool for the user to actualize the potential affordance which was there all along.

For examples like the book, or even the Kindle in some respects, this delineation seems rather tedious. However, if we take literally Janet Murray’s claim that all computational media are one medium, it becomes immediately much more useful. Murray claims that all digital media is “created by exploiting the representational power of the computer” (p. 8). In other words, she is stating that all digital media are defined by the set of affordances associated with the capacity to interpret and represent electrical impulses in a way understandable to human users (this definition is overly broad because it doesn’t account for differences in input like keyboards or touchscreens, but it works for our purposes here). Therefore, all computational media possess the potential affordances to do any number of actions associated with this process: scroll through a webpage, click a link, etc. The role of the designer, particularly the designer of computational software, is to guide the user to actualizing the limited set of useful potential affordances available in a particular program. As Norman says, “The affordance exists independently of what is visible on the screen” (p. 40), it is the designers role to make it visible or obvious to the user so they might actualize it.

Works Cited:

Murray, J. (2012). Inventing the Medium: Principles of Interaction Design as a Cultural Practice. Cambridge, MA: MIT Press.

Norman, D. (1999).  “Affordance, Conventions, and Design.” Interactions 6, no. 3: 38-43.

Extending the Speed Bump Metaphor

One question which technologists, designers, and philosophers of technology repeatedly return to asks, “is the way a technology achieves an end significant for those interacting with it?” Both Latour and Vermaas et al. reference the moral imposition of the speed bump (which demands compliance to the law because of the ways it interacts with the suspension system of a car) as opposed to the traffic officer holding a sign reading “slow down for students” (which politely asks compliance on moral or humanistic grounds).

I’m interested in the question of the role of different processes for achieving the same effect in terms of what it implies about technologies as agents of change and how extending the speed bump metaphor helps for understanding why we no longer need the social/technical divide. It seems to me that a technology might contribute to change in the world in one of two ways — through physical rearrangement or through changes in performative or procedural roles. Latour says so much in Reassembling the Social, although he calls the first ostensive change. What Latour, Vermaas, Debray, and others intend to achieve through an understanding of technologies as actors is not deterministic in scope, but instead has to do with the imagined divide between technology and society. Technology and society cannot be divided because they always already inform one another. The never-fully-ostensive performative class of students engender the technical (ostensive) reality of the speed bump which, in turn, results in the ostensive slowing down, even though the speed bump is only reinforcing the pre-existing performative value of “speed limit.” The point in destroying the divide between the social and the technical is not to say that there is no difference at all between performative roles and values (which we often think of as social) and ostensive artifacts (which we often think of as technical), but instead to recognize that both construct our political ecology — to use Latour’s phrase — in a way where they constantly overlap and cause change in one another.

Given the reason for eliminating the social and the technical divide, I wonder if Latour’s dichotomy of human and non-human actors effectively communicates the new definition of social which he promotes. In other words, while he certainly takes a great stride forward by establishing the role of non-human actors as crucial to understanding sociotechnical systems, this vocabulary fails to demonstrate clearly how the new social includes both abstractions which can only be performed, as well as manifestations which can be clearly seen, touched, or heard. The point is not only that non-humans act, in addition to humans, which was already assumed, but that the social world is not limited to that of abstract performances and also includes technological instances of “society made durable.” Perhaps in design thinking, it would be helpful to consider the “ostensive relations” as well as the “performative actions” an artifact might produce.

Works Cited:

Debray, R. (1999)  “What is Mediology?”, from Le Monde Diplomatique, Trans. Martin Irvine.

Latour, B. (1999) Pandora’s Hope: Essays on the Reality of Science Studies. Cambridge, MA: Harvard University Press

Latour, B. (2005). Reassembling the Social: An Introduction to Actor-Network-Theory. Oxford: Oxford University Press.
Vermaas P., Peter Kroes, Ibo van de Poel, Maarten Franssen, and Wybo Houkes. A Philosophy of Technology: From Technical Artefacts to Sociotechnical Systems. San Rafael, CA: Morgan & Claypool Publishers, 2011

Cognitive Artifacts and Pre-Inscription

Donald Norman’s delineation between the personal and the system view of a cognitive artifact provokes some interesting questions which may be helpful in further consideration of design thinking. The most interesting of these questions, however, may be his assertion that a cognitive artifact might only be conceived of as “extending cognition” from the system view. From the personal view, Norman argues, the artifact does not extend cognition, it merely changes the task by delegating the original task to the object and requiring a new task of the user. By focusing on the latter insight, I believe Norman implies some interesting design insights.

First and foremost, it harkens back to a similar concept which Bruno Latour (under the pseudonym Jim Johnson) introduced several years prior to Norman’s publication: pre-inscription. Latour situates this concept among a number of others which describe the relationships of humans and objects, but pre-inscription in particular has to do with the idea that all cognitive artifacts require some level of learning or work before they can be used by human actors. This pre-inscription can be nearly invisible (like the learning needed for use of a hammer) — we call this intuitive design — or it can be entirely visible and require a great deal of concentration (like the learning needed to understand writing). When it is visible, because the user must focus on the pre-inscription, the user is much more aware of the fact that, as Norman would say, the task has been changed, or as Latour would say, the action has been delegated.

It seems that with a combined view of Latour and Norman, we can understand how complex computational media ought to be designed in order to achieve maximum efficiency on the systems and the personal level. Most simply, successful design of computational media most certainly need to take into account the burden of pre-inscription as well as the actually efficacy of the artifact in relation to the task for which it was designed. In other words, if the designer does not ask, “what must the user now learn to do as a result of the design of this artifact,” he or she runs the risk of designing a product which requires more learning by pre-inscription than toil otherwise required by actual action. I think of a product like Microsoft Excel, which, while extremely powerful, requires a great amount of pre-inscription. For most people, the amount of learning required to master Excel dwarfs the actual amount of time they spend on Excel (or spend googling how to achieve certain functions or asking colleagues the same questions). The way the task is changed via the cognitive artifact requires more exertion from the user than the unaltered task of data entry into cells, so people only use that capacity. Most simply, the dichotomy Norman draws here implies that design of cognitive artifacts are most effective when the exerted effort required by the new task is significantly less than the effort required without the artifact, either in a single instance, or as made evident over time. For some artifacts, like Excel, this imbalance is necessary. However, for those concerned with user-centered design, it ought to be one of their foremost considerations.

Sources:

Norman, D. (1991) “Cognitive Artifacts.” In Designing Interaction, edited by John M. Carroll, 17-38. New York, NY: Cambridge University Press

Johnson, J. (1988). Mixing Humans and Non-Humans Together. Social Problems 35, 298–310.

Modularity and Convergence

It seems that that the stories of modularity in design and convergent designs share a great deal of similarities and provide a fertile ground for pushing the concept of modularity to its conceptual limits. Baldwin and Clark might sum up modularity by the maxim, “independent but interdependent.” In other words, just as in the story of Hora and Tempus, the parts do not depend on one another in a cumulative, linearly ordered fashion, per say, but instead, are each constructed as interchangeable and self-contained systems which then construct the larger system of the final design. Convergence, on the other hand, involves taking pre-existing systems and designs, and incorporating them into different (either new or pre-existing) systems. The modern car might be thought of as a commonplace example which contains both modularity and convergence — although Langlois is quick to point out that the original Model T was not exactly the paragon of modularity. I find the engine of a car to perfectly typify the principle of modularity, as this sub-system is by all accounts entirely essential for the functionality of the car, while the engine “module” still might be upgraded or downgraded within the designed system of the car. Even if the car in toto fails to be rigorously identified as a modular system, an engine certainly can be thought of as a modular component as its independence from and interdependence to the system define the nature of its design at large.

The car also demonstrates the principle of convergence in, say, the installation of the radio on the dashboard. Unless you drive a vehicle designed by some sort of technomasochist, your car probably still works when the radio cuts out — in other words, the radio and the car are not interdependent in the same way as the car and the engine. But the radio still seems to possess some modular qualities. For example, the car radio is specifically designed for a car, meaning that a car radio and a “non-car” radio are not quite the same thing. They may both serve the same end function of channelling signals cast over radio waves, but one is designed so as to connect with the speaker system on the interior of an automobile while the other is not (among other ways which exemplify the imbrication of the design of the car and the car radio). In this way, there seems to be a level of interdependency between the car radio and the car, albeit in one direction only — from the car radio to the car. To complicate matters further, car radios, like modules, might be upgraded or downgraded and therefore frequently subscribe to the same design specifications so as to make them interchangeable. What this also implies, however, is that the specifications of different car models themselves be designed according to even banal specifications such as depth and width so that the radio, even while non-essential to the car as a designed system, might achieve a level of pseudo-modularity.

As more and more technologies begin to be designed according to the principle of convergence, to what degree can designers or design theorists consider convergence to be different from modularity? Does modularity require interdependence from both the module and the system, or might it only require interdependence in a single direction? In other words, must a feature or function be essential to a designed system in order for it to be a module? Of course, the case of the car and car radio is a relatively simple example, however, it seems likely that this ambiguity remains relevant in more complex examples, including software design. Ultimately, I’d like to know whether the difference between modularity and convergence has to do with anything other than architectural structure (modularity) and utility (convergence). Is the difference between the two anything more than superficial?

Works Cited:

Baldwin, C. and Clark K. (2000) Design Rules, Vol. 1: The Power of Modularity. Cambridge, MA: The MIT Press

Langlois, R. (2002) “Modularity in Technology and Organization.” Journal of Economic Behavior & Organization 49, no. 1: 19-37.

Amazon Books and Storefront Design

Perhaps no company has mastered design as a product of systems thinking in the era of the Internet better than Jeff Bezos’ Amazon. Of course, Apple might hold equal rights to lay claim to this title, but more so than Apple, Amazon seamlessly integrates the ostensible world of material goods — and the social and economic practices which come with it — to the seemingly ethereal world hidden behind the screen. In other words, Amazon’s designed system incorporates interface design (and the programming implied therein), logistical design (both as an intermediary in shipping and between producer and consumer), and with the advent of products like Alexa or Echo, product and technology design. However, the aspect of Amazon’s design prowess which interests me most has nothing to do with any of this ecosystem of design, but with a small little storefront on M Street: the Amazon Bookstore.

In setting out to leave a footprint other than digital, Amazon faced the inverse design situation of most companies which pre-dated the turn of the century. Upon the advent of the world wide web, many companies asked questions about integrating their new online domain into their existing system or how to represent their physical manifestations on a new online interface. Amazon, however, existing originally as an digital company needed to design a method for translating the world of bits and bites into that of ostensible atoms.

In doing so, they designed a brilliant format for this storefront, which incorporates many affordances of digital media into the physical layout of the store. For example, the store foregoes the tradition of arranging books on shelves so that the binding faces outward — itself a design of another era of codices and readership — for the less spatially economical placement of books with the cover facing outward. As demonstrated below, this format retrieves the visual representations of books on an online platform or digital interface.

But perhaps this is simply a coincidence — not a choice in designing a system, but an appeal to the more banal impulse of the masses for visual imagery. It seems unlikely, however, in light of several other prominent design choices in the layout of this store. For instance, the choice to place reviews (all favorable, unsurprisingly) below the books on display. Of course, this calls to mind the affordances of interactivity latent online and the practice of reviewing products on Amazon’s online platform. My personal favorite aspect of the design of this store, however, is the prominent wall of books which display a sign noting “If you like this… you might like…” as shown below.

This flesh and blood retrieval of algorithmic processes brilliantly conveys the line Amazon walks between the designs of online interface and the affordances (or perhaps as they might see it, constraints) of legacy medias such as print.

I’m less interested in this design as any sort of ingenious breakthrough — while creative, it likely could be dreamed up by a number of people with a creative inclination — and more so in terms of what it means for design as a rhetorical device. While Norman is concerned with design and usability (a valid concern), design can also be used to send affective messages within systems and otherwise. Particularly when pushing one system to mimic the affordances and constraints of another, designs can carry messages and promote effects beyond even the scope of usability. I think Norman might even go so far to say that the real trick in the design of this storefront is in creating a system which is not only user-centered, but rhetorically clever and aesthetically pleasing.

Bibliography

Donald A. Norman, The Design of Everyday Things. 2nd ed. New York, NY: Basic Books, 2002.