Yutong Zhang

Abstract 

AR, which refers to the augmented reality, is a technology that can layer virtual content over reality. In recent years, AR has become an artistic choice for art museums. This paper briefly introduces AR and its relations with museums and the interpretation of artworks. By analyzing three AR projects in different museums (Invasive Species in The Pérez Art Museum Miami, ReBlink in The Art Gallery of Ontario, Hacking the Heist in Isabella Stewart Gardner Museum), AR’s affordance in art museums is discussed. In the end, this paper also discusses the limitations of AR and possible problems for other art museums to implement AR in art exhibitions.

Introduction

Mobile devices are omnipresent in modern society. Realized mobile phones and social media can help exemplify the influence of art and museums, museums, for example, the Metropolitan Museum of Art, gradually stopped entreating their visitors not to use their cellphones (Gilbert, 2016) After accepting visitors’ using mobile phones, museums started to think of a way of taking advantages of mobile devices as media, hoping to provide visitors with more information on the exhibitions and improves the traditional museum visiting experience. On the other hand, there is also a trend of digitalization among modern artists. Digitalization of art does not limit in using photoshop as a painting technique. Using different media and interacting with various technologies, digital artworks are transdisciplinary (Adams, Arisona & Gibson, 2008).

In such a background, augmented reality has come to the eyes of curators and artists. Usually, a museum may be sponsored by a foundation and then start an AR project. By collaborating with artists and technical teams, the museum can eventually launch the AR exhibition. The most commonly used way of applying AR technology in museums is to let visitors download an app specifically applicable to the exhibition on their mobile devices. Leveraging cameras, scanning technology, GPS and other technologies, AR tools augment the reality, add different layers of information, and strengthen the interaction in museums.

In the perspective of creating meaning and interpretation, AR technology also plays an important role. Because the applications of AR tools in different art museums are different, the analysis on how AR technology specifies the meaning of the artworks also cannot be generalized.

This paper picked out three AR projects in different museums:  Invasive Species The Pérez Art Museum Miami, ReBlink in The Art Gallery of Ontario, Hacking the Heist in Isabella Stewart Gardner Museum. Based on specific forms and purposes of digital artworks, I analyze how AR tools as an art media are applied properly and successfully in three exhibitions.

Literature Review

• What is AR

According to the definition given by Encyclopedia Britannnica (2013), augmented is “a process of combining or ‘augmenting’ video or photographic displays by overlaying the images with useful computer-generated data.” People usually are confused about the difference between AR and VR. AR, which refers to Augmented Reality, just as its name, can “augment” the reality with perceptual information generated from digital data, while VR (Virtual Reality) creates artificial environments to replace the reality with virtual views. Because AR technology provides different layers of information based on the real components in our physical world, it can alter people’s perception of the real world and simultaneously provides people with an immersive experience.

AR technology is a blackbox, in which there are more blackboxes of other technologies. In the book “Understanding Augmented Reality” (Craig, 2017), the author explains that all AR systems consist of at least three hardware components: sensor, processor, and display. Those are the most basic components and indispensable for the augmented reality system. Each of the components can take different forms but have similar functions. The sensor provides information for the system. The information provided usually is one of the followings: location, orientation, lightness or darkness, etc. The processor is like the brain of the system, in charge of receiving signals from the sensor, execute the instructions from the programming and then creates signals to drive the display. The display provides a signal for our sense. The most commonly used are visual displays, but there are also audio displays, haptic displays, and other sensory displays.

The popularity of AR technology in recent years probably benefitted from Pokémon Go, which is an augmented reality mobile game developed by Niantic and swept the world since 2016. However, AR is not a new-born technology. It was invented in the early 1990s. The commercial use of AR was mostly focused on the entertainment and game businesses. People are also increasingly aware of AR’s potential for applications in various scenes.

• AR and Museums

Ding (2017) concluded in her article that the global trends of museums in recent years are digitally “mediated personalization” and “personalized learning”. Trendwatch (2015) released a report in 2015, showing that at that time museums have already thought of proper ways to take advantage of wearable devices. The hand-held audio guide has become a mainstream way of providing with extra information on the exhibitions for visitors. To enhance the interaction experience in museums, curators were thinking of utilizing AR technology in a portable way. Usually through mobile apps developed by museums, visitors who hold mobile phones or tablets are able to have a more immersive and innovative experience by using AR technology.

The advantages of implementing AR in museum visiting experience have been praised by many experts. Most of the opinions are from the perspective of diverse information and interaction. AR tools make it possible that surrounding spaces in the museums become endless layers of information through the screen of mobile devices (Ding, 2017). Besides, AR tools can contribute to more engagement in museums because of its creative feature. Enabling visitors to explore the displayed artworks by themselves through AR apps actually arouses people’s interest and strengthens the interaction between the museum and its visitors. The use of AR can also enrich the experience in museums visiting because it is useful for stimulating the other possible works in progress comparing them with the artworks (Nofal, 2013). What’s more, AR could also bring exhibitions to life, make the figures or animals in the artworks move and provide visitors with a new perspective of interpretation.

• AR and Interpretation of Artworks

Museum visitors include both native and experienced viewers. They may have different types of appreciation of artworks, but, the common interpretation of artworks is usually the extension of everyday perception and is limited by one’s own knowledge construction (Cupchik &Gebotys, 1988), As Bartlett (1932) also mentioned, the effort after meaning in ordinary perception is as an “urge to perceive something in terms of a wider background of past experience (p.192).” The trained and experienced visitors have a wider scope of knowledge on art interpretation methods before coming to the exhibition, thus, they may have more qualitative art museum visiting experiences than naive visitors. In this aspect, providing additional and accessible information to enlarge the scope of naive visitors’ knowledge of the artworks’ background in the museum is a possible way of enhancing their appreciation experience in museums.

As early as in 2012, according to the 2012 Mobile in Museum Study, 1% museums in the United States have started with AR and introduce mobile devices into museum visiting experience (Ding, 2017), because museums have noticed that AR can provide hybrid layers of information and images, which can be used to provide visitors with more background knowledge and improve their level of appreciation or artworks.

On the other hand, an artwork as a meaning system reflects its artist’s thoughts and intentions. An artist’s thoughts and intentions are after all influenced by the social context. One of the factors in the complicated social context is technology. The development of technology brings more advanced and innovative techniques for artists.  In the process of meaning creation of an artwork, the specific techniques, also influence its meaning. Choosing AR tools as a specific technique may contribute to creating the intended meaning in artworks and further influence the visitors’ interpretation of them.

Case Studies

• Invasive Species in The Pérez Art Museum Miami (PAMM)

The Pérez Art Museum Miami (PAMM) coordinated with Felice Grodin, the local-based artist in 2017 and held an AR exhibition named “Invasive Species”. This exhibition includes four site-specific digital works that virtually interact with the architecture and outdoor spaces in PAMM. In order to see the artworks, visitors have to download PAMM app in iOS devices. Though the screen of iOS devices, people can see the digital artworks interact with the physical background in the museums.

A Video Guide: PAMM AR – Felice Grodin: Invasive Species

The main concept of this digital artwork, according to Felice Grodin, is to evoke people’s awareness of the fragility of our ecosystem (The Pérez Art Museum Miami, 2017). One of the artworks, Terrafish (2017-2018), like a woven net of bright pink neon lights with pulsating stalks reaching up, is aimed to indicate the invasive jellyfish found in South Florida waters. Terrafish is virtually spread over the floor at the entrance of the museum. The digital pulsating stalks reach up towards the cell of the museum, interacting with the 45-foot-long hanging gardens designed by French botanist Patrick Blanc in collaboration with the architects of PAMM, Herzog &de Meuron, which gives people the feeling that the unknown creatures are foraging and devastate the plants. The stalks are 49 feet tall and the pink base is about 100 feet wide (Mortice, 2018).

Another piece of the digital artworks has a three-dimensional creepy shape in a purple-blue-like color, standing on the terrace of the museum. It has many tentacles and legs. When visitors touch the screen, they can see the tentacles and legs of this unknown creature twitch as if it is crawling. It is also huge with similar size of a van (Mortice, 2018).

In the promotional video (Cuseum, 2017) released by PAMM, visitors were led by guides, holding iPad, excited about exploring the virtually existed invasive species. They continuously compare the physical background in reality with the virtual scene seen through the screen and marveled at the innovative way of digitalization of artworks. They also take photos of the invisible artworks through PAMM app.

The AR tools enable more possibilities for open space in a museum.  In this case, the open space in PAMM is apparently better utilized with AR artworks virtually overlapped with the physical museum background. Benefited from the AR artworks, the open space in PAMM has a better way of interacting with its visitors than just being a part of the architecture and being a resting place for visitors. With Grodin’s Invasive Species “hidden” in that open space in PAMM, visitors slow down their steps and carefully explore the open space that was neglected by them in the past. By virtually exhibiting digital artworks, the original museum space has been developed and gained more depth regarding the ability to exhibit artworks and strengthen the interaction between the museum and its visitors.

AR technology used in art museum may influence the interpretation of artworks in both positive and negative ways. On the one hand, AR tools provide visitors with a new context for interpretation. Because the AR-based digital artworks Invasive Species can only be seen through the screen of iOS devices, visitors in PAMM have to change from seeing the exhibition directly in front of their face to relying on handheld devices. The screen, as an interface of small size, limits the view. Thus, the screen also becomes a frame of the artworks, while the size of the screen may decide how large the digital artwork is perceived and how many details visitors can appreciate.  In this way, AR tools influence people’s interpretation of artworks in a negative way. However, Though the interface (screen), a new context is created. In addition to visual satisfaction, visitors can also have an immersive experience in virility and reality. In this case, if visitors hold iOS devices and move toward different directions, they may feel that they are embraced by the neon pink, honeycomb-like net of Terrafish. The artwork Terrafish itself becomes a context for visitors as a precondition of immersive experience. The innovative way of experiencing artworks in a context that is virtually constructed by the artwork itself can help visitors have a different perspective for the interpretation.

PAMM collaborated with Knight Foundation, which provided a grant, was planning an AR project. Grodin was interested in that project and after discussing with the curators from PAMM, she started to develop the AR project from her earlier drawings. AR technology also takes part in the process of creating the meaning of Grodin’s Invasive Species. She believes that drawings can “migrate between analog and digital” (Pérez Art Museum Miami, 2018). With the digitalization of her earlier painting by using AR technology, her new artworks now engage very actively with the architecture and respond strongly to the changing ecological environment in Miami. Grodin indicates that her new work is now not only “grow” from two-dimensional to three-dimensional but also in real-time. Those changes are perfectly in accordance with her purpose of showing the possible future due to the ecological change through the artworks. In this way, AR has been considered not only a technology but also an art medium, through which the meaning of the artworks can be better expressed by artists.

• ReBlink in The Art Gallery of Ontario

From July 6, 2017 to April 8, 2018, there was a special exhibition, Reblink, in The Art Gallery of Ontario (AGO).  It was worked by a digital artist, Alex Mayhew, who reimage historical artworks based on AR installation. After downloading the custom app Reblink onto personal mobile devices, people have to authorize the app to use the camera and then scan the historical artworks from the AGO’s Canadian and European Collections. The painting’s subjects come alive on the screen in a way of “reflecting on our daily reality in the 21st century.” (Art Gallery of Ontario, 2017). ReBlink’s highlight is that it vividly indicates the contrast between history and 21st-century life. By using AR, figures and subjects in the historical paintings are able to be transported across the limitation of historical context and presented in a modern way. The exhibition itself is benefited from the new technology, and meanwhile, it also reflects on social changes due to the development of technology.

 

For example, in the original painting Drawing Lots by George Agnew Reid, there are three figures closely sitting or lying on the stomach, whose head closely bending over their game (Impossible Things, 2017). In Mayhew’s reimagined version, those three figures are all absorbed in their phone’s screen and separately sitting with each other. The implication is obvious:  the distance between each individual is further than it is in the past, and face-to-face communication among people is harder due to the popularity of mobile devices.

For another example, Mayhew also reimaged the portrait Marchesa Casati originally painted by Augustus John in 1919. This portrait depicts the flamboyant Marchesa with her fiery red fiery hair contrasted with a muted background of mountains. Marchesa in the portrait is looking toward the possible visitors of the painting with smiling eyes. However, in Mayhew’s version, Marchesa is holding up a selfie stick, smiling at the camera, and immersed herself in posing and taking selfies. She no longer has eye contact with the possible visitors, because she is now looking at the camera of her mobile phone. Mayhew’s version again implies that the development of technology has changed our behaviors. With cameras embedded in mobile phones, taking selfies becomes more and more popular. One’s portrait is no longer drew by a painter, or taken by a photographer but taken by himself or herself.

This is also the case that AR technology participates in creating meaning for the artworks. When Mayhew was asked about his inspiration and purpose of reimaging the historic masterpieces from a modern perspective, he mentioned that using AR tools is aimed to get people “to look up, rather than look down” (Coates, 2018). Many technologies become a type of distraction for people. As a result of the fast development of many technologies, people consume information faster than they did in the past, including art pieces. But he wants to encourage people to slow down their steps and explore more about the artworks in the museums.

By using AR technology, Mayhew is also trying to turn technology into a way to help people more engaged instead of distracting. Making a contrast of the historic paintings with the scene in modern society also encourages people’s own reflection on the encroachment of technology in the 21st century. In this aspect, AR technology as the carrier of his digital artworks becomes one of the many layers of meaning and intentions of his artworks. The feedback from the museum visitors does show that the exhibition realized by AR technology arouses their interests and inspires them to take a closer look at the original paintings. According to Shiralee Hudson Hill, the Interpretive Planner in AGO, around 84% of the visitors to the exhibition ReBlink reported feeling more engaged with the art (Impossible Things, 2017).

 

• Hacking the Heist in the Isabella Stewart Gardner Museum

Hacking the Heist is an AR project aiming to digitally place the stolen art masterpieces back, in order to provide people with the opportunity of viewing those special art pieces and also to inspire people to think about the intersection of art, media, and technology (Cuseum, 2018).On March 18, 1990, thieves stole thirteen masterpieces in Isabella Stewart Gardner Museum at one time, whose total worth was more than $500 million, which makes this art heist one of the most notorious ones in history. Those paintings, including paintings by Rembrandt, Manet, and Degas, have never been returned back. The Isabella Stewart Gardner Museum hangs gilt frames of stolen masterpieces in the same place as before. Now, leveraging the Hacking the Heist app, visitors are able to view the stolen works on the walls through the screen.

From the perspective of the museum’s function, the AR technology reinforces the museums’ role of preservation, which is one of the initial functions of the museum (Buren, 1985). A historic painting is like a freeze-frame of the past. The value of a historic painting is eternal when it is well preserved. People just think habitually that museum is the place for preservation, but forget the possibility of damages on artworks due to improper ways of maintaining and the possibility of theft and loss. Who could imagine or even think of the possibility that such a huge heist happened to the Isabella Stewart Gardner Museum? Fortunately, benefited from the development of technology and the innovative use of AR, the stolen paintings can return to the walls in a different way. The digital restoration of those stolen masterpieces has more than just original historical values. The visual content is layered over the real world and makes a contrast with the empty frames in reality. Thus, visitors can be more aware of the loss in art fields and realize the preciousness of the world’s culture.

This experiment was conducted by a group of Bostonians who love art. Their choosing AR to bring back stolen artworks is an important process of endowing meaning in this project, while the digital form of restoration inspires visitors to interpret the artworks and this AR project in different perspectives. In this case, AR technology enables more visitors to learn about the biggest heist in the world that happens to the Isabella Stewart Gardner Museum, to appreciate the stolen masterpieces again, to realize the preservation function of the museum and the preciousness of art and to think about how art and technology intersect. This is also the intention of the group of art-loving Bostonians to have this AR project in the Isabella Stewart Gardner Museum.

 

Possible Problems of Implementing AR in Art Museums

With the development of technology, art and art museums have both adapted to the new conditions. The rise of interactive and digital art and the widely and commonly use of handheld audio guides among art museums proves the simultaneous development and intersection of technology and art. Although AR has been increasingly popular among art museums and artists in recent years, it cannot be denied that there are also some limitations or disadvantages of applying it. Art Museums may also face some obstacles when implementing AR in exhibitions.

First, art museum may have financial problems. To launch an AR project in art museums actually costs a large amount of money. Some of the museums are founded by the special foundation, for example, Knight Foundation, which funds technology innovation for art-making and engagement, but most of the museums are not as lucky. The financial problem is the most realistic problem many art museums face. Also, to implement new innovative technology is an issue of the overall business and operational structure. It is also related to the infrastructure in the museum. A museum should decide whether to develop its own AR app or use some existing one considering their financial conditions. Also, a museum that is determined to embark with AR, its infrastructure should also be improved, for example, free and stable WiFi for the AR app in the whole museum.

Second, AR technology as an emerging technology may cause the challenge for collection in the future (Trendwatch, 2015). We cannot anticipate the technological progress in the future so it might be possible that the artworks we are now excited experiencing with mobile devices are not viewable in the future because of iteration or retirement of one technology involved in AR tools. Similar things have already happened. Take cassette tape for example: only after decades its popularity, it is now hard to find a proper player for an old cassette tape.  Art museums that consider implementing AR for permanent or temporary collections must take the risk of technological iteration.

What’s more, researchers and experts are still working on some fundamental problems in the process of designing AR. Tracking limitations may cause registration problems. How artists deal with the possible scene and depth distortions and visibility issues also remain questionable. (Kruijff, Swan & Feiner, 2010). If the digital artworks presented by AR have the issue of scene and depth distortions or visibility issues, the imprecise augmentation will apparently negatively influence visitors’ visiting experience and their interpretation of the artworks.

Conclusion

The analysis of three AR projects in the Pérez Art Museum Miami, the Art Gallery of Ontario, and the Isabella Stewart Gardner Museum shows that AR can afford

From the perspective of visitors, AR can arouse their interest and also inspire them to explore more details of the artworks. Visitors’ interpretation of artworks can be enhanced through such an innovative and inspiring way. From the perspective of artists, AR can help them to realize their specific intention of creating the artworks. When artists actively choose AR tools as the form to present their artworks, it means that AR as an art medium also participates in the process of meaning creation. For art museums, AR tools can make more use of museum spaces inside and outside of the museums and help artworks interact actively with the architecture. Also, using AR to restore historic stolen artworks help to maintain and reinforce the museum’s function of preservation. Since different AR project has different themes and different techniques and settings, AR’s affordance listed above cannot be generalized to all art museums and exhibitions.

In spite of the affordance, AR’s application also faces some problems. Some basic problems are fundamental technical problems. If AR is implemented in the art exhibition, museums should consider the financial and infrastructural ability, while artists should make sure that AR’s technical defects do not negatively influence visitors’ interpretation of the artworks.

 

References

Augmented Reality. (2013), In Encyclopædia Britannica. Retrieved from: http://www.britannica.com/EBchecked/topic/1196641/augmented-reality

Adams, R., Arisona, S., & Gibson, S. (2008). Transdisciplinary digital art: sound : vision and the new screen : digital art weeks and interactive futures 2006/2007, Zurich, Switzerland and Victoria, BC, Canada, selected papers . Berlin: Springer.

Battlett, F. C. (1932). Remembering. Cambridge, England: Cambridge University Press.

Buren, D. (1984). Function of the Museum. In R. Hertz (Ed.), Theories of Contemporary Art (2nd ed., pp.189-192). Englewood Cliffs, NJ: Prentice-Hall.

Craig, A. (2013). Understanding Augmented Reality (1st edition). Morgan Kaufmann.

Coates, C. (2018). How Museums are using Augmented Reality. [website post]. Retrieved from https://www.museumnext.com/2019/02/how-museums-are-using-augmented-reality/

Cuseum. [Screen name]. (2017). PAMM AR – Felice Grodin: Invasive Species [Video file]. Retrieved from https://vimeo.com/259939997

Cuseum (2018). Hacking The Heist. [website]. Retrieved from https://www.hackingtheheist.com/

Cupchik, G., & Gebotys, R. (1988). The Search for Meaning in Art: Interpretive Styles and Judgments of Quality. Visual Arts Research, 14(2), 38-50. Retrieved from http://www.jstor.org/stable/20715675Copy

Ding, M. (2017). Augmented Reality in Museums. In B. Crawford. & E. Kane (Eds.), The Augmented Museum: Essays on Opportunity and Uses of Augmented Reality in Museums (pp1-17). Pittsburgh: Carnegie Mellon University: ETC Press.

Impossible Things. [Screen name]. (2017). ReBlink Teaser. [Video file]. Retrieved from https://vimeo.com/223661495

Impossible Things. (2017). ReBlink. Art, Augmented Reality. [Website Post]. Retrieved from http://www.impossiblethings.co/project/reblink/

Gilbert, S. (2016). Please turn on your phone in the museum: cultural institutions learn to love selfies and social media.(TECHNOLOGY). The Atlantic, 318(3), 32–33.

Kruijff, E., Swan, J. E., & Feiner, S. (2010). Perceptual issues in augmented reality revisited. 2010 IEEE International Symposium on Mixed and Augmented Reality, 3–12. https://doi.org/10.1109/ISMAR.2010.5643530z

The Art Gallery of Ontario. (2017). REBLINK. [website post]. Retrieved from https://ago.ca/exhibitions/reblink

The Pérez Art Museum Miami. (2017). Felice Grodin: Invasive Species. Augmented Reality (AR) Exhibition. [website post]. Retrieved from https://www.pamm.org/ar

The Pérez Art Museum Miami.  [Screen name]. (2018). In the Studio with Felice Grodin [Video file]. Retrieved from https://www.youtube.com/watch?v=HM50cCj5gUo

Trendwatch. (2015, May 1). Wearable Tech: when “bring your own device” means shirt and shoes. [Wesite post]. Retrieved from: https://www.aam-us.org/2015/05/01/wearable-tech-when-bring-your-own-device-means-shirt-and-shoes/

Mortice, Z. (2018, Apr 12). The Invasive Species Exhibit Wriggles Into the Art World Using Augmented Reality. [website post]. Retrieved from https://www.autodesk.com/redshift/augmented-reality-art/

Nofal, E. (2013). Taking Advantages of Augmented Reality Technology in Museum Visiting Experience, 6th International Congress “Science and Technology for the Safeguard of Cultural Heritage in the Mediterranean Basin”, Athens, Greece, Volume: III. Retrieved from: https://www.researchgate.net/publication/258510269_Taking_Advantages_of_Augmented_Reality_Technology_in_Museum_Visiting_Experience

Trendwatch. (2015, May 1). Wearable Tech: when “bring your own device” means shirt and shoes. [Wesite post]. Retrieved from: https://www.aam-us.org/2015/05/01/wearable-tech-when-bring-your-own-device-means-shirt-and-shoes/