Design Process Sutton House Stories

Sutton House Stories: Augmented Reality Experience Design Through the Lenses of Dramaturgy

This article introduces dramaturgy as methodological lenses for designing Augmented Reality (AR) experiences in cultural heritage through the case study of Sutton House Stories, an AR experience for Sutton House, a Tudor National Trust historic house in London, UK. I wrote the text originally for a publication that did not happen, so here it is.

Sutton House Stories was developed over a period of eight months in 2019 in collaboration with the house's heritage educators, curators, visitor experience experts, and with a team of AR developers. Visitors wear an AR smart glass device and are guided around the Great Chamber room by three invisible characters who lived in the house across three different eras. The experience lasts for four minutes or more if the viewers pause to observe and interact with the material. Along this walkthrough they see through the glasses and hear a collection of virtual material (3D characters, videos, animations, visual effects, soundscapes) superimposed realistically onto the physical space and tied together in an overarching narrative. The experience was evaluated over several days with visitors following an iterative process where feedback was incorporated into a revised application. In the following sections I will introduce the new methodology through the description of the project’s design process and in articulation with the relevant literature and the evaluation results.

This research considers cultural heritage as an enabling space for designing interactive AR experiences. Cultural heritage is 'the full range of symbolic and artistic materials, delivered to each culture from the past to the present’ (Jokilehto 2005) and is divided into tangible and intangible (UNESCOa). Tangible cultural heritage includes buildings, monuments, artworks, and artefacts. Intangible heritage, on the other hand, is an ever-expanding definition that includes materials ‘born digital’, refers but is not limited to traditional festivals, oral traditions, folklore, knowledge, rituals, customs, ways of life, languages, traditional crafts, and dance (UNESCOb). This research considers in particular heritage sites. Heritage sites are physical places that incorporate both tangible and intangible heritage, they are the holders not only of the physical structure but also of stories: the narratives of past events which happened in their grounds. Heritage sites frequently use storytelling as a powerful tool to guide their visitors through their spaces. Stories are an entertaining medium through which important information about the place and its history are passed on in a memorable and enjoyable form (Rickly-Boyd, 2009; Paolini and Di Blas 2014; Soerjoatmodjo 2014). Therefore, the narrative and the way it unfolds play a paramount role in making the experience compelling. However, in digital heritage, storytelling is often framed in a linear, didactic way stripped of its ability to evoke emotional and affective responses (Perry et al. 2017; Tivers, 2002).

Engaging with Historic Sites

In a typical heritage site visit, such as a castle or a house, walking through the place surrounded by the objects within it creates an in-situ experience which is inherently immersive. Heritage educators and curators have sought to assist and enrich the heritage site walkthroughs with labels, accompanying texts, tour guides, and exhibitions; especially for sites that are empty of physical objects. These, more traditional, methods of engaging the public are usually dissociated from the experiential way visitors interact with the physical space and are often at odds with the experiential demands of millennials and digital natives. In order to lessen the dissociation and increase experiential and situated knowledge building (Kolb 1984), and invite younger audiences, curators have been exploring new methods of interpretation that make the physical walkthroughs part of an engaging educational experience: representations of life in the site in the past through live performance, narration used on guided tours, audio guides and binaural audio soundscapes that use sound to tell the place’s story as the visitor walks through it, Virtual Reality (VR) installations, 360 VR videos, interactive touch screens and mobile guides which research has found promotes active learning (Hauser et al 2009; Dillenbourg and Evans 2011), and AR, which has demonstrated its capacity for increasing engagement and situated learning (Dunleavy, Dede and Mitchel 2009; Sylaiou et al. 2009; Radou 2014; Moorhouse, Tom Dieck and Jung 2019; Lin, Lo and Yueh 2019). Significant evidence of the success of these tools as educational methods can be seen in the enhancement of empathic situated learning for all visitors, the attraction of visitors to sites who wouldn’t usually visit museums or heritage sites, e.g. teenagers and young adults, as well as the offer of bespoke, contextual information that would be very difficult, expensive or totally impossible to present with other media used in heritage sites (Helle et al., 2017; Tom Dieck, Jung and tom Dieck 2016).

A brief history of AR in Cultural Heritage

AR is a form of Mixed Reality (Milgram 1994) whereby digital content is superimposed on the real-world environment. Its ability to show the viewers reconstructed material of the past makes cultural heritage a particularly suitable field of application. An AR environment, in contrast with VR, does not exclude the real environment from the view. Instead, the digital elements are projected on top of the real environment and can be viewed by means of a display device. This allows viewers to engage both with their lived environment of the heritage site and the augmented experience without also missing out on the social experience. AR display devices range from mobile and tablet devices to Head Mounted Displays (HMDs), and light weight HMDs called Augmented Reality smart glasses “wearable Augmented Reality (AR) devices that are worn like regular glasses and merge virtual information with physical information in a user’s view field” (Ro, Brem and Rauschnabel 2015, 6). Sutton House Stories uses such glasses, in particular Microsoft’s Hololens 1. A plethora of companies are developing glasses of different shapes and with different technical characteristics. Glasses have the advantage of allowing for whole body interaction as they don’t require the user to hold a mobile device. Most importantly though, they offer a more unified experience. They are lightweight and wireless, they integrate headphones, and eye-tracking, as well as hand and finger tracking. These characteristics enable exciting opportunities for a combination of direct interaction, personalised soundscapes, and free movement. Since hands are no longer needed to hold the device, the whole body can be used for interaction, mimicking the way we normally interact with the world around us. For example, by performing a pinching gesture the viewer can pick up a virtual letter on a physical table. The freedom to move and use many of their senses in a unified way brings AR glass users closer to the way they experience a real-world environment by acting on it.

One of the first research projects for cultural heritage was MARS (Höllerer et al. 1999) and the Archeoguide (Vlahakis et al. 2001). The Archaeoguide used a laptop, tablet and a Personal Digital Assistant - the available mobile technology at that time - to provide visitors with contextualised and personalised information of the ancient Olympia site in Greece based on the visitor’s position and orientation. The experience was deemed very interesting and educative by the evaluation participants, and was met with particular enthusiasm by those who were computer literate (younger audience). However, as with most of the projects back then, ergonomics and usability were not good because of the heavy equipment (Archaeoguide had an Head Mounted Display - a pair of large and heavy glasses connected to a computer that the user had to wear on the back), lack of multimedia content, and poor screen visibility. With the advent of mobile devices, better optics and multimedia, development frameworks, and smart glasses, AR applications and relevant research has grown quickly and AR experiences are now much better and widely available to the everyday museum visitors: Museum of London’s StreetMuseum: Londinium (Museum of London 2017), MixAR (Narciso et al. 2015), MARCH (Choudary et al. 2009), KnossosAR (Kasapakis, Gavalas and Galatis 2016), the CityViewAR system (Billinghurst and Duenser 2012), MR Museum in Kyoto’s Kennin-ji (Mixed Reality Museum in Kyoto, 2019) and the The Indigital MR app (Old and New, 2019) to name some recent ones as the list is very long.

The majority of the mobile AR applications offer additional information by means of superimposing either images or 3D models and combining this with textual and aural information and video recordings (Krogstie and Haugstvedt 2015; Noh, Sunar and Pan 2009; Lieastol 2019). A lot of research focus is also given on technological developments, hardware and software, and exploring different AR systems including authoring tools. Notwithstanding the importance of this body of research, very few, recent, works have focused on the stories, the tangible and intangible information, and, in the case of historic sites, those incredibly immersive spaces to walkthrough and interact in. All these comprise the visitor’s experience and the application of AR can be viewed as one aspect seamlessly integrated within this holistic context. Therefore, its design has to be meaningful within the context. Focusing only on what the technology can do without paying much attention to the context, runs the risk of producing experiences that are engaging based primarily on the novelty of the technology, but detached from the overall visitor journey. The design process for Sutton House Stories considered not only the technology but also the narrative that guides it, the space it takes place, and the intent of the experience - learning the house’s history.

The lenses of Dramaturgy

To date, smart glass research focuses mainly on the engineering and computer graphics aspects. In digital heritage, there are a handful of AR applications that use AR headsets most of which have been commissioned by heritage organisations without much prior in–depth research. In the academic sector, existing research focuses on exploring AR glasses in terms of usability, without, again, research on the particular requirements for learning through experiencing (Pedersen et al 2017, Geronikolakis and Papagiannakis 2017). Usability studies in context are important, as good ergonomics and interface design are fundamental to a seamless visitor experience. At the same time, as (Patel 2015) suggests, the digital side of storytelling is much more than just technology. It is also about the context within which the technology is used; who you want to engage, why and how you want to engage them, what stories you want to tell, and what you would like the visitor to feel, remember, and do. Sutton House Stories sought to ask what kind of content should be developed to improve the engagement of the visitors with the place’s history. The connection between engaging experiences and learning (Lepper and Malone 1987) puts, early on, the focus on the visitor’s emotional journey. Therefore, storytelling was placed at the heart of the design process. The content, based on curatorial decisions, and the supporting role of the technology were built around it. In order to shape the content accordingly, understanding the technology we work with, and ensuring that the final experience is of educational value to the visitor, the house’s educators, historians, visitor experience experts, the AR developers, and myself as an experience designer collaborated closely throughout the duration of the project. It is through this collaborative, user-centred, creative process that starts from an artistic rather than engineering perspective, but includes engineering considerations, that I draw the parallel to dramaturgy and bring it in as lenses through which to view digital design and digital interpretation/curation.

There are many definitions for dramaturgy, and the scope of this article is not to analyse them all. For the purpose of my methodology, I define dramaturgy as the structure of a performative experience, its ‘composition’, or ‘fabric’ (Turner and Behrndt 2016). Adam Versenyi’s definition of ‘dramaturgy’ as ‘the architecture of the theatrical event, involved in the confluence of components in a work and how they are constructed to generate meaning for the audience’ (in Turner and Behrndt 2018, 18) is also useful as it allows me to directly draw parallels between this and the role of an AR designer which is to create seamless, meaningful experiences. I choose to draw on dramaturgy as opposed to interactive digital narrative design (IDN) as I consider narrative design part of the dramaturgical process. While IDN explores the narrative in detail, dramaturgy is about how narrative relates to the plot, how elements such as sound, visuals, and staging are arranged to support this relationship, what is the role of the audience, and, overall, how meaning is constructed through the reshuffling of these elements.

Within the realms of technology-mediated experiences the artwork elements extend to include the technological components, and dramaturgy becomes the glue between the artistic and technological aspects of the work, extending at the same time the design space for both. As Causey states: “Technology tends to form a new perspective of dramaturgy beyond the traditional theatre, and performance can be transformed from two aspects: first, technology simulates the art of performance into an interdisciplinary format, opening minds; and mix-technology can combine performance, dance and installation art into a new performance space.” (Causey 2006).

In the quest for meaningful experiences there are different disciplines involved with expertise in the different aspects of the experience. Sutton House Stories started with two one day workshops that took place a month apart. The first workshop started with connecting with the house and its history, getting acquainted with the historic characters who lived in through the centuries, its different uses, identifying the appropriate room to use, and defining the role of the visitor within the experience. As a knowledge exchange event it helped heritage professionals understand the feel and use of AR glasses. During the workshop they tried the glasses on so that they get an idea of how they feel, how the embodiment changes when one puts them on, and in order to help them better imagine how they could be used for digital interpretation. The workshop focussed on two main points: understanding the curatorial strategies of the team, and identifying the limitations and opportunities of the smart glass technology.

Curating learning

The dramaturgical vision that drives the forthcoming decisions is shaped by asking the contextualising questions: What do you want to say? What do you want your audience to experience? In the case of cultural heritage, the answers can be found in the fundamental premise of most heritage sites, which is to better inform their visitors about the past. This largely includes anticipating what the visitor might wish to know and understand about the site and its history by the end of their visit. It is also about which stories or histories will be emphasized, if so, which is a curatorial decision of the heritage site. Some sites might be linked to one single historical event, while for others the knowledge about the place is considerable and might be fragmented across the centuries. Establishing what the site’s learning team wishes to communicate about the site is crucial to the development of the narrative and the route(s) that the visitor will take in their visit. Which places/objects/rooms are important and why? What stories does each one narrate? Who were the people who lived there? What kind of events happened on the site? What were the connections with the local community? How did the place look, feel, sound, and even smell back then? Which stories should be foregrounded? Should there be a selective process that may privilege some stories or histories over others? These are some of the design driving questions that heritage educators offered valuable input during the workshop.

The main question with regards to interpretation, which was a major challenge for the heritage educators, was how best to represent the rich history of a house that’s heritage spanned four centuries in a short and coherent way. Sutton House is unusual in being one of the longest continuously inhabited houses in Hackney. Indeed, since its erection circa 1535 when Hackney was a village, to the 1990s when it was bought by the National Trust, it has been a residence, a school for girls, and a church institute for men. In each era, it has had important connections to the local community, to historic figures, and to wider socio political events. At the same time the building has undergone structural changes and is itself of great architectural interest. After discussion it was determined that, of all the rooms, the Great Chamber was the ideal room to sew together the different pieces of the house’s history since it was the heart of the house across the centuries. Like the rest of the house, it is a sparsely furnished room, with richly decorated walls that have original Tudor panels. Its floor has reinforced floor boards from the time it served as a ballroom. Therefore, our design challenge was what intangible heritage the experience would bring in the room and how best it will use it to tell these multiple narratives.

Narrative Design

Shifting the focus from smart glasses as a novel technology, to smart glasses as a storytelling medium opens new design spaces. If we consider digital reconstructions and audiovisual augmentations (text, videos etc) as elements of the experience and not the experience itself, important questions with regards to linking the elements together arise. (Moreno, MacIntyre and Bolter 2001) write:

“Often a new medium such as AR develops from the work of technical innovators. Initial research focuses on the mechanics of the technology, while issues of effective use of the technology as a medium are often secondary. The new medium may enjoy some initial success as an entertainment form based completely on the novelty of the technology. In contrast, established media rarely depend solely on technology to provide a gratifying experience”. (in Moreno, MacIntyre and Bolter 2001, 149)

They sought to explore AR as a storytelling medium by reimagining the Mad Hat tea party from Alice in Wonderland as a participatory AR experience. The user assumes the role of Alice and sits at the tea party wearing a first prototype of modern AR glasses through which they see three interactive characters sitting with them: the Mad Hatter, the Dormouse, and the March Hare. The user’s objective is to get directions to the garden, located somewhere in Wonderland. Each video-based character can perform a few gestures such as serving tea, or asking riddles. The characters view the user as an interruption to the party already in progress. The unique character behaviour and their realistic interaction with the viewer creates a meaningful experience. This was one of the first important works on narrative and AR.

There is little and fragmented research that focuses on narrative design or that considers AR as a storytelling medium. Commonly, narratives develop around specific objects offering an object-oriented and, thus, compartmentalized view of the place’s history (Spierling, Winzer and Massarczyk 2017). The CHESS project has sought to create tools for digital storytelling in order to help museums create interactive rich-media stories (Pujol et al. 2013). Gunnar Liestoel discussed the absence of narrative techniques in AR experiences (Liestoel 2011, 197) noting that the majority use asynchronous storytelling, narrating through a set temporal loop of events and actions. However, as Liestøl exemplified with the reconstruction of the battle in Omaha Beach (Liestoel 2018) there is a close spatio-temporal connection within the asynchronous storytelling that can be used as a design device. In that project viewers have the option to pause and access background information of the presented material, and move along the temporal loop using the mobile app and physically in their spatial surrounding at the same time. Ulrike Spierling developed a location-based mobile AR app where the visitor is required to find ‘spirits’ on location. These spirits are connected to a story which is narrated by a fictional character during the walkthrough (Spierling, Winzer and Massarczyk 2017). Despite these attempts to base the design in storytelling, the design process is rarely explored in depth (Shilkrot, Montfort and Maes 2014; Nam 2015); Rouse and Barba 2017; Lugthart, van Dartel and Quispel 2017). In the meantime, literature that discusses projects where narrative design was central to the AR experience (Liestoel 2011; Liestoel 2018; Pujol et al. 2013) still limit themselves to reconstructions of events, objects or buildings. Through the proposed methodology, Sutton House Stories sought to go beyond this. The narrative design process was driven more from an artistic intent to weave together the house’s history and its various roles across centuries in a way that was meaningful to today’s audience rather than seeking to simply recreate its past (hi)stories. The dramaturgical challenge was how to link the different eras and make the experience compelling with regards to the past and relevant with regards to its modern visitors. The dramaturgical perspective is central to this methodology because it leads to more in-depth narratives, structurally and emotionally.

The role of the visitor as an interactor in the experience is an important part of the narrative, and thus, an important dramaturgical decision. Are they interacting and, if so, how? How much agency do they have? And, most importantly, who are they in the narrative? The visitor's participation starts as early as the moment they step into the place. Any role they may have in the overall experience should be designed to start then. This is commonly called ‘designing the on-boarding’ experience. Are they just visitors? Have they come here with a goal? How is the device as an object justified in the narrative?

In her book Challenge for the Actor, Hagen maps out the essential steps of character analysis (Hagen 1991). She suggests a few questions to guide the creation of character backstory for actors that I find incredibly relevant to narrative design in this discussed situation:

Who am I? What is my present state of being? How do I perceive myself? What am I wearing?
What are the circumstances? What time is it (year, season, day)? Where am I (city, neighbourhood, building, and room)? What surrounds me (immediate landscape, the weather, the objects around)? What are the immediate circumstances (what has just happened, what do I expect will happen later on)?

What are my relationships (to circumstances, place, object and other people)?
What do I want (overall and immediate needs and objectives)?
What is my obstacle (what’s in my way and how can I overcome)?
What do I do to get what I want? How do I achieve my objective? What’s my behaviour? What are my actions?

These questions led the narrative design during the first workshop. The question ‘Who am I?’ is one of the most important ones as the visitor puts on their AR glasses, which, by including attached headphones and the ability to capture gaze, location, and finger tracking, come closer to the way visitors normally interact with the space. The narrative can be driven by the answer to this question, by the role that the visitor assumes, including not assuming any role at all. The roles are numerous, a few that were suggested during the workshop included a time traveller, a spy, a historian, a historian's assistant, a journalist, and a person from the history of the place. Some of these roles invite a game design approach to the experience. For example, a spy can listen to classified transmissions, intervene to change the route of events, a time traveler can look for objects or face the dilemma of changing the route of events. The rest of the questions support the narrative design. Why is a spy there? What is their objective? How do they go about fulfilling it? These can also be enhanced with questions that can drive game narrative. This approach brings into play game design elements such as identification of game design pillars, the creation of mechanics and gameplay. The visitor can also be a character with a connection to the history of the place or assume the role of multiple persons looking at the story from different points of view. The visitor can also be themselves in which case interactivity could give way for a more complex narrative. Interestingly, the workshop participants felt more comfortable with not giving the visitor any role at all because it would introduce additional complexity to the challenge of having to familiarise themselves with the new device. There was discussion, however, about how the background of the target audience, e.g. gamers as opposed to non-tech savvy visitors, would influence this decision. Interestingly, the evaluation partially confirmed this hypothesis. Participants who used technology frequently, and even played games, discussed how they would have liked to see game elements in the experience such as ‘choose the character to guide you’ but were sceptical about assigning a role to the visitor that would require them to carry out specific tasks as this would add to their cognitive load and potentially negatively affect learning.

Once the role of the visitor was defined, the narrative design, following the character building questions, focused on the space and the other people present. After many iterations, three main eras were picked from the four centuries of history during which the house had different functions for a considerable number of years: circa 1535 when it was first built and occupied by Sir Ralph Sadler, who served as Secretary of State under King Henry VIII, during the Georgian period when it was used as a girls school (1700), and shortly before WW1 when it was a men’s Social Club (1914). In order to showcase the different histories the visitor should be taken on a journey across these eras, in a vaguely described way of time travel or dream, since they had no specific role. With the help of the site’s historians three characters from each era were selected to guide them through: Sir Ralph Sadler, Mrs Freeman a schoolteacher who ran the girls' school for 43 years, and Ernest Alfred Munday, a man who frequented the social club and went on to fight and die in WW1. There were discussions about how the visitor would traverse through the eras, what each character would say and how they would behave, how the transitions from one era to the other would happen, what the virtual material would be, and, of course, what is the overarching narrative. In Sutton House Stories it was about the actual walkthrough, and how visiting these sites and learning about them, and inviting other people to visit them and sharing their stories, preserves their memory and creates new ones. The experience ends in 1598 with Sir Ralph Sadler asking the viewer to look through a window (the actual window) which instead of the modern buildings displays the countryside as it was back then, while telling them: ‘Lots of things have changed over the years...but my house and its stories are still here. Will you share them, so they are not forgotten?’

This narrative was the result of a fuzzy, creative process during the two workshops; however, the start of its formation is traced in the decision to let the house tell its stories. This manifested in two ways. Because of technological limitations, it was decided to not have three virtual characters interacting with the visitor but to have their voices instead. This was later changed to have at least Sir Ralph Sadler as a virtual character as the house’s first owner. Each character would talk about how things were back then giving a glimpse of everyday life, surrounding community, and sociopolitical situations of the era, accompanied by relevant soundscapes and visual material. Writing the script was fundamental to shaping the overarching narrative. In the first workshop it was also decided that the visitor would be taken through the eras in reverse chronological order starting from WW1, which is the most recent historic period, back to the person who built the house. In this respect the visitor starts from the present house through to its transformations over the centuries. Therefore, the house was omnipresent in the experience physically and through the virtual material. In between workshops, reading about the modern history of the house sparked the idea that would become central to the narrative. In the late 1990s Sutton House was saved from being demolished and turned into a block of flats literally last minute by a local activist.

There are many such sites today in London and other places in the world with local initiatives trying to save them from being forever lost. Local communities are important to a site’s oral histories, as is the case with Sutton House which even today is home to community social events. From this information and the stories as they were being shaped through the workshops and script writing, emerged the opportunity to make visitors think beyond the emotional connection between them and the house on to how this connection at present can enable them to act in the future, how their acts, including their present visit, contribute to the house’s and other houses’ life, and help them realise how histories, stories and people are interconnected across time.

Sound, Choreography, and the Archive

Having the structure of the experience defined, in the second workshop the goal was to add the details of what each character will talk about, the soundscape that will accompany it, the models, videos and any other visual material. In the search for material we looked in depth into the house’s archives with the help of the historians. Like many sites, the house has an extremely rich archive of physical artefacts, which are not on display. An AR experience alone cannot and should not convey all the archival material about the site, but its role can be to convey its richness and entice the visitor to make further enquiries about things of interest. The narrative design and the search through the archives were a bi-directional act: sometimes archive findings guided the storytelling rather the other way around. For example, there was little information about the lives of men who frequented the social club before WW2 and who went on to fight in the war. Alfred Murray’s was one that was relatively well documented, and his life story directed what he would be talking about.

The soundscape for each era was an important consideration in the dramaturgical process as sound, even without image, is fundamental in changing the way one interprets a place. Throughout the experience there were environmental sounds, e.g. fire cracking coming from the fireplace, paint brush on canvas, and birds singing outside the window. Occasionally sound was used for a music-related interaction such as when the viewer watches a pair of footsteps dancing a minuet during the girls’ dance lessons, and as part of a video displaying the horrors of war. The voice over actors for the characters were carefully chosen for their accent and appropriately directed to animate the characters and not simply narrate the script.

Alongside the different eras, it is crucial to carefully craft the journey of the visitor from one place to another and to guide them from place to place by means of the story. For this construction it is imperative to choreograph and to some extent time the route the visitor will take. By choreograph I do not imply that there is no agency at all in what the visitor can do. In fact, the evaluation showed that participants wanted to be able to go back to places they visited before, to replay a sequence, meet the characters again, or pause to observe a virtual object without being urged to continue. We designed some of these in from the beginning, however, there was a specific route the visitors had to follow which subtly unfolded in each interaction. The experience was divided into ‘chapters’, one for each era that would start with a gesture over a virtual pin. In this way the viewers could pause in each chapter and explore the virtual material for longer if they wanted to. There were subtle visual arrows on the glasses that showed where the pin for the next chapter was. We evaluated whether having to gesture click pins would break their experience but the evaluation feedback unanimously indicated that they liked being guided in this way, and being able to go through the eras in separate parts as they may wanted to replay some or chose which ones they were interested in. Although we did not implement this level of agency in Sutton House Stories, it is a parameter that should be included in the choreography. While creating the content for the part where the viewer meets Sir Ralph Sadler, we encountered a choreography issue that was due to the device’s limitations. It was then that the importance of recognising and taking into account in the design the technology limitations became apparent.

Technological Limitations and Opportunities

The AR glasses that were used, and all AR glasses in the market, have a limited field of view. The width of the observable area with superimposed content that can be viewed creates what is called the ‘window effect’. The digital material is superimposed on a rectangle in front of the user’s eyes at about one meter distance from them in the physical space. This means that it does not cover the full 210/150 degrees horizontal/vertical field of view of human eyes. Consequently, the visitor must be guided to view things that happen relatively close to them or very far away so that they occupy the whole window. This can be done with arrows for visual feedback, but they could also be incorporated in some way into the narrative. In the encounter with Sir Ralph Sadler, for example, Sadler sits on a table that occupies a large space of one of the walls facing the viewer, and talks to them while writing a letter. The first thought was that for the viewer to have a full view of a scene, they will have to sit on a chair that is positioned on the opposite wall. They would be guided there by Sadler’s voice, which would work in conjunction with the visual feedback. The limited field of view affects the distance from which the user looks at things and how small or big these things are (objects, people, videos, etc). In the above example we deliberately chose to choreograph things to pause movement for a while. In doing so, in effect the visitor was made to sit because from the place of the chair they had a view of the whole room. This allowed us to recreate material that could be viewed seamlessly from this position. The field of view was not the only constraint that led us to craft this scene. One of the aspects of AR that has been hailed by AR developers is the opportunity to see and interact with highly realistic 3D models. What is rarely discussed is the user response to close encounters with 3D models through the glasses. Close encounter with 3D models whose characteristics, positions and behaviour only relatively resembles that of real persons creates an eerie feeling that may result in them falling into what Mori describes as “the uncanny valley” (Mori 2012). This can break the experience. Thinking through this limitation, we decided to avoid a face to face encounter with Sadler’s 3D model so positioned him with his back turned to the viewer sitting in front of the table, which was also the actual table. We were still not content with restricting the viewer’s free movement, but the choice to turn Sadler to not face the viewer meant that it was more meaningful to move closer to him and peak through his back to see what he was writing or other objects on the table while he was speaking to them. Because the visitors would still be able to see some of his face, we used a translucent shader for the character’s head, which hid detailed facial characteristics but gave an impression of a ghost-like figure from the past.

This process demonstrates clearly that the limitations of the technology do not have to prohibit design decisions. They can be turned into opportunities and the design process is a fertile ground for this to happen. The process of exploring the interplay between the visitor path and the limitations of technology is a fuzzy creative discourse and it is difficult, and also pointless, to extract specific design principles, especially since the device’s constraints will change from model to model. However, the technical limitations and how to turn them into opportunities via choreographing the user’s journey, is an important consideration in the process of creating the dramaturgy.

Scaffolding a User Experience model

Through the development of Sutton House Stories, I have sought to demonstrate that looking at AR experience design through the lenses of dramaturgy can offer a fruitful methodology for creating a compelling educational walkthrough of a heritage site. Under the dramaturgical lens, educational material, narrative design, and technology limitations and opportunities become the three important pillars of the design process. These three aspects frame the process and come in dialogue in order to drive the design decisions. The structure of the different elements is shaped through dramaturgical decisions with the goal to create a meaningful, compelling and learning in-situ experience. The design goal is that when visitors leave they have not only looked at fragments of the place’s history but also walked through the memories of the place, connected to the history of the place and carried with them their own interpretation of what they experienced. Moreover, creating an overarching narrative can boost the dramatic aspect and closely links the three components into an unfolding story in which the visitor is centre stage. The presence of place, characters, time and events, the levels of interactivity, usability of technology, and complexity of narrative, and the consideration of the overall environment that includes these elements is essentially creating the dramaturgy of a piece. Designing the AR experience then becomes similar to the act of doing dramaturgy.

The approach is holistic as it considers the narrative and the overall context in which it unfolds. As do the few projects that understand AR as a storytelling medium, I look beyond only digital reconstructions and the overlay of information, seeking to identify a connecting narrative that ties these things together. However, I also consider the broader context in which the overlaying happens and where the visitors find themselves physically and emotionally with this new medium.

The methodology is presented here with the view that it provides a starting point for further research into how dramaturgy and interaction design can be fused to create meaningful AR experiences.

References

Billinghurst, Mark and Andreas Duenser. 2012. "Augmented Reality in the Classroom." Computer (Long Beach, Calif.) 45 (7): 56-63.

Choudary, Omar, Vincent Charvillat, Romulus Grigoras, and Pierre Gurdjos. 2009. "MARCH: Mobile Augmented Reality for Cultural Heritage."ACM. doi:10.1145/1631272.1631500.

Causey, Matthew. 2006. Theatre and Performance in Digital Culture: From Simulation to Embededness. Vol. 5;5.;. Abingdon: Routledge.

Dillenbourg, Pierre and Michael Evans. 2011. "Interactive Tabletops in Education." International Journal of Computer-Supported Collaborative Learning 6 (4): 491-514.

Dunleavy, Matt, Chris Dede, and Rebecca Mitchell. 2008;2009;. "Affordances and Limitations of Immersive Participatory Augmented Reality Simulations for Teaching and Learning." Journal of Science Education and Technology 18 (1): 7-22.

Geronikolakis, Efstratios, George Papagiannakis. 2017. Holographic and Augmented Reality simulations for cultural heritage. Technical report, accessed 2 June 2019. https://www.vi-mm.eu/wp-content/uploads/2017/10/%CE%A4%CE%914-Case-study-2.pdf

Hagen, Uta. 1991. A Challenge for the Actor. New York;Toronto;: Scribner's.

Hauser, Walter, Annette Noschka-Roos, Eva Reussner, and Carmen Zahn. 2009. "Design-Based Research on Digital Media in a Museum Environment." Visitor Studies 12 (2): 182-198.

Helle, Seppo, Hannu Salmi, Markku Turunen, Charles Woodward, and Teijo Lehtonen. 2017. MIRACLE Handbook – Guidelines for Mixed Reality Applications for Culture and Learning Experiences University of Turku, ISBN 978-951-29-6884-8, https://www.utupub.fi/bitstream/handle/10024/143218/Miracle_handbook.pdf?sequence=2&isAllowed=y

Höllerer, Tobias, Steven Feiner, Tachio Terauchi, Gus Rashid, and Drexel Hallaway. 1999. "Exploring MARS: Developing Indoor and Outdoor User Interfaces to a Mobile Augmented Reality System." Computers & Graphics 23 (6): 779-785.

Radu, Iulian. 2014. "Augmented Reality in Education: A Meta-Review and Cross-Media Analysis." Personal and Ubiquitous Computing 18 (6): 1533-1543.

Jokilehto, Jukka. 2005. “Definition of cultural heritage: references to documents in history”. ICCROM Working Group ‘Heritage and Society, pp. 4-8, http://cif.icomos.org/pdf_docs/Documents%20on%20line/Heritage%20definitions.pdf

Kasapakis, Vlasios, Damianos Gavalas, and Panagiotis Galatis. 2016. “Augmented reality in cultural heritage: field of view awareness in an archaeological site mobile guide”. Journal of Ambient Intelligence and Smart Environments. 8(5): 501–514

Kolb, David A. 1984. Experiential Learning: Experience as the Source of Learning and Development. Englewood Cliffs, N.J: Prentice-Hall.

Krogstie, John, and Anne-Cecilie Haugstvedt. 2015. Use of Mobile Augmented Reality for Cultural Heritage. In Barfield, Woodrow (ed.) Fundamentals of Wearable Computers and Augmented Reality, 2nd Edition CRC Press, 411-432.

Lepper, Mark and Thomas W. Malone. 1987. Intrinsic motivation and instructional effectiveness in computer-based education. In R. Snow and M. Farr (Eds), Aptitude, Learning and Instruction, vol. 3: Cognitive and Affective Process Analysis. Hillside, NJ: Lawrence Erlbaum Associates.

Lin, Weijane, Wen-Ting Lo, and Hsiu-Ping Yueh. 2019. "How the Multimodal Media in Augmented Reality Affects Museum Learning Experience."IEEE. 12th Asia Pacific Workshop on Mixed and Augmented Reality (APMAR), Ikoma, Nara, Japan, 2019, pp. 1-4doi:10.1109/APMAR.2019.8709286.

Liestoel, Gunnar. 2011. Situated Simulations Between Virtual Reality and Mobile Augmented Reality: Designing a Narrative Space. In: Furht B. (eds) Handbook of Augmented Reality. Springer, New York, NY

Liestoel, Gunnar. 2019. "Augmented Reality Storytelling – Narrative Design and Reconstruction of a Historical Event in Situ." International Journal of Interactive Mobile Technologies 13 (12): 196-209.

Lugthart, Sarah, Michel van Dartel, and Annemarie Quispel. 2017. "Plans Versus Situated Actions in Immersive Storytelling Practices." In . Vol. 10690, 38-45. Cham: Springer International Publishing.

Mixed Reality Museum in Kyoto. A unique insight into centuries-old Japanese artwork, accessed 4 July 2019. https://news.microsoft.com/apac/features/mixed-reality-museum-kyoto-unique-insight-centuries-old-japanese-artwork/

Milgram, Paul, Haruo Takemura, Akira Utsumi, and Fumio Kishino. 1994. Augmented reality: A class of displays on the reality-virtuality continuum. Telemanipulator and Telepresence Technologies. 2351. 10.1117/12.197321.

Moreno, Emmanuel, Blair MacIntyre, and Jay David Bolter. 2001. Alice’s adventures in new media: An exploration of interactive narratives in augmented reality. Conference on Communication of Art, Science and Technology.

Mori, Masahiro, Karl MacDorman, and Norri Kageki. 2012. "The Uncanny Valley [from the Field]." IEEE Robotics & Automation Magazine 19 (2): 98-100.

Moorhouse, Natasha, M. Claudia tom Dieck, and Timothy Jung. 2019. "An Experiential View to Children Learning in Museums with Augmented Reality." Museum Management and Curatorship (1990) 34 (4): 402-418.

Museum of London. StreetMuseum: Londinium, accessed 2 June 2019.

https://www.museumoflondon.org.uk/discover/museum-london-apps

Nam, Yanghee and Yanghee Nam. 2015. "Designing Interactive Narratives for Mobile Augmented Reality." Cluster Computing 18 (1): 309-320.

Narciso, David, Luís Pádua, Telmo Adão, Emanuel Peres, and Luís Magalhães. 2015. "MixAR Mobile Prototype: Visualizing Virtually Reconstructed Ancient Structures in Situ." Procedia Computer Science 64: 852-861.

Noh, Zakiah, Mohd Shahrizal Sunar, and Zhigeng Pan. "A Review on Augmented Reality for Virtual Heritage System." In . Vol. 5670, 50-61. Berlin, Heidelberg: Springer Berlin Heidelberg.

Old and New, HoloLens technology brings new life to ancient indigenous stories, accessed 4 July 2019, https://news.microsoft.com/apac/features/old-and-new-hololens-technology-brings-new-life-to-ancient-indigenous-stories/

Pedersen, Isabel, Nathan Gale, Pejman Mirza-Babaei, and Samantha Reid. 2017. "More than Meets the Eye: The Benefits of Augmented Reality and Holographic Displays for Digital Cultural Heritage." Journal on Computing and Cultural Heritage (JOCCH) 10 (2): 1-15.

Pujol, Laia , Maria Roussou, Stavrina Poulou, Olivier Balet, Maria Vayanou, and Yannis Ioannidis. 2013. Personalizing interactive digital storytelling in archaeological museums: the CHESS project. In G. Earl, T. Sly, A. Chrysanthi, P. Murrieta-Flores, C. Papadopoulos, I. Romanowska, & D. Wheatley (eds.).

Archaeology in the Digital Era. Papers from the 40th Annual Conference of Computer Applications and Quantitative Methods in Archaeology (CAA). Amsterdam University Press.

Ro, Young, Alexander Brem, Philipp A. Rauschnabel. 2015. Augmented reality smart glasses: definition, conceptual insights, and managerial importance. In: Jung T., tom Dieck M. (eds) Augmented Reality and Virtual Reality. Progress in IS. Springer, Cham. https://doi.org/10.1007/978-3-319-64027-3_12

Rouse, Rebecca and Evan Barba. 2017. "Design for Emerging Media: How MR Designers Think about Storytelling, Process, and Defining the Field." In . Vol. 10690, 245-258. Cham: Springer International Publishing.

Paolini, Paolo, Nicoletta Di Blas. 2014. Storytelling for Cultural Heritage. In: Contin A., Paolini P., Salerno R. (eds) Innovative Technologies in Urban Mapping. Sxi — Springer per l’Innovazione / Sxi — Springer for Innovation, vol 10. Springer, Cham.

Patel Samir, Creative Delivery, Blue State Digital, keynote speech, accessed 2 June 2019, https://www.museumnext.com/insight/digital_storytelling_in_museums/.

Perry, Sara, Maria Roussou, Maria Economou, Hilary Young, and Laia Pujol. 2017. "Moving Beyond the Virtual Museum: Engaging Visitors Emotionally."IEEE. doi:10.1109/VSMM.2017.8346276.

Rickly-Boyd, Jillian M. 2009. "The Tourist Narrative." Tourist Studies 9 (3): 259-280.

Shilkrot, Roy, Nick Montfort, Pattie Maes. 2014. nARratives of augmented worlds. IEEE International Symposium on Mixed and Augmented Reality - Media, Art, Social Science, Humanities and Design (ISMAR-MASH'D), Munich, 2014, pp. 35-42, doi: 10.1109/ISMAR-AMH.2014.6935436.

Soerjoatmodjo, Gita. 2015. Storytelling, Cultural Heritage and Public Engagement in AkhirPekan@MuseumNasional, Procedia - Social and Behavioral Sciences, 184: 87-94,ISSN 1877-0428, https://doi.org/10.1016/j.sbspro.2015.05.057.

Spierling, Ulrike, Peter Winzer, and Erik Massarczyk. 2017. "Experiencing the Presence of Historical Stories with Location-Based Augmented Reality." In . Vol. 10690, 49-62. Cham: Springer International Publishing.

Sylaiou, Styliani, Liarokapis Fotis, Kotsakis Kostas, and Patias Petros. 2009. "Virtual Museums, a Survey and some Issues for Consideration." Journal of Cultural Heritage 10 (4): 520-528.

Tivers, Jacqueline. 2002. "Performing Heritage: The use of Live 'Actors' in Heritage Presentations." Leisure Studies 21 (3-4): 187-200.

Tom Dieck, M. Claudia, Timothy Hyungsoo Jung, and Dario tom Dieck. 2016;2018;. "Enhancing Art Gallery Visitors’ Learning Experience using Wearable Augmented Reality: Generic Learning Outcomes Perspective." Current Issues in Tourism 21 (17): 2014-2034.

Turner, Cathy and Synne K. Behrndt. 2016. Dramaturgy and Performance. Revis ed. London: Macmillan Education.

UNESCOa. Definition of cultural heritage. accessed 21 May 2019https://en.unesco.org/ .

UNESCOb. Definition of intangible cultural heritage, accessed 21 May 2019.

https://ich.unesco.org/en/dive&display=constellation.

Vlahakis, Vassilios, John Karigiannis, Manolis Tsotros, Michael Gounaris, Luis Almeida, Didier Stricker, Tim Gleue, Ioannis Christou, Renzo Carlucci, and Nikos Ioannidis. 2001. "Archeoguide: First Results of an Augmented Reality, Mobile Computing System in Cultural Heritage Sites."ACM. doi:10.1145/584993.585015.