Psychnology Journal, Volume 6, Issue 2, Summer 2008

PSYCHNOLOGY JOURNAL The Other Side of Technology EDITORS-IN-CHIEF Luciano Gamberini Department of General Psychology, Padova University, Italy. Giuseppe Riva Catholic University of Milan, Italy. Anna Spagnolli Department of General Psychology, Padova University, Italy.

EDITORIAL BOARD Mariano Alcañiz Raya: Universidad Politecnica de Valencia. Valencia, Spain. Cristian Berrío Zapata: Pontificia Universidad Javeriana. Bogotà, Colombia. Rosa Baños: Universidad de Valencia, Valencia, Spain. David Benyon: Napier University, Edinburgh, United Kingdom. Cristina Botella: Univeritat Jaume I. Castellón, Spain. Antonella de Angeli: University of Manchester. United Kingdom Jonathan Freeman: Goldsmiths College, University of London. United Kingdom. Christine Hine: University of Surrey. Guildford, United Kingdom. Christian Heath: King's College. London, United Kingdom. Wijnand Ijsselsteijn: Eindhoven University of Technology. Eindhoven,The Netherlands. Giulio Jacucci: Helsinki Institute for Information Technology. Helsinki, Finland. David Kirsh: University of California. San Diego (CA), USA. Matthew Lombard: Temple University. Philadelphia (PA), USA. Albert "Skip" Rizzo: University of Southern California. Los Angeles (CA), USA. Ramakoti Sadananda: Rangsit University. Bangkok, Thailand. Angela Schorr: Universität Siegen. Siegen, Germany. Paul F.M.J. Verschure: Universitat Pompeu Fabra. Barcelona, Spain. Alexander Voiskounsky: Moscow State University. Moscow, Russia. John A Waterworth: Umeå University. Umea, Sweden. Brenda K. Wiederhold: Interactive Media Institute-Europe. Brussels, Belgium.

CONSULTING EDITORS Hans Christian Arnseth: University of Oslo. Olso, Norway. Marco Casarotti: University of Padova. Padova, Italy. Roy Davies: Lund University. Lund, Sweden. Andrea Gaggioli: Cathilic University of Milan. Milan, Italy. Pietro Guardini: Padova University. Padova, Italy. Frode Guribye: University of Bergen. Bergen, Norway. Raquel Navarro-Prieto: Universitat Oberta de Catalunya. Castelldefels, Spain. Stephan Roy: Hospital Sainte Anne. Paris, France. Carlos Ruggeroni: National University of Rosario. Rosario, Argentina.

EDITORIAL ASSISTANTS Fabiola Scarpetta, Teresa Tona: University of Padova. Padova, Italy.

PSYCHNOLOGY JOURNAL, PNJ PUBLISHED ON-LINE SINCE SUMMER 2002 W EB SITE: HTTP://WWW.PSYCHNOLOGY.ORG SUBMISSIONS: [email protected]

110

TABLE OF CONTENTS

Editorial Preface ………………….

p. 113

SPECIAL ISSUE: Mixed Realities in the Urban Environment The Use of Virtual and Mixed Reality Environments for Urban p. 119 Behavioural Studies………..................................................................... Andrew J. Park, Thomas W. Calvert, Patricia L. Brantingham, Paul J. Brantingham

Tags and the City…..……………………………………………………….. p. 131 Minna Isomursu

Experiences of Evaluating Presence in Augmented Realities………………………………………………………………………. p. 157 Rod McCall, Anne-Kathrin Braun

Experience Design for Interactive Products: Designing Technology p. 173 Augmented Urban Playgrounds for Girls................................................ Aadjan van der Helm, Walter Aprile, David Keyson

Other Contents Decoding Cognitive States from fMRI Data Using Support Vector Regression……………………………………………………………………. p. 189 Maria Grazia Di Bono, Marco Zorzi

Contrasting the Effectiveness and Efficiency of Virtual Reality and p. 203 Real Environments in the Treatment of Acrophobia............................... Carlos M. Coelho, Carlos F. Silva, Jorge A. Santos, Jennifer Tichon, Guy Wallis

111

112

Editorial Preface Urban mixed realities represent a growing and exciting area of research, which requires new ways of thinking about issues such as usability, place and presence. Urban situations are dynamic and can change rapidly, with a vast array of complex and exciting rhythms. They cover a whole spectrum of complex and chaotic happenings which span organisational and material configurations. These characteristics are both challenges and motivations for exploring mixed reality technology solutions, in particular with respect to finding methods to improve the ways in which participants can relate to the environment and to others. Mixed realities cover all situations in which digital objects are combined with physical features of the environment. Technologies include pervasive, ubiquitous, multimodal, and augmented reality solutions. Current projects explore and evaluate forms of interaction and presence in urban environments which use mixed reality technologies to improve or create new practices. This can be achieved in two ways: either by augmenting the engagement with others (including encounters, feelings, exchanges, coexperiences) or through augmenting the engagement with the environment (places, or things), which includes playing, understanding and interpreting the environment in new ways. This special issue originates from two preparatory events we co-organised at conferences, both of which created the context to discuss design, experience, and mixed reality technology for the urban environment. A panel at the Presence 2007 conference “Urban Mixed Realties: Challenges to the Traditional View of Presence” (McCall et al. 2007) created an engaging discussion on the need to find new perspectives on media technology and human experience. The discussion had as one of the starting points to better comprehend how available information

technology is expanding beyond virtual reality to mixed reality, tangible and ubiquitous computing with implications on user experience phenomena and how to study them. Using concrete cases it was discussed how new perspectives should oppose and go beyond the traditional Presence perspectives with its Cartesian dualism (Floridi 2005) and passive view of subjects putting forward constructivist and relational conceptions of human perception (Zahorik an Jenison 1998, Flach and Holdem 1998, Mantovani and Riva 1999, Sheridan 1999, Mantovani and Riva 2001). For example, instead of being and experiencing as a passive subject influenced by external stimuli, a person is assumed to be an active, intentional actor in an environment that offers different resources for actions. In connection to this discussion emerging themes were debated like the growing interest in ”places” instead of ”spaces” – while spaces are neutral and dimensional, places are personally, socially and culturally significant (Spagnolli and Gamberini 2005, Turner and Turner 2006). Following the Presence 2007 panel we organised a workshop at CHI 2008 “Urban Mixed Realities: Technologies, Theories and Frontiers” (McCall et al. 2008) In this workshop 16 presentations of work were discussed which illustrated a variety of approaches and interpretations currently connected to urban environments and mixed reality. The workshops participants contributed to the discussion about what Urban Mixed Realities can be in terms of technologies, experiences, and applications. In particular the workshop aimed to explore which fields and disciplines from science, technology and art should we draw upon and the diversity of backgrounds of participants contributed to review of which approaches are suitable in the

113

development of urban mixed realities. Also as participants proposed concrete examples it was possible to reflect on what mixed reality systems are and what are the challenges faced by such systems (for example usability, technical and other issues). The review of different applications also served to discuss the potential personal and societal benefits of urban mixed realities. The aim of this special issue is to continue the work from these two events and to document some of the diverse examples of research in this rapidly growing area with the aim of drawing attention to these. Moreover we aim to contribute to establishing a longer term thread of published works on this subject that illustrate applications, systems, designs, experience studies, theories and the research agenda. The title to this special issue uses an original and open concept, Mixed Realities (MR), to refer to experiences and environments, therefore the concept is not merely technological or referring to human experience. The decision to use this concept means that it is not just a development or extension of virtual reality but a field of its own with roots as strong as in other fields such as ubiquitous tangible computing. In this sense the concept of Mixed Reality with a singular noun is very precise and refers to a well defined area and community that aims to extend the virtual reality paradigm. The urgency of the topic is on the one hand due to the availability of technologies that enable to combine physical and digital, real and virtual aspects. On the other hand the urgency arises due to the fact that urban environments are very promising for research as they include complex socio-material situations that provide both opportunities for applications but also higher development challenges when compared to developing and

114

undertaking research in labs and offices. To better chart the research challenges we describe how we have structured it within an European integrated project which operates within the field of urban mixed realities. IPCity (www.ipcity.eu) is an EU funded project which explores mixed reality systems to support presence and interaction in urban environments. From a technological perspective this includes developing new hardware and software platforms for use in socially dynamic settings. The objective of these systems is to easily weave into everyday practices using pervasive solutions such as mobile phones, wearable systems, architectural solutions and interventions. The project also explores the development of authoring tools so that developers and end users can easily create, modify and store content. In turn this leads to a number of interesting theoretical questions regarding the use, construction and interpretation of urban spaces. The IPCity project contributes also by defining application areas: • Urban planning applications using tangible and augmented reality interfaces (Macquil et al, 2007) • Large-scale events where interaction between people through the use of multi-touch displays and mobile devices (Peltonen et. al, 2008) • Augmented reality games which let users experience a city in the past, present and future (Herbst et. Al 2008) • Story telling applications which let users create and experience stories of others via online and pervasive applications (McCall et. Al 2007) From the perspective of themes such applications cover a whole range of topics, many of which are also reflected within the papers presented in this special issue. These themes include: spatial, temporal, ambient and material aspects all of which play a role in shaping the users, designers and

evaluators views of urban mixed realities. Spatial aspects such as scale – working with objects of different scale, changing scale and dimension – as well as borders and layers. MR technologies can be used for changing the scale of virtual objects and for making invisible objects (borders, archaeology, infrastructure) visible. Temporal aspects that support the experience of presence are of relevance for an urban situation, such as making traces of the past visible, envisioning future development or the evolution of an event. Urban rhythms play a large role in experiencing a city, such as differences between day and night as well as flow and movement (of people and traffic). Materiality. In MR environments people engage with material and immaterial (virtual) aspects. The material environment is not only rich in awareness cues, material aspects also contribute to engaging the capacity of objects to absorb people’s attention, thereby increasing their engagement with each others and the world. They are also sources of ‘reality’ and ‘haptic directness’. Ambience. The urban experience includes all forms of sensations about the surrounding environment. It is a notion crossing feeling (physicalmaterial dimension), perception (human interpretation) and life experience (social interaction). It is both a subjective and collective notion. All sorts of elements participate in the creation of the overall feeling. Social aspects. Social interaction and scenes contribute to the experience of urban settings. The social life in urban setting is constantly changing. Culture and experience of people using and populating urban space is hidden and just partially inscribed in the material scene and only partially visible in the embodied performances of everyday (shopping, going to work) and extraordinary (demonstrations).

The four articles selected in this special issue are complementary for different reasons. First of all they are different in terms of theme including aspects such as interaction design cycles, subject studies, application and systems reports, and evaluation approaches. Moreover the articles all embody different perspectives and agenda and finally represent distinct application areas. However they all explicitly address mixed reality technology and experiences in the urban environment. In their article “Experience design for interactive products: designing technology augmented urban sport facilities for girls” Aadjan van der Helm, Walter Aprile, David Keyson contribute with interesting insights on a interaction design project that addresses tangible computing for augmenting urban sports. An exploration of the interesting interface technologies is accompanied by rich description of interaction design processes involving users and aims to support social issues. The article by Andrew J. Park, Thomas W. Calvert, Patricia L. Brantingham, and Paul J. Brantingham “The Use of Virtual and Mixed Reality Environments for Urban Behavioural Studies” shows how virtual and mixed reality technologies can be used to study urban settings phenomena such as “fear” but in a laboratory setting. 3D virtual/mixed reality models of the realworld environments are constructed in the laboratory and experiments are run with real human subjects in these environments. While on one hand the environment can be controlled in the experiments, on the other hand, the set up must ensure “suspension of disbelief” so that phenomena to be observed. In “Tags and the City” Minna Isomursu et. al investigates a specific technology for mixed realities namely NFC (near field communication) tags in urban environments. The analysis of this technology is supported by field

115

trials and aims to consolidate the requirements and challenges of using such technology. The paper by Rod McCall and AnneKathrin Braun “Experiences of Evaluating Presence in Augmented Realities” not only shows an original application of Mixed Realities in urban settings but also provides insights into evaluation approaches including theories and methods. The article presents the concept of “Unified experience”. An original game “Timewarp” from the perspective of the game design is presented along with results of user trials and how these should shape a future evaluation approach. References Floridi, L. (2005). The Philosophy of Presence: From Epistemic Failure to Successful Observation. Presence, 14(6), 656-667. Flach, J.M., & Holden, J.G (1998). The Reality of Experience: Gibson’s way. Presence, 7(1), 90-95. Herbst, I., Braun, A, Broll, W., & McCall, R. (2008, September). TimeWarp: Interactive Time Travel with a Mobile Mixed Reality Game. Paper presented at Mobile HCI 2008, Amsterdam, Netherlands. Maquil, V., Psik, T., Wagner, I., & Wagner, M. (2007). Expressive Interactions Supporting Collaboration in Urban Design. In Proceedings of GROUP 2007 (pp. 69-78). New York, USA: ACM Press. Mantovani, G., & Riva, G. (1999). “Real” presence: How Different Ontologies Generate Different Criteria for Presence, Telepresence, and Virtual Presence. Presence, 8(5), 540-550. Mantovani, G., & Riva, G. (2001). Building a Bridge between Different Scientific Communities: On Sheridan’s Eclectic Ontology of Presence. Presence, 10(5), 537-543. McCall, R., Wagner, I., Kuutti, K., & Jacucci, G. (2007). Urban Mixed Realities: Challenges to the Traditional

116

View of Presence. In Moreno, L., & Starlab Barcelona S.L. (Eds.), Proceedings of Presence 2007 (pp.159166). Barcelona, Spain: Starlab Barcelona. McCall, R., Wagner, I., Kuutti, K., Jacucci, G., & Broll, W. (2008). Urban Mixed Realities: Technologies, Theories and Frontiers. In Proceedings of CHI2008, Extended Abstracts on Human Factors in Computing Systems (3973-3976). New York, USA: ACM Press. Peltonen, P., Kurvinen, E., Salovaara, A., Jacucci, G., Ilmonen, T., Evans, J., Oulasvirta, A., & Saarikko, P. (2008). "It's mine, don't touch": Interactions at a large multitouch display in a city Center. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI 2008) (pp. 1285-1294). New York, NY: ACM Press. Sheridan, T.B. (1999). Descartes, Heidegger, Gibson, and God: toward an Eclectic Ontology of Presence. Presence, 8(5), 551-559. Spagnolli, A., & Gamberini, L. (2005). A Place for Presence. Understanding the Human Involvement in Mediated Interactive Environments. PsychNology Journal, 3(1), 6-15. Turner, P., & Turner, S. (2006). Place, Sense of Place, and Presence. Presence, 15(2), 204-217. Zahorik, P., & Jenison, R.L. (1998). Presence as Being-in-the World. Presence, 7(1), 78-89.

Rod McCall Fraunhofer FIT Schloss Birlinghoven, Sankt Augustin Germany

Giulio Jacucci Helsinki Institute for Information Technology Finland

Wolfang Broll Fraunhofer FIT Schloss Birlinghoven, Sankt Augustin Germany

The issue of PsychNology Journal also includes two more papers. The first one is “Decoding Cognitive States from fMRI Data Using Support Vector Regression” by Maria Grazia Di Bono and Marco Zorzi; it concludes the series of works presented at CHItaly 07 and selected for publication because of the high scores received by the conference reviewers. The authors report on a decoding method that could be used to achieve the ultimate goal of correlating fMRI data to cognitive states. In the case discussed in the paper, the cognitive task consisted of a game played in a virtual environment; the results are flattering with respect to the possibilities opened by the Support Vector Machine technique described. The second paper is entitled “Contrasting the Effectiveness and

Efficiency of Virtual Reality and Real Environments in the Treatment of Acrophobia” by Carlos M. Coelho, Carlos F. Silva, Jorge A. Santos, Jennifer Tichon, Guy Wallis. The paper deals with virtual reality treatment of phobias, addressing one central issue in this field, namely the comparison between a virtual and a real exposure. The paper provides arguments for the similarity in the effectiveness of the two types of treatment in case of acrophobia, and suggests that Virtual Reality is nonetheless more advantageous given the shorter length of each session. The Editors-in-chief

117

118

PsychNology Journal, 2008 Volume 6, Number 2, 119-130

The Use of Virtual and Mixed Reality Environments for Urban Behavioural Studies Andrew J. Park , Thomas W. Calvert , Patricia L. Brantingham and Paul J. Brantingham School of Interactive Arts & Technology Simon Fraser University (Canada)

School of Criminology Simon Fraser University (Canada)

ABSTRACT Virtual/mixed reality 3D models of real-world environments can be used to run behavioural and other experiments with real human subjects, replacing the traditional approach where studies are conducted in physical environments. Use of the virtual/mixed reality environments can minimize problems related to feasibility, experimental control, ethics and cost, but care must be taken to ensure that the environments are immersive and create “suspension of disbelief”. In this position paper the issues involved are discussed and illustrated by a 3D virtual model of an urban environment that is being used to study the role of fear in pedestrian navigation. Keywords: Virtual Environment, Urban Environment, Pedestrian Navigation, CPTED, Fear of Crime. Paper Received 01/06/2008; received in revised form 27/07/2008; accepted 07/08/2008.

1. Introduction

As an alternative to studies in physical environments, three dimensional virtual or mixed reality models of real-world environments can be used to study how built urban environments influence human behaviour. Experimental studies can be conducted with human subjects and the experimenter has much more control over the environment than in a physical experiment. On the other hand, care must be taken to ensure that the virtual or mixed reality environment is sufficiently immersive to ensure “suspension of disbelief”. This paper discusses the issues involved and describes a case study Cite as: Park, A., Calvert, T., Brantingham, P. L., & Brantingham, P.J. (2008). The use of virtual and mixed reality environments for urban behavioural studies. PsychNology Journal, 6(2), 119 – 130. Retrieved [month] [day], [year], from www.psychnology.org. Corresponding Author Dr. T. Calvert, Simon Fraser University, #250 – 13450 102nd Avenue Surrey, BC, Canada, V3T 0A3 E-mail: [email protected]

119

A. Park, T. Calvert, P.L. Brantingham, P.J. Brantingham

where a 3D virtual model of an urban environment was used to study the role of fear of crime in a pedestrian navigation model.

2. Virtual and Mixed Reality Models in Social Science Research

There are many instances where social science researchers wish to study how human subjects react to physical environments in general and urban built environments in particular. These studies can be behavioural, sociological or criminological. They could also involve urban planning, landscape architecture or building design. The common need is to study some aspect of how human subjects behave in a built environment. Virtual or mixed reality environments where the real-world is simulated provide new investigative tools for social science research. The advantages of using such environments include: •

Control - it is much easier to control and modify situations in a virtual

environment than in a real-world environment; also, in real publicly accessible physical environments it can be difficult to avoid interference from vehicles and pedestrians not involved in the study; •

Safety - by using a virtual environment we can avoid any real danger, harm,

or risk to human subjects but still achieve many of the dynamics of a real-world environment. •

Cost - building a virtual environment can be less expensive than building or

arranging access to a real-world environment and it can be relatively fast and inexpensive to modify the environment. These advantages must be balanced against the concern that the virtual or mixed reality environment will not provide sufficient immersion for suspension of disbelief. This immersion depends on the technical quality of the virtual environment and on the nature of the tasks the subjects are asked to perform. The more engaging the task, the more likely it is that the subject will achieve suspension of disbelief. There is also the issue of how “real” the mixed or virtual environment should be? For example, computer models of buildings and streets tend to be unrealistically clean. It may be that a mixed reality environment where imagery of the real physical environment is combined with computer models of buildings and people can provide a useful compromise.

120

The Use of Virtual and Mixed Reality Environments for Urban Behavioural Studies

Computer models and virtual environments have only begun to be used for research on urban environments. One tool that we have found was developed by a company in the Netherlands (Oxley et al, 2005). This tool focuses on visibility problems in a model environment. Another study by Cozens, Neale, Whitaker, and Hillier (2003) involved safety concerns at Welsh railway stations. Photography was used to create several 360 degree panoramas at various points in and around the stations. These photographs were then stitched together to create a VR walk-through. Participants can virtually navigate the station environment, freely zooming in and out and panning left and right at any time during the navigation. This VR system provided a dynamic visual stimulus based on which participants' judgments and perceptions were generated. Motivated by of the increasing interest in simulating our man-made environment, especially cities, the IEEE Computer Graphics and Applications magazine has published a special issue on “Procedural Methods for Urban Modelling” (Watson & Wonka, 2008). This special issue captures a good snapshot of work in this emerging area and includes a paper by Mol, Jorge and Couto (2008) on the use of a game engine as the basis for a tool to simulate evacuation planning.

3. Case Study: The Role of Fear of Crime in Pedestrian Navigation

A study using a virtual environment to investigate the role of fear of crime in pedestrian navigation is summarized here as a case study. The approach presented illustrates how a relatively simple and inexpensive VE can be used to study human behaviour in an urban environment.

3.1 Background: Crime Prevention Through Environmental Design Human-constructed city environments play an important role in peoples’ lives. The influence of an urban environment on human behaviour and life style has become a popular research topic. One of the pillars of research on people in urban environments, Jane Jacobs, carefully observed people and their activities in New York City. She identified many different urban planning problems, including crimes, caused by bad urban design (Jacobs, 1961). Her work has greatly influenced urban planners and architects and caused them to see cities in a new perspective. Another person who also observed people in New York was William Whyte, who was known as “people watcher”. The behaviour of people was systematically recorded

121

A. Park, T. Calvert, P.L. Brantingham, P.J. Brantingham

using still cameras, movie cameras, and notebooks over a long period of time. His work shows architects and urban planners what works and what does not (Whyte, 1980). Following these ideas, criminologists began to think of new ways of reducing crime in urban environments. Criminologist C. Ray Jeffery used the phrase, “Crime Prevention Through Environmental Design (CPTED)” (Jeffery, 1971) for the first time. The idea of CPTED is that careful design of environments can reduce possible crimes and fear of crime so that it improves quality of life. Another strategy called the “broken windows” theory was added to CPTED (Kelling & Coles, 1996). This theory states that if people do not fix broken windows, more windows will be broken. And the situation gets worse with worse crimes occurring, including breaking into buildings. The relation between crimes and the environment was further elaborated by Paul and Patricia Brantingham who pioneered a new field, called “Environmental Criminology” (Brantingham & Brantingham, 1981, 1997). By examining the time and the place where crimes occurred, the spatial pattern of the crimes and the behavioural patterns of the offenders can be discovered. There has been much research that supports the ideas of CPTED, particularly, the relationship between environments and fear of crime. Fear of crime is not necessarily fear of real dangers, but rather the perceived fear of being victimized by possible crimes. Criminologists and environmental psychologists have studied how people, particularly the most vulnerable, feel fear of crime in various environmental settings. Their findings show that narrow walkways without any escape routes, hidden spaces created by corners, tall bushes, and the presence of threatening individuals generate fear in people (Fisher & Nasar, 1995; Nasar, Fisher, & Grannis, 1993). Lighting is another element that can influence fear of crime (Hanyu, 1997).

3.2 Modelling the Role of Fear in Pedestrian Navigation We have developed a quantitative model of how pedestrians navigate through an urban environment that creates fear of crime. The model chooses a path through the environment that minimizes passage close to features known to generate fear. These are: •

Narrow passageways with no escape,

•

Passageways with hidden spaces off to the side that might hide a threatening individual,

•

Passageways with garbage dumpsters and other hiding places,

•

Passageways with a threatening individual,

122

The Use of Virtual and Mixed Reality Environments for Urban Behavioural Studies

•

Passageways with multiple threatening individuals.

At this stage the model does not handle combinations of these features that occur together (Park & Calvert, 2008). 3.3 Validating the Model In order to validate this model we needed to study pedestrian navigation using human subjects. However, it was difficult to find suitable physical locations that could be used to test our model. Fear of crime research has mainly been done by traditional methods such as surveys, interviews, case studies and experiments with human subjects. For example, some research has been done by asking human subjects to visit various sites on a university campus. The subjects were then asked to answer questionnaires regarding fear of crime and sense of security. (Nasar, Fisher, & Grannis, 1993). Other research has been done by showing human subjects photos of various city sites and asking them how they feel about the sites. The former kind of research could have endangered the human subjects. Because of the real possibility of danger, the research must be limited. The latter kind of research loses dynamics and a sense of real-life situations. Please note that in case of images no line-space needs to be used to separate the image from the footnote. On the contrary, in case of tables please use a line-space of 1.5 to separate the table from the footnote. 3.4 Using a VE to Study Pedestrian Navigation Since observing real pedestrians in a fear-generating area is difficult due both to ethical issues related to the risk and danger involved in the experiment and to our inability to control experimental variables, it was decided to construct a virtual urban environment. This environment can achieve relatively good realism using textures from photographs of real buildings and objects on the streets. It is also easy to create and modify the layouts of a fear-generating area to correspond with the goals of the experiment. Control of experimental variables is straightforward in the virtual environments and adding animated human figures enhances the realism. Human subjects can freely navigate the virtual environments as if they were in the real-world environments. The design of our experimental setting was arranged to help the participants experience presence during the experiments and to suspend their disbelief. Originally we wanted to make an accurate model of a well-known fear-generating area. Using a

123

A. Park, T. Calvert, P.L. Brantingham, P.J. Brantingham

3D modeling tool (3D Studio Max), we created 3D buildings and roads in accordance with satellite pictures and road networks from Google Map. We then visited the area and took photographs of the buildings. It was difficult to take perfect pictures without any interfering objects such as pedestrians, trees, cars, or street lights in front of the buildings. Different light densities due to the weather changes were another problem. We also tried to avoid shadows on the buildings by taking photographs on cloudy days. However, we had to spend quite a lot of time editing the photographs in Adobe Photoshop, removing unnecessary objects and shadows from the buildings and adjusting the brightness. We then mapped these textures onto the 3D models of the buildings created in 3D Studio Max. A layout of the streets was designed in which we could create fear-generating features such as narrow walkways, hidden spaces, bushes, garbage containers and so on. The

VE

was

controlled

by

a

game

engine

–

Dark

Basic

Professional

(http://darkbasic.thegamecreators.com/). The game engine architecture provides many of the features needed to navigate a VE. These include: •

User control of navigation.

•

Support for complex 3D models.

•

Support for fast rendering of the scene.

•

Collision Detection.

•

Ability to easily implement a vision system.

Figure 1. A large screen and surrounding curtains increases the sense of immersion.

124

The Use of Virtual and Mixed Reality Environments for Urban Behavioural Studies

Several pilot studies with different experimental set ups were conducted. We tested an Elumens Vision-Station with a few human subjects. Although it generated a good immersive experience, most of the subjects complained about dizziness during the navigation. We then tried a regular screen with a projector. Subjects sat down in front of the screen and navigated the VR environment with a keyboard and a mouse. But this was not immersive enough for them to feel presence. Some people felt that they were playing a game. To increase the level of immersion, we set up a large screen that reached to the floor (about 5m x 4m) so that participants could feel better presence with the big display size, the wide field of view, and the wide field of regard. To overcome subjects' feeling that they were playing a game with a keyboard and a mouse, we used a Nintendo Wii remote controller so that subjects could stand in front of the big screen and navigate the virtual environment intuitively as if they were walking in the street (Figure 1). To help participants feel even more presence, we surrounded the experimental space with thick black curtains. We also played ambient background sound of traffic during the experiments.

Figure 2. Views of the five Decision Points.

As we were developing this virtual environment, we realized that we were reaching the performance limits of the available computer hardware – it became difficult to achieve a high enough frame rate with the increasing complexity of the model buildings and the large number of different textures. Eventually, we decided to develop a reduced model of the area with the existing textures so that the participants could still feel that they are in the fear generating target area. Since we were not creating an

125

A. Park, T. Calvert, P.L. Brantingham, P.J. Brantingham

exact model, we modified the environment to set up Decision Points for certain environmental features (wide vs narrow streets, hidden spaces vs no hidden spaces, garbage dumpsters vs no garbage dumpsters, threaten individual vs no threatening individual, multiple threatening individuals vs one threatening individual). Figure 2 shows snapshots of each decision point. As shown in Figure 3, human subjects can navigate the environment from a first person’s point of view as in first person shooter games using a Nintendo Wii controller for navigation.

Figure 3. A subject navigates the environment using a Nintendo Wii controller.

Experiments were conducted in which human subjects navigated from a starting position to a destination position in the VE. On the way, there were the five decision points. Subjects judged for themselves which route they should choose. The results from our initial experiments are very promising. Most importantly, human subjects felt as if they had indeed been in the target area and behaved accordingly.

126

The Use of Virtual and Mixed Reality Environments for Urban Behavioural Studies

Figure 4. Age and gender distribution of 59 subjects.

Figure 4 shows the demographic statistics of the subjects' age and gender. In contrast to some such experiments, subjects were recruited from the community at large, not just from university students. All subjects provided informed consent. The results for the 59 subjects are summarized in Table 1.

Decision Point

Shorter Route

1

Narrow Alley

31%

Wide Alley

69%

2

Hidden Space

68%

No Hidden Space

32%

3

With Dumpsters

32%

No Dumpsters

68%

4 5

One Threatening Individual Multiple Threatening Individuals

Longer Route

25% 53%

No Threatening Individuals One Threatening Individual

75% 47%

Table 1. Routes chosen by subjects at each decision point.

In summary, the subjects chose the wide passageway (62%) more than the narrow one (38%), the street without garbage dumpsters (68%) more than the one with dumpsters (32%), and the street without a threatening individual (75%) more than the street with one (25%). The fact that most subjects avoided the street with a threatening individual suggests that social incivilities generate more fear than physical incivilities. The results at the decision points #3 and #5.

127

A. Park, T. Calvert, P.L. Brantingham, P.J. Brantingham

In post-experimental interviews, we reviewed the screens recorded during the experiment, questioned subjects about their levels of fear and asked them which features in the environment triggered the fear.

4. Lessons Learned from the Case Study

A number of useful lessons can be learned from this case study: •

Relatively simple and inexpensive VE’s can be effective for behavioural and

other studies in a built environment. •

The VE layout does not have to be identical to the target physical environment

in order for subjects to feel that they are present in the target environment. •

The layout of the VE can be structured to provide subjects with decision points

appropriate to the issues being studied. •

A game engine provides a base system that is quite appropriate for creation of

the VE. Game engines support navigation, rendering, collision detection and other features need to navigate the VE. •

Creating realistic detail in the scenes can be tedious and time consuming – for

example, it is much quicker and easier to create images of clean streets rather than dirty streets with miscellaneous garbage. However, the experimental subjects were quite tolerant of the lack of such detail in our scenes. •

More realistic scenes can be created economically by using digital

photography to create panoramic background views photographed from the physical environment. Then a mixed reality (MR) environment can be created by adding computer generated objects and characters in the foreground.

5. Conclusions

The use of a virtual environment to test a model of human behaviour in an urban environment has many advantages in terms of cost, time, flexibility and safety. Our studies show that a VE can also be an inexpensive and effective alternative to realworld environments. This approach promises to be particularly useful for urban planners and criminology researchers.

128

The Use of Virtual and Mixed Reality Environments for Urban Behavioural Studies

6. Acknowledgments

Support from the Institute for Canadian Urban Research Studies (ICURS) and from Simon Fraser University is gratefully acknowledged. The authors also acknowledge substantial input and advice from Professor Greg Jenion, Kwantlen Polytechnic University.

5. References

Brantingham, P.J., & Brantingham, P.L. (1981). Environmental Criminology. Beverly Hills California: Sage Publications. Brantingham, P. J., & Brantingham, P. L. (1997). Understanding and Controlling Crime and Fear of Crime: Conflicts and Trade-Offs in Crime Prevention Planning. In Lab S. P. (Ed.), Crime Prevention at a Crossroads (pp. 43-60). Cincinnati: Anderson Pub. Cozens, P., Neale, R., Whitaker,J., & Hillier, D. (2003). Managing crime and the fear of crime at railway stations-a case study in South Wales (UK). International Journal of Transport Management, 1(3), 121-132. Fisher, B., & Nasar, J. L. (1995). Fear Spots in Relation to Microlevel Physical Cues Exploring the Overlooked. Journal of Research in Crime and Delinquency, 32(2), 214-239. Hanyu, K. (1997). Visual Properties and Affective Appraisals in Residential Areas After Dark. Journal of Environmental Psychology, 17(4), 301-315. Jacobs, J. (1961) The Death and Life of Great American Cities. New York: Random House. Jeffery, C.R. (1971). Crime Prevention Through Environmental Design. Beverly Hills, California: Sage. Kelling, G.L., & Coles, C.M. (1996). Fixing Broken Windows: Restoring Order and Reducing Crime in Our Communities. New York: Martin Kessler Books. Mól, A., Jorge, C., & Couto, P. (2008). Using a Game Engine for VR Simulations in Evacuation Planning. IEEE Computer Graphics and Applications, 28(3), 6-12. Nasar, J.L., Fisher, B., & Grannis, M. (1993). Proximate Physical Cues to Fear of Crime. Landscape and Urban Planning, 26(1-4), 161-178.

129

A. Park, T. Calvert, P.L. Brantingham, P.J. Brantingham

Oxley, J., Reijnhoudt, P., van Soomeren, P., Beckford, C., Jongejan, A., & Jager, J. (2005). Crime Opportunity Profiling of Streets (COPS): A Quick Crime Analysis – Rapid Implementation Approach. Garston, Watford: BRE Bookshop for BRE. Park, A., & Calvert, T. (2008). A Social Agent Pedestrian Model, Computer Animation and Virtual Worlds, 19(3-4), 331-340.Whyte, W.H. (1980). The Social Life of Small Urban Spaces. Washington, D.C.: Conservation Foundation. Watson, B., & Wonka, P. (2008). Procedural Methods for Urban Modeling [Special Issue]. IEEE Computer Graphics and Applications, 28(3), 16-17.

130

PsychNology Journal, 2008 Volume 6, Number 2, 131-156

Tags and the City Minna Isomursu VTT Technical Research Centre of Finland (Finland)

ABSTRACT This paper analyzes the findings of a set of field studies that explored the use of near field communication (NFC) tags in a mixed reality environment for providing access to digital services by touching a tag with a mobile phone. The field studies provide insight into user experience, usability, user acceptance and technical implementation issues that need to be considered when designing tag-based services. The paper proposes that if NFC technology becomes common, there is a compelling need for methods and practices for tag management. If such practices are not used and available, tags can form “tag litter” that ruins the user experience by corrupting the trust towards tags and tag-based services. Keywords: mixed reality, NFC, tags, physical browsing. Paper Received 30/05/2008; received in revised form 05/08/2008; accepted 08/08/2008.

1. Introduction

The near-field communication (NFC) and RFID (radio frequency identification) technologies are finding their way into our everyday lives, for example, in ticketing applications (Card Technology Today, 2005) and payment solutions (Ondrus & Pigneur, 2007). The offerings of NFC-enabled devices are expected to increase slowly but steadily during upcoming years. For example, ABI research (ABI research, 2007) estimates that over 20% of all handsets sold globally will be NFC compatible by 2012. The first NFC standards were released by ISO/IEC in 2003, and there are several ongoing standardization efforts for tackling the issues related to wider adoption of NFC into versatile domain areas.

Cite as: Isomursu, M. (2008). Tags and the City. PsychNology Journal, 6(2), 131 – 156. Retrieved [month] [day], [year], from www.psychnology.org.

Corresponding author: Name Minna Isomursu Address VTT, Kaitoväylä 1, FIN-90570 OULU, Finland E-mail: [email protected]

131

M. Isomursu

The research presented here has been done within the context of the SmartTouch project (http://www.smarttouch.org). The project explores the possibilities of touchbased user interaction enabled by NFC technology (see Figure 1) through experimental field studies.

Figure 1. With NFC tags, users can access mobile services and content just by touching a tag placed in a physical space with their mobile phone.

In this paper, the findings of the field studies are analysed from the viewpoint of how to manage a large amount of tags that are distributed in an urban space. Without tag management, there is a danger that tags will create the problem of “tag litter” that only increases the complexity and information overflow in our everyday environment instead of supporting our lives with a simple interaction mechanism. Tags can provide support in our everyday life activities by establishing a bridge between the physical and the digital worlds when they are ubiquitous in the everyday environments of users. This requires a large amount of tags distributed in different types of spaces – public and private, outside and inside, urban and rural, etc. The tags become an integral part of physical space, altering the way humans perceive and behave in it. Tags are also components of digital space and act as components of larger service systems that need to be maintained and monitored. As tags can be numerous and distributed into a geographically wide and versatile area, and as parts of complex networks of services, their management is challenging. Tag management requires digital tools that can be implemented as part of tag management platforms.

132

Tags and the City

This paper discusses the requirements and problems of tag management that have arisen in the field trials undertaken in the SmartTouch project.

2. Bridging the Physical with the Digital

Digital services provide us with support and enrich our everyday lives. Access to digital services and applications can be provided by embedding technology into our everyday surroundings so that we can reach the digital world and its services whenever needed. This paper examines how tags can provide service and content access points for a mobile user in versatile environments to access and interact with the digital world. 2.1 Physical Browsing Physical browsing (Ailisto et al., 2006) allows connecting the physical world with the digital world via tags. Tags can be used for direct access to the services and information without the need for browsing menus, typing addresses or using search engines. The physical browsing paradigm eliminates the need to type URLs or keywords for accessing services. This is especially beneficial in mobile contexts, as the effort needed to enter a word on a cell phone keypad is more than double the effort required to enter the same word on a full QWERTY keyboard (Kamvar & Baluja, 2006). NFC technology realises physical browsing with touch-based interaction (Rukzio et al., 2006). Initiating action with an NFC tag requires a close range contact with the tag and the reader (approximately 5 centimeters). By integrating the NFC reader with a mobile phone or other mobile device, NFC technology enables a touch-based user interface for ubiquitous access to mobile internet. The tag can be embedded, for example, in a sticker, that can be attached into objects or surfaces – virtually anywhere. The tag contains the information that is required to access the digital service, for example, the URL that is accessed when the user touches the tag with her mobile phone. The location and context of the tag should provide all necessary information that helps the user understand what services the tag offers. Physical browsing provides a promising intuitive user interface solution to the problem of identifying and activating digital services in a space (problems are described, for example, in Lindenberg, Pasman, Kranenborg, Stegeman, & Neerinex, 2007). Touchbased interaction is fast to use, and it is easy to integrate into physical objects and structures. Tags are cheap and they are easy and fast to program. NFC tags can be

133

M. Isomursu

programmed with a normal mobile phone with NFC functionality. All this provide possibilities for end users, communities, and commercial service providers to adopt the technology and contribute to the creation of a ubiquitous network of tags that can be used for accessing mobile internet and related services with a touch. 2.2 NFC Technology NFC is a short-range wireless technology that allows electronic devices to exchange data upon touching. NFC combines both read and write modes into the same device. It is also capable of receiving and transmitting data at the same time. NFC standards have been built over existing radio frequency communication standards (e.g. RFID and smart card standards). It is also designed to be compatible with the old standards and infrastructures, which makes it versatile. (NFC forum, 2008) NFC can be used for emulating smartcard functionality. In smartcard mode, NFC should be able to transmit an ID of the device or tag to any compatible reader. However, perhaps a more lucrative scenario is to use an NFC-capable device as an active reader device, which receives data from a passive tag. In addition, NFC enables peer-to-peer communication, where two active NFC devices communicate by transmitting and receiving data from each other upon touch. In the research settings described here, the hypothesis is that the NFC reader capabilities will be widely integrated into mobile devices, such as mobile phones, in the future. Therefore, the data management requirements discussed here are based on an assumption that the users would use their mobile phone (or mobile computer) for accessing NFC-based services. In the experimental settings described here, the NFCenabled mobile device is a mobile phone, but the users participating in experimental pilots used the mobile phone usually solely for accessing NFC-based services and chose additionally to carry their personal mobile phone with them. Another assumption made for research purposes is that as the NFC-based mobile devices become common, the amount, variety and visibility of tags in our physical environment and surroundings will also grow. In the experimental settings described here, the amount and variability of tags was restricted to those that were designed and distributed by this one research project, as tags are not yet visible and widely used in our everyday environment.

134

Tags and the City

3. Research Setting

In the SmartTouch project, NFC-based applications have been implemented and piloted within several application domains in field settings. The goal has been to evaluate new technological solutions based on NFC with real users in a realistic use environment. The hypothesis was that this can provide information about problems and issues that can be expected with large-scale use in the real-world usage setting. The trials aimed at exposing the technology to real use under circumstances that can be observed and followed. Our trials aimed at making the effects of adopting new technology visible by creating realistic use situations where real-life user experience can happen, and where opportunities and problems related to realistic usage and behavior can be observed. Arranging a trial in a field setting requires that the technology under evaluation is mature enough that it can be used by real users in real usage environments, which was the case with the NFC technology, as all technical components were available on the market as commercial products. However, due to the novelty of the technology, the service concepts are not yet common and there is not much knowledge about user acceptance, usability and user experience related to NFC-based services. This paper contributes to existing knowledge by exploring how tag management affects service design and user experience. Even though the components needed for building the NFC-based service system are commercially available, the service where the technology is integrated can be part of an infrastructure or process that does not exist yet. The non-existing parts can be simulated during the trial. These can be, for example, payment systems or legislation. Simulating crucial parts of the service creates challenges for interpreting and analysing the results. They must be tackled case-by-case by careful research planning. For example, the absence of real payment action can have a profound effect on how users trust the service and what their expectations are for security and reliability, etc. The next subchapters briefly describe the trials used to derive the results of this paper. The first two field trials represent applications that use NFC technology for providing service access points in the everyday physical environment for accessing digital services. In these cases, the NFC tags are integral parts of service concepts. The last field trial represents a case where the tag provides fully autonomous functionality for accessing the content of mobile internet. In this case, the tag is not a component of a larger service system, but can operate in stand-alone mode.

135

M. Isomursu

3.1 Meal Ordering Application The first field study evaluated an NFC-based application for enhancing a mealordering service provided by the city of Oulu. It was for elderly users who were not able to prepare balanced meals by themselves because of various health conditions. This study represents a ubiquitous computing application in a private space in an urban environment. The application enhanced an existing meal delivery service by providing the elderly home-care clients with the opportunity to choose which meal they would receive the following day (more details in Häikiö et al., 2007). Without the application, the elderly homecare clients did not have the possibility to express their preferences and the same food was delivered to all clients. The goal of the enhanced service was to increase the quality of life of meal delivery service clients by: •

empowering them by providing more control of their lives by allowing them to choose their meals, and

•

ensuring the satisfaction of meals, which decreases the risk of malnutrition that has been found to be surprisingly common with the elderly (Pirlich & Lochs, 2001).

The user interface consisted of a meal menu that embedded NFC tags, and a mobile phone that was used for touching the tags to make meal selections (see Figure 2.). The meal selection was then delivered to the meal service providers. This application is an example of tags that are embedded inside the home of the user, therefore part of a private environment, but offering access to the digital services provided by external service providers.

Figure 2. Meal ordering application in use. Options A and B are alternative meal options, and C is used for cancelling meal delivery for the following day.

136

Tags and the City

The trial with the meal-ordering application was made with nine users for a period of eight weeks. Special research emphasis was directed towards analysing and ensuring the ethical consequences related to the pilot and the service (see Schulz & Hanusa, 1978 for reference). 3.2 Parking Payment Tags in public spaces can provide access to public services. A trial was done in the city center of Oulu for enabling mobile payments of parking fees. The parking payment could be processed by touching tags that were attached to lamp posts and parking meters available at designated parking areas. The user was then able to pay the parking fee by touching on arrival the tag attached to the car for identifying the car to be parked, then touching a tag nearby to start measuring parking time (shown in Figure 3.). When leaving a parking slot, only the tag in the car needs to be touched to end the parking time. In a parking house, the tags were placed at the arrival and departure gates. Parking payment started rolling upon entry and was stopped when the car exited the parking house. Traffic wardens also used an NFC-capable mobile phone to check that the car had been correctly tagged for parking. They simply touched the tag in the car to check that parking was correctly reported.

Figure 3. Parking payment is initiated by first touching a tag in a car, and then touching a tag on a parking meter. When leaving the parking place, the user touches the tag on the car to mark the end of parking time.

137

M. Isomursu

The parking pilot was used by approximately 50 users and four traffic wardens for a period of two months. In this setting, the payment action was simulated, i.e. the money transfer from the customer to the service provider did not actually happen. This had to be taken into account in interpreting and analysing the results. 3.3 Information Tags In the city of Oulu, NFC tags have been distributed throughout the urban area as “information tags”, which means that they are visible in public places, and can be used for accessing predefined mobile services. For example, touching one of the tags initiates a phone call to a taxi. Another information tag initiates access to mobile internet, and displays a view to local news through the local newspaper. An example of a banner that includes several grouped information tags is presented in Figure 4. Information tags have also been placed in specific places to provide information relevant particular to that place thus enabling access to location-based services and content. Examples include: •

a pub where clients can use information tags for browsing information about the selection of beer available

•

at the theatre for accessing information about play performances and their cast and background information

•

at the bus stop for accessing bus timetables and real-time information about arriving buses

Figure 4. Example of a banner displaying information tags. The blue icons indicate the area to be touched, and the text and icons explain the service accessed through the tag. The back of the banner is sticky, so it can be attached virtually anywhere.

Hundreds of users were able to use the information tags through a period of half a year. The amount of users and tags, as well as the selection of tags available at different times, varied throughout the experiment as information tags were usually a part of other trials.

138

Tags and the City

4. Research Methods

The research results presented here are based on the findings of field trials that used new NFC-based applications and services in settings that aim at as high an experimental reality (Aronson, 2004) as possible. This means, that the research conditions of the field trial are arranged so that all actors and practices of the application and the service chain would be as close to real as possible. As trials are done in a real world setting, the research setting is uncontrolled. However, the users are selected and recruited based on predefined criteria, and the duration of the experiment is restricted. Also, the users are committed to use the technology under evaluation during trial period, which means that there are constraints in choosing competitive technology alternatives. The limited duration of the trial had effects on the constructs developed. As the expected life time of the applications, services and special technology developed for the purpose of the experiments was relatively short, compromises were made, for example, on how the new technology was integrated with existing systems. The experiences related to the applications and services used in the experiments were collected from people experiencing the new technology in different roles. For example, in the elderly meal care experiment, experiences were collected and analysed from elderly users, meal delivery personnel and home care personnel (more details on methods in Häikiö et al., 2007). Different data collection methods were used in different cases, including interviews, questionnaires, web surveys and feedback, observations, diaries, etc. For each case, several different methods were used in different phases of the experiment (see summary in Table 1). The goal was to adopt data collection methods that would disturb the use of the technology as little as possible, but that would be able to capture user experiences when they occurred. This was very challenging as the technology and services were tightly integrated into the everyday lives of their users (Isomursu, 2008). In addition to the experiences of people directly using the piloted applications, an important data source for the requirements analyzed here was the constructive research made to implement the applications and services used in the experiments. They were designed and implemented by the research group consisting of professionals from different disciplines and presenting different actors in the service chain, such as mobile network operator, technology providers (tags, mobile devices with NFC readers, software platforms, etc.) and service providers.

139

M. Isomursu

Trial

n

Data collection

Meal ordering

9

before use

•

face-to face interviews

•

observation of training

•

diaries

•

observation: logs and support visits

after use

•

face-to-face interviews

follow-up study

•

face-to-face interviews

before use

•

paper questionnaires

during use

•

logs

•

web feedback forms

•

paper questionnaires

•

face-to-face interviews

before use

•

paper questionnaires

during use

•

logs

•

web feedback form

•

contextual observation

•

paper questionnaires

•

web questionnaire

•

face-to-face interviews

during use

Parking payment

55

after use

Information tags

238

after use

Table 1. Summary of data collection methods used in each trial. “n” indicates the number of trial users, and data collection methods list all methods used in a trial (some methods were used only for selected user groups).

In all trials, the trial users were voluntary users that represented the expected user group of the service. The users were requested to sign a written agreement to (1) use the trial services for the trial period, and (2) give feedback about their experiences through methods specified for each study. In addition, for the elderly meal care pilot, a research permission was sought from the management of the social services of the city of Oulu, who also participated in the field study planning by ensuring that the users were informed well enough about the practical details related to the research, and that the trial users were physically and mentally able to participate and understand the research context. In the elderly meal care study, special attention was given to ensuring that the trial did not cause negative effects in the trial participants. This was done, for example, by providing non-stop support and monitoring by researchers who had experience both with the technology and elderly care, ensuring that system failures would not leave users without meals, and with a follow-up study performed a year after

140

Tags and the City

the trial period. In other trials, the privacy of the participants was protected by making the collected data anonymous in a very early phase, which made it impossible to later identify the individuals behind tracked activities.

5. Requirements for Tag Management

Using tags as access points to various services and applications usually means distributing tags into an environment that cannot be strictly controlled. There can be a high number of tags. The tags must resist weather, wear-and-tear and vandalism. As the tags are components of both the physical and digital realm, they need to evolve as the physical and digital environments change. In this paper, various requirements and design considerations that need to be taken into account in managing tags in such environments are analysed. The requirements presented here may be solved at different levels of the service platform architecture: they may need tools for the personal device of the user, for the mobile network operator or for the back-end system of the service provider. As a result of analysing the pilots from the tag management viewpoint, a set of functional requirements was derived. In this paper, these requirements are described both by explaining the issues tackled and the problems faced during the experiment (i.e. the findings that led to deriving requirements for the specific functionality) and then by generalizing the requirements into more general requirements that are not specific to any individual use case and that aim to address generic problems. 5.1 Logs In every trial, the tag providers wanted to have visibility on the usage of the tags. Logs can be used for monitoring which tags are used most frequently, at what times of day or year, and sometimes even by whom (depending on the service). During the experiments, logs were used for research purposes for monitoring and analysing tag use. In real service environments, the tag providers would probably want to exploit logs for evaluating the operation of tags as a part of the service system. Log information can then be used, for example, for deciding where to optimally place the tags. Tag placement was found to be one of the most important factors affecting to the user experience of tags. Especially in public spaces the placement of tags can have an effect on how people locate themselves in the space, and even on how they interact

141

M. Isomursu

with each other. In the information tag pilot, some of the tags were placed on the walls of a theatre waiting hall. Our observations indicate that placing tags on the posters on the walls had an effect on how people placed themselves in the space. Normally, people gathered around the centre of the space, where the lighting was brightest and where most of the furniture was located. The space near the walls of the room was more dimly illuminated, and there was no furniture. Some users reported that they had to leave the group they were socializing with in the centre of the room to use the tags that were located on the posters on the walls. As tags can initiate different types of actions, one kind of log system does not fit all purposes. In the pilots, the needs for following kinds of logging tools arose: •

logging tag use in a mobile terminal

•

logs generated by mobile network operators

•

logs generated by the back-end service system

Logging Tag Use in a Mobile Terminal. Tags can initiate local action that only takes place between the tag and the tag reader, e.g. a mobile phone. For example, a tag can provide a predefined piece of information stored in the tag that is presented to the user via the user interface of a mobile phone, or the tag can request the phone to make a phone call to a predefined place. In these cases, the only way for generating automated logs about tag usage is to do it in a mobile terminal, as there is no connection to a back-end system or network during the transaction. As the vision of large-scale use of NFC assumes that people use their own mobile devices to interact with the tags, the service and application providers would probably not have access to logs stored into the memory of a personal device of the user. However, logs that would be generated in a mobile device and stored locally could be used for other purposes, such as for the end-users to track their own interactions, for example with a ticketing service, using a tag-based user interface. In the case of problem or error situations, the user may be willing to share the logs generated and stored in a personal mobile device with a service provider. For example, in the parking ticketing application the log stored in the mobile terminal could be used to prove that the user had touched a tag to start the parking period, if the system failure had caused problems in managing the transaction in later phases.

Logs Generated by Mobile Network Operators. When a tag initiates an action that is processed through a network operated by a network operator, activity logs can be

142

Tags and the City

automatically compiled by the operator monitoring the network traffic. The network activity initiated by tags can be monitored and logs can be compiled. Typical data items to be included in logs are the identification code of the tag, identification of the reader device and the time of activity. The user who initiated activity can be identified through the device used for reading the tag, e.g. a mobile phone. For example, cellular operators can compile logs that indicate which mobile phones have initiated actions in monitored tags. However, if the mobile terminals are connected with services through peer-to-peer networks or other networks that do not have centrally organised operators, compiling log data can become more difficult. When network operators compile logs about the use of tags, privacy issues may arise. As tags are always physically located in some specific place, using a tag does not only give information about the services used, but also reveals the location of the user. By combining the identification of the user with the location data provided by the tags and the observed service traffic, the network operators are able to generate personalised location-aware service profiles for users. However, all this information is available for the mobile network operators even without the logs related to tags. This problem becomes more serious if the networks operators provide logs and monitoring services for tag-based service providers. For example, the service provider (e.g. news providers) who provide mobile services through information tags could get information about when and where specific users accessed their services.

Logs Generated by the Back-End Service System. The third possibility is to monitor and log tag-generated actions in the service system that receives and processes the initiated action requests. These tools and systems can be specified, implemented and controlled by the service providers, so they can be tailored to their specific requirements. However, two challenges were identified in the pilots. First, it may be difficult to track error situations when the logs are generated only for the action requests that are received by the service provider. If the error in the taginitiated activity has taken place already in the tag-reading activity, or when the action request is processed through the network, the request may never reach the service provider in the first place. This means, that no logs can be generated by the service provider. This problem was faced in the elderly meal care pilot. Some meal orders were not received by the back-end system at all, and as the logs were created only by the back-end system, it was impossible to know where the error had occurred. With elderly meal-care service, the problem was especially severe, as failures in processing meal

143

M. Isomursu

orders might, in worst cases, result in users not receiving their meals. However, in the service discussed here, this was handled by monitoring the meal orders. If no meal order or cancellation was received, the user was called by telephone, and if not reached, meal option one was delivered as the default option. However, this is possible only with services that have regular customer relationships and default service behavior. With sporadic service usage this would be more difficult. Second, the service provider often has restricted access to the information they receive from the activity. For example, if the tag initiates an activity that opens a mobile internet site in a browser of the mobile phone, the service provider is unable to see who initiated the action. This adds to the privacy of the action, but decreases the service provider’s ability to generate logs. The information tag pilot used this mode of action. The action request processed via the tag can be programmed to include the identification of the tag, but not identification of the mobile terminal requesting the web page. This information is available only for the network operator. However, if the service is accessed through a service specific application that the user is required to use to access the service, the application can be programmed to send user or mobile device identification with the action request to the service provider. This procedure was used in the elderly meal care case. In these cases, the service provider can track taginitiated activities of individual users. 5.2 Tracking Tag Placement Distributing tags is very easy. Tags are relatively cheap, and programming a tag is a fast and simple procedure. A tag in a sticker can be attached to almost any surface. Therefore, the effort and cost related to distribute large amounts of tags into the environment is relatively low. However, when tags are used as a part of a digital service system, it becomes crucial to keep track of the location of the tags. This requires some kind of register of tags and their placement. The tag management platform should be able to manage and store the tag placement register. The simplest method for registering tag placement is to indicate the location of the tag with a street address or with GPS coordinates. However, as tags are very small, this level of granularity might not be detailed enough for specific maintenance purposes. For example, for locating a specific tag in a meal menu illustrated in Figure 2, a street address or a GPS coordinate obviously does not provide enough information for identifying an individual tag. In addition, tags are often placed inside buildings, so receiving a GPS signal for locating a tag cannot be trusted. To make things even more

144

Tags and the City

complicated, tags can be completely invisible as they can be hidden into objects or structures. The street address is, in many cases, not descriptive enough for indicating the location of a hidden invisible tag. Moreover, tags can be placed in moving objects, which makes it even more difficult to keep track about their locations. For example, information tags (described in chapter 3.3) can be attached in public transportation vehicles such as busses (as in Figure 1) and trains. In these cases, the exact location of the tag cannot be tracked unless the object where the tag is attached can be tracked. Therefore, the most feasible way to indicate tag location can be to store information about the object where the tag is attached or placed, and instructions for how to locate and recognize the particular object in a physical space. Keeping track of tag locations is crucial from several viewpoints. Knowing the location of tags enables several advanced application possibilities, such as location-based services and payment schemes. The information tag field trial revealed that users expected the content and services accessed through tags to be very location-specific. They were very annoyed, if the tag provided access to a generic information service which required browsing or searching to access location-specific information. For example, one of the tags used in the information tag trial provided access to a generic event service. The service used the capital of Finland, Helsinki, as a default value; therefore displaying the events currently ongoing in Helsinki. In order to access event information in other cities, the users had to change the city from a pull-down menu. The users were very annoyed and surprised to access event information in Helsinki when they were 600 km away in Oulu. As tags are always located in some specific place, the location information should be used by the service for providing direct access to location-specific services and content for optimal user experience. The owners of places and items where the tags are located can usually decide where tags can be placed, and on what conditions. Walls in a city are always owned by someone who most likely wants to control what is attached to the walls (or other surfaces). As the tag is connected with the digital world, usage-based payment schemes can be introduced for optimizing the income and monetary efficiency of tag placement. The space owner may receive a certain amount of money from each interaction initiated by touching the tag. This would result in higher income in places where tags can be easily used and accessed by their potential users.

145

M. Isomursu

Tag locations are also crucial for maintenance purposes. It is impossible to collect old and invalid tags from the environment if the tag locations are not known. Eventually, all applications become obsolete and at this point the tags need to be removed or reprogrammed. As NFC tags can be operated only from a close distance by physically touching them, all tags need to be visited. If obsolete, non-operating tags are not removed or re-programmed, they become “tag litter” that was found to evoke a negative user experience. Even when our trial users were exposed to very small amounts of tags, our experiences from the trials clearly show that broken or outdated tags corrupt the trust towards tags, and make users less willing to use tag-based service access. This problem would probably multiply, if tags would be more common and distributed by a wide range of service providers and actors. When the users touch tags that do not respond, or that include outdated information, they get annoyed and are less willing to use tags. Also, if the tags become ubiquitous and could be distributed into our everyday environment by virtually anyone, owners and maintainers of physical spaces and surfaces need to know which tags are allowed and needed in a certain place, and which tags are unwanted and need to be removed. As with graffiti, it is probable that property owners would want to remove unauthorized tags from the surfaces and objects they own. Information about the placement of tags is needed by maintenance personnel in order to know which tags to remove. Solutions for communicating this to the maintenance personnel are also needed. In the information tag trial (described in chapter 3.3), one of the most serious problems faced during the trial was that the bus stop maintenance personnel removed the tags that were placed at bus stops. Even when the maintenance company had given permission to attach tags to the bus stop for the duration of the trial, this information had not reached each and every maintenance person. It is easy to imagine how complicated this can get, if the amount of tags would be high and different kinds of tags are allowed in different spaces. Avoiding a bad user experience caused by tag litter requires that the people distributing tags actually care to maintain the tag-based services by keeping the content up-to-date, and by removing broken or obsolete tags from the environment. Of course, this might not always be the case. As tags are very cheap, easy to program and can be attached virtually anywhere, it might be easier and cheaper for the service or content provider to just distribute the tags and forget them.

146

Tags and the City

5.3 Detecting Faulty Tags The NFC tags are able to communicate with the digital world only when they are activated by an NFC tag reader device. This means, that it is impossible to see in realtime, if a tag is faulty. If the tag is faulty, the tag reader is unable to perform any actions, and therefore no trace of a malfunction can be stored in logs or background systems. Testing and detecting malfunctioning tags manually is time consuming. This requires that all tags distributed are visited for testing their functional operability. Based on the pilot findings, the reliability of the tags is high. The durations of the pilots were some months, and no problems involving faulty tags were identified. However, in pilot planning some precautions related to faulty tags were taken. Two main approaches for detecting faulty tags were adopted: the first requested the users to report faulty tags, and the second was based on monitoring logs to identify tags that were not generating activity even if they were supposed to do so. Both approaches assume that the service provider distributing the tags is responsible and motivated to maintaining the tag infrastructure, and wants to identify and remove broken tags from the environment. However, as discussed in the previous chapter, this might not always be the case. For removing broken or obsolete tags that have been abandoned by their original owners, there might be a need for public “tag removal troops” who would clean public places of tag litter. However, this problem was not explored or addressed in our trials.

User Generated Error Notifications. In all pilot setups, the users were able to directly contact research coordinators for reporting problems with tags and the service. In some cases, the reporting channel was telephone (e.g. the elderly meal care) and in some, a web-based feedback application (e.g. parking pilot) was used. As the pilot applications were limited in scope and the users were recruited specifically for the trial, it was easy to establish procedures for reporting faulty tags. However, this would be considerably more difficult, if the users would use tags related to wider variety of services, and the users of tags would be random users with no special attachment with the service provider. For large scale use, the procedures for reporting faulty tags must be made more transparent and easy for the user. For example, each tag could include instructions for the user when experiencing problems with a tag. This can be, for example, calling a service number. However, as reporting faulty tags probably will require extra effort and even monetary contribution from the user (cost of a phone call, for example), the users

147

M. Isomursu

could be rewarded for reporting faulty tags. Service numbers could even be shared by different service providers through tag management service providers. This would make it possible to establish commonly known tag service customer touch points. However, our experience with tags in public spaces show, that the users are usually very mobile and busy, and are not too willing to contribute very much time and effort. User generated error notifications would probably work best for tags that are placed inside homes or other private spaces where the users would feel motivated to maintain service infrastructure, and would have a regular usage relationship with the services provided through tag infrastructure.

Monitoring Activity Generated by Tags. When the service provider monitors the use of tags, the tag logs can be analysed for identifying possible problem situations. In practice, this means that if the activity patterns generated by specific tags change radically, faulty tags may be a reason for the change. For example, in the elderly meal care pilot, if no meal selection was received by the service system, the elderly user was called to check why they had not placed an order for the following day. By monitoring tag-generated activity patterns, the abrupt absence of activities received from a tag that normally generated activities can be interpreted as a sign of a faulty tag. However, at this point the tag already has disappointed several users as their service requests have not been successfully processed. 5.4 Visual Design The demonstrations and applications utilising physical browsing propose two different visions of the future with tags. The first vision uses tags as a technique for adding digital nature into physical objects that already exist in an environment. In this vision, our physical world would still look exactly the same, only with added functionality. The tags would be attached to existing objects and they could be even invisible or hidden. The other vision brings digital services and content as an addition to the existing reality. In this vision, links and access points to digital content and services do not directly relate to a specific object that is already present in the environment. Rather, they bring new objects into our environment that relate to a service need that the users may have in the context of the given location. This vision brings new offerings and affordances into our everyday environment and at the same time, changes our environment not only on the functionality offered, but also by adding new objects and visual components into our everyday environment.

148

Tags and the City

If tags are numerous and visible in the environment, they all compete with each other about the visual attention of the user. Public spaces are already loaded with visual information, so designing tags that can be found and recognized with ease is challenging. Standardisation efforts may help in creating widely recognized standardised icons and symbols that can be used for indicating a tag (Mäkelä, Belt, Greenblatt, & Häkkilä, 2007). Currently, the size of NFC tags is around four (4) centimeters. Both round and square tags are available. Our experience indicates that this size is well-suited for tags that are identified and used from a close proximity, e.g. when they are placed on a stand on a table and the user sits by the table. However, when tags placed on objects that are far from the user, such as posters hung on the walls of theatre waiting hall, the users reported that they had more difficulties to see and identify the tags, as they were viewed from a distance. In cases like this, it might be convenient to have tags that would have large touch areas, and therefore could be better seen from a distance. Of course, the visual marker of the tag can be larger than the actual touch area of the tag. However, then the user would need to locate the touch area within the visual marker. Tags can be placed behind surfaces or they can be visually designed to blend into the environment to the degree that the tag is invisible for the user. Hidden tags can provide a strong feeling of magic (Isomursu, Hinman, Isomursu, & Spacojevic, 2007), as the user is able to initiate action without actually seeing the technology behind the action. For example, touching a picture frame holding a photograph of a grandson with a mobile phone can be used to initiate a call to the grandson. The picture frame would look just like any picture frame as the tag can be placed behind or inside the frame. If the tag is invisible, the user may face problems in finding it and knowing which objects provide digital functionality. NFC tags need to be touched, so the user needs to know where the tag is. Therefore, a more common scenario is that the tags are visible.

Standardized NFC-Specific Icons. In parking and information tag pilots, the visual logos designed by the NFC forum were used to visualize tags (see blue icon in Figure 4). Generic icons for indicating tags can help users to identify where tags are available, and where the touch area is located. However, a generic tag indicating the location of the NFC tag does not give the user any information about the service provided through the tag. In order to provide the user this information, additional visual cues and/or textual information is needed. Some researchers (e.g. Välkkynen, Tuomisto, &

149

M. Isomursu

Korhonen, 2006) have proposed sets of tag visualizations that would give the user information about the service provided.

Accessibility Issues. In the elderly meal-care pilot, the visual language of tags was designed to be readable and usable for elderly users who have poor visual sight and trembling hands. As the tags were used for limited purposes in a controlled environment in the user’s home, the users learned to find and recognize the tags quickly. In this pilot, the fixed size and form of the tag forced the UI design to follow a cognitively challenging table form (see menu illustrated in Figure 2), as the tag could not be placed directly behind the text indicating the meal selection. The possibility to specify the form and size of the area to be touched would give more powerful tools for user interface design. For the visually impaired, finding a tag can cause problems. A tag can be difficult, or even impossible, to find without visual clues. However, for visually impaired users, the tag could be marked with the Braille system that would both indicate the place to touch and provide a visually impaired user with information about the service the tag offers.

Visual Design as a Tool for Creating Trust. The visual design of the tag and its surroundings needs to give the user information about the purpose and actions that are initiated by the tag. As the touching action itself does not define which action will be initiated, the visual cues of the tag and the surrounding environment need to give the user all the information that is needed for understanding what the tag is supposed to do. Visual design is one tool for creating the sense of trust and security in the user. Using visual language, such as logos and colors that are well known and easily associated with reputable and well-known services and organizations can be used to create a feeling of trust. Existing product and company logos and visual look and standardised icons for services can be used. 5.5 Security As programming and distributing tags is very easy and, at least for now, nonregulated, it can be assumed that if they become common, tags will be used for quite a variety of purposes. Some of these purposes may be unwanted or even harmful for end users.

150

Tags and the City

Tags embedded into our environment may lure unsuspecting browsers to access content or services they do not wish to enter, causing financial or emotional worries. Especially, if the tag can initiate payment or billing actions, there always is a risk that the users are lured into paying on the basis of false information. For example, in the parking payment pilot the pilot system covered the parking areas maintained by the city of Oulu. Tags with similar outlook and functionality, only directing the payment to a private bank account, could be added by some entrepreneurial persons to areas that do not have parking fees, and a private non-legal parking business could be quickly established.

Walled Gardens. One solution is to allow only tags provisioned by a trusted service provider to initiate action. However, this would set up “walled gardens” [10] and limit the freedom of the user. Humans tend to prefer free, unlimited choice, even if they would be happier with limited or fixed choice (Brown, Read, & Summers, 2003). Management of trusted service accounts can be done by a tag management layer in the NFC reader device. Security tools could keep track of trusted service providers, and the tools could warn the user if non-trusted parties are involved in the service request initiated by the tag. NFC forum has been active in creating security enhancing procedures such as signatures for secure identification of trusted service providers. Signatures can be used for identifying the issuer of a tag.

Confirmation of Tag-Initiated Activity. It is common that users express security concerns related to RFID technology because tags are viewed as “active” and therefore users are concerned that they would initiate actions without even noticing (Mäkelä et al., 2007), or that the expectations of tag-initiated behavior would not equal the service received. Therefore, in most of the experiments, a confirmation was requested before an action was processed. For example, when the user touched a tag where the text “Call a taxi” was displayed (one of the tags in the information tag bar described in chapter 3.3), the phone number of the taxi to be called was displayed on the screen of the phone, and the user was requested to confirm that the phone call would be made. However, in some cases it was more convenient for the user to proceed with the taginitiated activities without confirmation requests. These cases included, for example, meal ordering for elderly people. In this application, the users were able to fully trust the tags as they used them daily over a long period of time, and the tags were located

151

M. Isomursu

inside their own homes. Moreover, as the users had memory problems and weakened eyesight and motor skills, reading text from the phone display, procedural activities and pressing small buttons were avoided.

6. Limitations and Validity

Even though the goal of the field trials was to aim for as high an experimental reality (Aronson, 2004) as possible, there are issues in the trial settings that probably have effects on the results. Perhaps the most severe limitation of our research setting was the availability, selection and content of the services and content accessible through the tags. Tags are not widely used and common, so the trial users were exposed only to tags provided by the research group. As the tags were evaluated in the research project, there were no actual business goals or goals for the public good for providing the users access to mobile content. Tag placement, design and accessed information content were not rigorously designed to meet any specific goals, such as optimal coverage of a certain user group. The content provided through tags was selected in a brainstorming session of the researchers, and the selection criteria used were probably very different from those that would be used if the tags were used for commercial, or for any purpose other than research. As a result, some tag content was obviously very poorly suited for the specific place it was offered in. For example, many users commented that a tag that helps you call a taxi when you are paying a parking fee was pretty useless. On the other hand, if in the future the tags become commonly used and ubiquitous, poorly chosen and placed tags will probably be rather common, too, as some tag providers might find it faster, cheaper and easier to attach tags randomly than to do a proper analysis for the optimal locations for tags. Also, as the tags were evaluated in one single research project, the availability, selection and variety of tags was very limited. If tags become popular Mobile Internet access points, there will be more variety in selection, and tags will be more numerous and available. Another issue that may have an effect on the results is that none of the users could use their own mobile phone to access Mobile Internet through tags, but had to use a special NFC-enabled trial phone. This means that users usually carried two mobile

152

Tags and the City

phones with them, and used their own mobile as a phone, and the trial phone only for NFC-enabled features. This might have an effect on the usage frequency, perceived accessibility and ease-of-use, as the users suggested that they were more familiar with their own phones, and used them more frequently than the trial phone. The fact that the user experiences were collected in a trial setting probably has an effect on the motivation of the users. The users were recruited as trial users, so they were committed to try out the provided services. Therefore, the first usage was probably initiated by this commitment, and not purely because of interest or curiosity towards the services provided. The trial users also knew that the services would be available only for the duration of the trial. As users are aware that the evaluated technology is part of their lives only for a limited period of time, their commitment to adopt the technology as an integral part of their lives may be weak. If users would think that they are stuck with the technology (e.g. if they had invested a significant amount of their own money for buying the technology), they would need to create strategies to successfully integrate it into their everyday life. If problems would arise, knowing that they would need to use the technology in spite of problems, would trigger a process for reducing cognitive dissonance (Festinger, 1957) which might make them feel and behave differently than in an experimental pilot. Irrevocability of decision has been shown to be an important contributor to user experience (Frenkel & Doob, 1976).

7. Discussion

Only time will tell how quickly NFC technology will penetrate markets and become ubiquitously accessible for all mobile users, or if it will make it at all. The first mobile devices with NFC capabilities have been on the market already for some time, but the low quantities still hinder application development. Adoption of NFC technology is in the typical egg-and-hen situation, where the device manufacturers are waiting for signals from application providers and users for a need to integrate NFC technology into devices, and the application providers and end users are waiting for the technology to become more common, allowing more uses and thus economies of scale. The findings presented in this paper are derived from the constraints and possibilities of NFC technology, but many remain valid also with other tag technologies, such as visual tags that rely on the use of a camera (López de Ipiña, Mendonça, & Hopper, 2002), or other tag technologies based on the use of radio frequency.

153

M. Isomursu

The importance and applicability of the requirements presented in this paper varies in different settings and between services. For example, the required durability and expected use time of the tag may vary from a very short time, even single usage to permanent infrastructures that are expected to serve users without a specific ending time. Examples of tags that do not have a long life span are tags attached to brochures or other printed material that are used once and then discarded. The differences with the expected life span of the tag affect the requirements for the management of tags during their expected time of use.

8. Acknowledgement

This work was done in the SmartTouch (www.smarttouch.org) project (ITEA 05024) within ITEA 2 (Information Technology for European Advancement), a EUREKA strategic cluster. The project has been partly funded by Tekes, the Finnish Funding Agency for Technology and Innovation.

9. References

Ailisto, H., Pohjanheimo, L., Välkkynen, P., Strömmer, E., Tuomisto, T., & Korhonen, I. (2006). Bridging the physical and virtual worlds by local connectivity-based physical selection. Personal and Ubiquitous Computing, 10, 333-344. ABI research (2007). Twenty Percent of Mobile Handsets Will Include Near Field Communication

by

2012.

Retrieved

August

29.

2008,

from

http://www.abiresearch.com/abiprdisplay.jsp?pressid=838 Aronson, E. (2004). The Social Animal. Worth Publishers. Ninth edition. Brown, N., Read, D., & Summers, B. (2003). The Lure of Choice. Journal of Behavioral Decision Making, 16, 297 – 308. Card Technology Today (January, 2005). NFC technology begins new trial. Card Technology Today, 17(1), 4-5. Festinger, L. (1957) A theory of cognitive dissonance. Stanford: Stanford University Press. Frenkel, O., & Doob, A. (1976). Post-decision dissonance at the polling booth. Canadian journal of behavioural science, 8, 347–350.

154

Tags and the City

Häikiö, J., Isomursu, M., Matinmikko, T., Wallin, A., Ailisto, H., & Huomo, T. (2007, September). Touch-based user interface for elderly users. MobileHCI 2007, September 9-12, Singapore. Isomursu, M. (2008, September). Benefits and Challenges of Evaluating Ubiquitous Technology in Field Settings. Ubicomm 2008, September 29 – October 4, Valencia, Spain. Isomursu, P., Hinman, R., Isomursu, M., & Spacojevic, M. (2007). Metaphors for Mobile Internet. Journal of Knowledge, Technology, Policy, 20 (4), 259-268. Kamvar, M., & Baluja, S. (2006, April). A Large Scale Study of Wireless Search Behavior: Google Mobile Search. CHI06, the SIGCHI Conference on Human Factors in Computing Systems, April 22-27, Montreal, Canada. Lindenberg, J., Pasman, W., Kranenborg, K., Stegeman, J., & Neerinex, M. (2007). Improving service matching and selection in ubiquitous computing environments: a user study. Personal and Ubiquitous Computing, 11, 59-68. López de Ipiña, D., Mendonça, P., & Hopper, A. (2002). TRIP: A Low-Cost VisionBased Location System for Ubiquitous Computing. Personal and Ubiquitous Computing, 6 (3), 206-219. Mäkelä, K., Belt, S., Greenblatt, D., & Häkkilä, J. (2007, April). Mobile interaction with visual and RFID tags: a field study on user perceptions. CHI07, the SIGCHI Conference on Human Factors in Computing Systems, April 28 – May 3, San Jose, USA. NFC forum (2008). About NFC. Retrieved July 29, 2008 from http://www.nfc-forum.org. Ondrus, J., & Pigneur, Y. (2007, July). An Assessment of NFC for Future Mobile Payment Systems. International Conference on the Management of Mobile Business, July 9-11, Toronto, Canada. Pirlich, M., & Lochs, H. (2001). Nutrition in the elderly. Best Practice & Research Clinical Gastroenterology, 15 (6), 869-884. Rukzio, E., Leichtenstern, K., Callaghan, V., Schimdt, A., Holleis, P., & Chin, J. (2006, September).

An experimental Comparison of

Physical

Mobile Interaction

Techniques: Touching, Pointing and Scanning. The Eighth International Conference on Ubiquitous Computing, September 17-21, Orange County, USA. Schultz, R., & Hanusa, B. (1978). Long-term effects of control and predictabilityenhancing interventions: Findings and ethical issues. Journal of personality and social psychology, 36, 1194-1201.

155

M. Isomursu

Välkkynen, P., Tuomisto, T., & Korhonen, I. (2006, May). Suggestions for Visualising Physical Hyperlinks. Workshop on Mobile Interaction with the Real World (PERMID), May 7, Dublin, Ireland.

156

PsychNology Journal, 2008 Volume 6, Number 2, 157-172

Experiences of Evaluating Presence in Augmented Realities Rod McCall and Anne-Kathrin Braun Fraunhofer Institute for Applied Information Technology (FIT) Department for Collaborative Virtual and Augmented Environments (Germany)

ABSTRACT This paper presents an overview of a study of 24 people who used an augmented reality game called TimeWarp. The paper initially discusses the game and evaluation methods chosen, it then explores emerging issues from the evaluation which are applicable to other augmented reality games and how existing user testing methods require further improvements in order to capture data relevant to the issues. Keywords: Augmented Reality (AR), Multimodal Interfaces, Mobile Gaming, Pervasive Gaming, Mixed Reality (MR). Paper Received 13/06/2008; received in revised form 05/08/2008; accepted 08/08/2008.

1. Introduction

Location aware technologies such as widespread mobile computers and varying location sensors provides a vast array of possibilities for extending game playing into streets, buildings and even the rural landscape. New and extended forms of locationaware games including mobile or pervasive phone games, smart toys, role-playing games as well as Augmented Reality (AR) games all demonstrate promising new forms of game play. Substantial work has also gone into new game concepts, sophisticated technology and viable business models. However, research on the methodological issues of studying mobile player experiences in augmented reality games. This paper explores our experiences of using standard methods which were modified to specifically explore place, presence and usability in augmented reality games.

Cite as: McCall, R., & Braun, A-K. (2008). Experiences of Evaluating Presence in Augmented Realities. PsychNology Journal, 6(2), 157 – 163. Retrieved [month] [day], [year], from www.psychnology.org. Corresponding Author: Fraunhofer FIT, Schloss Birlinghoven, 53754 Sankt Augustin, Germany E-mail: [email protected]

157

R. McCall, A-K. Braun

The paper initially explores the underlying theories which the methodologies intend to address, followed by descriptions of an existing system and evaluation approaches. While it presents some findings relevant to the design of mixed reality systems the objective is to explore the underlying methods. A further description of the results can be found in Herbst, Braun, Broll, & McCall (2008). It concludes by indicating that there is a need to further develop the approaches so that they are more able to explore changes in sense of presence within augmented realities.

2. Related Work

Unified Experience of People, Activities, Objects

Virtual Reality

AR

Reality

Figure 1. The overall Augmented Reality (AR) experience.

One of the main challenges within augmented reality environments such as the one discussed later is how to create a unified sense of place and presence. By this we mean that the user feels as if the virtual elements are as real and natural as those from the real environment and that they are constantly within the overall AR experience (see Figure 1). This diagram to some extent is based upon the work of Milgram (Milgram, Takemura, Utsumi, & Kishino, 1994) who developed the idea of the virtual to reality continuum. The idea of a unified sense of presence also has many commonalities with Gibson's concept of affordances (Gibson, 1979), where he sees no difference between real or virtual. Instead affordances arise due to the user’s perception of the features in the environment. It has been argued by some that through these affordances the user interacts in the environment and thus feels present. At the outset one key area of exploration is the user’s sense of presence within such environments. Research from virtual reality points to presence being a combination of physical and social attributes, for example, feeling present in the environment and with other people. Both of these are highly relevant within AR contexts however for slightly different reasons. For example if the desire is to make people feel simultaneously

158

Experiences of Evaluating Presence in Augmented Realities

present with real people and places, as well as virtual people and places. This also requires an examination of some of the elements contained within standard presence questionnaires such as attention, awareness, interest, engagement and involvement. For a unified experience to be created it would therefore be necessary to maintain the user’s interest and attention in the experience even when there are little or no virtual aspects. Furthermore, as the experiences take place in reality they should also posses some awareness of real people, places and objects. Therefore the sense of presence is co-constructed through the experience of real and virtual elements and as such understanding this relationship becomes critical. All AR experiences by default occur in a real space, which is augmented with virtual elements, which through the user's personal interpretation will contain meanings and significances. This gives rise to the idea of place, and as noted by Relph (1976), a place is a combination of physical properties, activities and meanings. Tuan's (1977) conceptualization of place encompasses these aspects in a four layer model consisting of cultural significance as well as social, personal and physical aspects. There are other models, for example Gustafson (2001) emphasizes the importance of self in relation to the environment and other people. Regardless of which model of place a developer or evaluator adopts, it should serve as a starting point when considering where to locate AR experiences, as well as which virtual and real elements to include as part of the experience. Otherwise there is the potential to create virtual elements which ill fit the environment in which they are located. Thus possibly making the user largely ignore the rich experiences provided by reality and focus their attention on the purely virtual elements. This may in turn give rise to them feeling that they are more present in the virtual world and not in the overall AR experience.

3. The TimeWarp Approach

3.1 Design Objectives TimeWarp was designed to provide a rich gaming experience which explores the full potential of 3D animation and spatial AR sound. The game takes place within a real city and allows the user to experience the city in several different time periods. A combination of virtual objects, augmented sounds and music which represent appropriate aspects of the various time periods are used to alter the player’s sense of temporal, physical and social presence. It also support non-linear gameplay.

159

R. McCall, A-K. Braun

3.2 Story and Structure of the Game The TimeWarp game is staged in the old part of the city center of Cologne and is based on the famous tale of the so called “Heinzelmännchen” (Figure 2). These small elves worked clandestinely for the citizens during the night carrying out their household tasks. However one morning they disappeared thus then forcing the citizens to carry out their own tasks. The story has been modified for use in the game so that now the Heinzelmännchen are still in the city but are trapped in different time periods (epochs) and it is the players task to bring them back to the present day. It is for this reason, that the player is equipped with an Augmented Reality system. Using the AR system the player is able to see all artifacts from the particular time period in which they are situated. To travel in time, the player has to reach one of the time portals, which are distributed in the city. In each time period, several tasks have to be solved to free an elf. These tasks are related to the history of Cologne and to the current epoch. Once all the Heinzelmännchen have been freed the game is over.

Figure 2. TimeWarp Challenge

3.3 System Concept and User Controls The AR system used in this game consists of a head-mounted display (HMD) with an orientation sensor attached (Figure 3). The player’s position is tracked via GPS and as they walk around the virtual content (which they see through the visor) is placed at the relevant locations. The system runs on a laptop which is inside a backpack and which is worn on a so-called AR vest. The vest contains the connections between the various devices. In addition to the mobile AR system, a handheld-based device supports the player during the game. On an interactive map the current position is shown either in map or in satellite mode. Besides the guidance of the player, the map tool provides

160

Experiences of Evaluating Presence in Augmented Realities

several features such as zoom-in and zoom-out and the display of important game related locations. Furthermore, the actual game state, information about the game play and system help is provided by the handheld device.

Figure 3. AR system (left) and handheld device (right).

In the first prototype of TimeWarp, three different types of interaction techniques were developed. The first technique uses the physical proximity of the player to a real object and when the user approaches an object from a preset distance an event or action will occur. The second approach allows the user to focus on an object by using a gaze based pointer and by pressing the mouse button, a special action for the selected object will be activated. The third technique uses a gyroscopic mouse, this approach lets the user interact with and place objects in mid-air.

4. User Tests

A large-scale test was conducted during summer 2007, the study took place on location in the City of Cologne and the objective was to test many aspects of the system from user experience (usability, sense of place and presence) through to

161

R. McCall, A-K. Braun

technical issues. The study utilised a number of accepted methods which are outlined in more detail below these included a questionnaire, observation method, video observation and a semi-structured interview. Broadly speaking all users experienced the system for around 1 hour, although a few participants took part for up to three hours. A total of 24 people took part in the study these ranged from IT students through to city tour guides, however not everyone completed the questionnaire this was due to a number of issues including technical problems causing the studies to last too long or problems understanding the questionnaire. Our objectives in using a variety of data sources was to look for corroborating evidence but also to identify where existing methods did not accurately capture interesting phenomenon. 4.1 Questionnaire A review of existing presence methodologies was conducted in order to ascertain which if any would be applicable for the study. One of the main problems with most methods was that they were derived from presence research conducted within virtual environments and often under strict laboratory conditions. These approaches while relevant in many respects ignored several aspects of urban-based AR experiences namely that they are not laboratory based and sense of social and spatial presence is the direct result of the blending of real and virtual elements. In particular users have the ability to compare the feeling of reality between real and virtual elements instantaneously. The MEC questionnaire (Vorderer et al, 2004) was chosen as a starting point as it provided many of the aspects which were relevant to our study, in particular social and physical presence, moreover, it has been extensively validated. The additions were also made so as to reflect the idea of where people feel present. In addition the Bailenson, Blascovich, Beall, & Loomis (2001) social presence questionnaire was added as this addressed many of the elements of interacting with real people (including game and non-game players) as well as virtual characters which we wished to explore. Switches between the real and virtual elements of the game were a common occurrence and the questionnaire was modified to reflect this Also the attention section of MEC was widened to include more elements relating to mixed reality. Firstly the rating scale was modified to include seven points with 1 representing the real environment, 4 the overall (or blended environment) and 7 virtual environment. A seven point scale was used to allow for a finer grained analysis of the results. This was different from the standard MEC questionnaire in that it only asked people to rate on a

162

Experiences of Evaluating Presence in Augmented Realities

scale of 1 to 5 whether they agreed with a certain proposition such as “my attention was focussed more on the medium”. An additional five questions were added which were intended to explore where people felt the game actions took place and whether they were focussing more on the real or virtual objects for navigation tasks. In order to remain consistent with respect to scoring, the rating scale for all remaining sections ranged from 1 to 7, with a 1 reflecting a strong agreement with a proposition and 7 a strong disagreement. Unlike virtual reality we are not as concerned with issues related to the creation of mental maps. Consequently fewer questions were asked from MEC and three new questions relating to the perceived reality of the virtual elements were added This resulted in this component of MEC reflecting slightly different themes than before. MEC also places significant emphasis upon the sense of spatial location which participants have. For this section we retained the general feel of the initial MEC questionnaire instead fitting our wording to suit the experience, in this case a game. However as noted earlier one of the main interests was any switches which occurred while taking part in the experience, in particular when putting on or taking off the visor, as this in theory should change the sense of place and hopefully presence which the users experience. For this we added questions specifically related to changing sense of presence when entering or leaving the experience. MEC does not deal with temporal presence and so a number of questions were added which specifically covered this area. These included specifically asking people if they had felt like they had visited different time periods. In order to ensure further certainty additional questions were added to explore if people felt any change while moving between time periods and whether they felt different towards the environment or it altered their behaviour. Social presence forms a key part of TimeWarp either from the perspective of player/player interaction (although the game is designed only for single players), player/character or player/non-player interaction. We modified the Bailenson social presence questionnaire to reflect these aspects by specifically addressing the array of relationships which can ensue within location-aware augmented reality games. The objective of measuring the various forms of social presence which may exist within such an experience was to allow for comparisons between different forms of social presence. Furthermore we were interested in exploring the awareness others, in particular with respect to whether people felt that non-players felt they were acting strangely.

163

R. McCall, A-K. Braun

We were particularly interested in how people felt their sense of place would change when in different time periods, and indeed whether they felt the building, characters and other aspects had any impact. For this we added in a number of additional qualitative questions drawn from The Place Probe (Benyon, Smith, O’Neill, McCall, & Carrol 2006), in particular those which would help us identify differences between different time periods and general information regarding the overall experience.

4.2 Observation/iPerG Method During their experiences the participants were recorded and/or observed, this approach allowed us to capture the social elements of their experience in particular responses from non-game participants. Furthermore it allowed the observation of how the users responded to real and virtual elements, navigated and interacted with the technology. Where possible an additional observer also followed the users, this observer took notes covering aspects such as player-player interaction, player-non player interaction, player-game element interaction and interaction with the technology – this approach was predominantly devised from work carried out within the EU funded Integrated Project on Pervasive Games (IPerG). 4.3 Interviews After each trial (and if they subjects agreed) they took part in a short-semi structured interview. The objective of the interviews was to probe users further on their experience, in particular to explore any interesting observations which were made during the trial or issues which arose within the questionnaire answers. Such areas would include where people appeared heavily involved in the experience, or when they experienced problems. The questionnaire data was also used to form particular lines of questioning, for example when participants gave conflicting responses about the experience or indicated strongly they felt that time the periods had changed they would be asked to explain their position.

5. Results

5.1 Summary of Implications for Re-design The study resulted in a number of core issues being identified, these included: understanding attention allocation, simplifying the interaction scheme, user safety,

164

Experiences of Evaluating Presence in Augmented Realities

design appropriate paths through the environment, understanding the locale, interaction with others, seamful design, using a combination of real and virtual objects and finally providing a continuous experience (see Table 1). The issue of user safety will not be fully addressed within this paper however, it should be noted that game elements which draw the user’s attention away from real elements, in particular cars and other people should be taken into consideration from the outset. This paper will focus on the results from the study and how these core elements should form the basis of new evaluation approaches for mixed or augmented reality games. A more thorough review of the results can be found in (Herbst et. al, 2008).

Understanding attention allocation Simplifying the interaction scheme User safety Design appropriate paths through the environment Understanding the locale Interaction with others Seamful design using a combination of real and virtual objects Providing a continuous experience Table 1. Design guidelines.

As noted in the results, users appeared to alter their focus of attention between the real and virtual elements, typically seeking out virtual elements then returning to reality when there were no virtual elements. Such switches in attention play a crucial role in shaping the experience, for example when attention is more focussed on virtual elements users may feel more part of the virtual experience; and possibly even ignore aspects of reality. Therefore making blends of experience somewhat difficult, conversely when focus of attention is on real aspects of the experience (for example when chatting with the evaluator) the user will be missing out on elements of the virtual experience. In addition to the spacing between elements (which is discussed later) another driving factor was the design and placing of elements. For example, use of virtual objects which appear radically different from the surrounding environment will also draw the user’s attention. Interaction within mixed or augmented realities is a new experience for most users. In the case of TimeWarp this involved a range of devices including a mouse and PDA, as well as a visor and a range of mixed reality interface techniques. These interaction

165

R. McCall, A-K. Braun

techniques not only alter experience from the perspective of usability but also can distract the user from the surrounding environment and overall game experience. For example if the user constantly has to look at the PDA in order to navigate then they are arguably ignoring elements of the mixed reality game content. Alternative approaches include using auditory cues to direct people towards specific locations or objects – in later versions of the system the latter approach was adopted. Furthermore interface widgets and interaction styles within AR games are often quite complex. As a result there is a need to consider carefully the nature and type of devices used as well as how to make interacting with widgets as natural as possible. Methods could include making interacting with the virtual elements similar to interacting with comparable real elements. Moreover, where possible users should be familiarised with the interaction scheme through the use of a training scenario, however this should form an integral part of the game. AR games are by default situated within a real space and they rely on the relationship between the real and virtual elements to create the overall experience. Within TimeWarp it was clear that although the game was situated within the urban environment it did not make use of the underlying structure of the location (beyond using specific locations or allowing people to walk between them). This is potentially quite a substantial loss as paths between locations form the basis of cognitive models of spaces, thus aiding in navigation and also in the construction of a sense of place (Appleyard, 1970; Devlin, 1976; Lynch, 1960: Norberg-Schultz, 1971). Therefore when designing experiences the MR environment should take into account the nature and layout of paths, which can encourage participation within objects or locations (e.g. paths which intersect through content) or allow a more passive observation approach (e.g. paths which pass-by locations). Furthermore path structures can be used to heighten experiences, for example the use of clear, starting, middle and end points can be used to provide a spatial narrative as well as to improve navigation. From the perspective of AR this approach should involve creating paths which bring together real and virtual elements. Following on from the idea of path structures is that of using and understanding the locale in which the game is situated. In TimeWarp many locations exist within the area in which the game takes place, including shopping streets, cafes, open spaces (such as the grassy area near the Rhine Promenade) and a Cathedral. Each of these provides a rich tapestry of physical properties, people, meanings and experiences. Thus when situating such games these aspects should be taken into account such that

166

Experiences of Evaluating Presence in Augmented Realities

for example drinking a coffee and socialising with people at a Café or simply using at as a location to stop observe people. There has already been substantial work on layouts within towns and cities, for example Ching (1996) provides information on more generic layouts while Alexander (1977) provides a list of common patterns which can be found in cities. Therefore understanding the people, activities and potential meanings of any location could prove invaluable in designing mixed reality systems. Social presence is a key aspect of such systems, ranging from interacting with other players and virtual characters through to causal encounters with non-game participants. The lack of believable interaction with virtual characters, and other players was a major problem within the game. Furthermore non-game participants only became part of the experience when they interrupted the gaming experience. These interruptions had the effect of distracting the player from the gaming experience, rather than being useful. Given the vast array of possibilities to include real people in such experiences e.g. to answer questions about the location etc, it would appear logical to include them within the game. Related to the topics discussed above is the ability of mixed reality to create a continuous experience, by this we mean that as users walk around they feel as if they are in the given time period or place for the duration that they are intended to be in such an experience. In order to support this concept we propose two further criteria the idea of seamful design which was initially developed by Chalmers (2003). We also propose a further concept of integrating real elements into the gaming experience. The idea of seamless design approaches technical problems from an alternative angle. For example where wifi signals are weak then these black out areas should form an integral part of the experience for example providing locations where people can hide from other players without detection. The use of real and virtual objects should also be considered with care. For example real objects should be integrated into the experience where it is possible to do so and where any virtual equivalent would result in more usability problems. Also real elements should be integrated into the experience where they form a key part of the game play, for example collecting objects to complete a task. However it should be noted that integrating real objects within such experiences can be problematic without accurate computer vision or marker based techniques. 5.2 Evaluation Methods The interviews, video analysis and data observation proved the most successful in obtaining data which formed the basis of the results above. Due to the completion and

167

R. McCall, A-K. Braun

return rate of questionnaires they were broadly speaking only useful for identifying possible interview strategies. Furthermore this was the first iteration of questionnaires and there is room for improvement, in particular reducing the number of questions. Video and direct observation proved very useful in looking at aspects such as shifting focus of attention at different points in the game. While it is impossible to say that a specific observation is a direct result of any internal cognitive state it does alert us to the changing behaviours of players throughout the experience. Examples include observing people wearing or removing their headsets, interacting with the evaluator or running towards or interacting with virtual objects. This would imply that direct observation approaches which use semi-structured forms to capture data should explore where and when people appear to switch between experiences. While this was partially addressed within the IPerG approach, i.e. it asked for a log of communications between player/player and non-player/player, it does not specifically deal with these issues. Although many of the methods pointed to interesting themes, in themselves they did not permit an adequate exploration of the issues surrounding them. Such issues included using the underlying locale or using paths from within the environment. This is despite the fact that they could play a part in shaping the game experience. There are of course other methods which explore navigational perspective such as ENiSpace (McCall & Benyon, 2003), however ENiSpace focuses on designing and evaluating purely electronic environments and as a result it is not entirely applicable within the domain of mixed or augmented realities. Moreover approaches such as the Place Probe upon which parts of the questionnaire only captured basic overall experiences and not information regarding experiences which altered during the game e.g. the changing sense of place and presence. The evaluation techniques chosen also focused heavily on presence related issues, although they were capable of detecting some usability problems when they arose - in particular people having problems with the training scenario. However the methods chosen and alternatives from the VR community many do not fully explore how to integrate real elements into the game space, hybrid objects which use a blend of augmentations and real elements or purely real or virtual aspects. Social presence was a critical element of the gaming experience and this was reflected within the questionnaire as well as observations and video analysis, for example it was often noted how the users felt out of place, in particular with respect to non-players – some of whom made comments in the street. The approach was broadly

168

Experiences of Evaluating Presence in Augmented Realities

speaking successful however there is still room for improvement, in particular allowing for more details to emerge with respect to the various forms of social presence within the experience.

6. Discussion

The start of this paper focussed on how mixed or augmented realities require an understanding of where an experience is meant to occur, for example more within the real or virtual space or a balance of the two. Furthermore whether during the game experience the user should be aware of changes in where they are meant to feel present e.g. when then move between different time periods. During the study we uncovered that failing to make use of the real environment effectively within the game space often led to people appearing to enter or leave the experience. This was characterised by them removing the visor, chatting with the evaluator or appearing not to take much interest in the content. Conversely they would appear to return to the game experience when near or interacting with virtual content. As a result there were often large gaps between locations when there was little for players to do, and therefore they did not appear to be part of TimeWarp game space. While this is not always a problem it can lead to users feeling bored, which is not desirable. Therefore there is a need to bring reality into the virtual game world, either through the use of path structures in real space which can excite the user, or by allowing real objects or people (non-players) to become part of the game experience. The range of themes uncovered during the study are heavily geared towards the idea of letting people interact within a new place which is a blend of real and virtual, rather than focussing purely on the virtual experience. As noted later many of the evaluation methods used were heavily geared towards purely virtual experiences and thus ignore these blends. Furthermore they did not permit an adequate examination of such themes, in particular allowing the detection of where problems arose or how to rectify them. While certain themes such as seamfulness and attention have been explored by others a more detailed analysis is required, in particular how to support evaluators and designers of mixed reality experience on issues such as selecting appropriate paths, or making more effective use of the locale. Additionally there is also a need to explore methodologies which can support the detection of such issues. Approaches such as MEC and the Place Probe,

169

R. McCall, A-K. Braun

do not allow for an examination of when switches or breaks in presence occur. While observation does to a limited extent illustrate when people start interacting with virtual objects it only provides a very crude approximation of when people switch in and out of an experience. Further details can be drawn from interviews, but these are post experience and thus are again not ideal. Alternative methods such as measuring breaks in presence (Ijsselsteijn, De Ridder, Hamberg, Bouwhuis, & Freeman, 1998) allow people to self report during the experience but require the user to carry yet more equipment and may in themselves cause breaks in presence as people need to interact specifically with the measurement device.

7. Conclusion

The study presented in this paper uncovered a number of themes which are relevant to the design and evaluation of mixed reality games. However to date many of the themes are not adequately addressed by existing presence research and there is a need to focus on developing methodologies which explicitly deal with the complexities of mixed or augmented reality environments. In addition to the themes it is our belief that the complex cues which form part of mixed reality games require a variety of methods to be adopted from observation through to interviews, as it was through this approach that the current themes emerged. The work presented here is not complete but rather is intended to help people understand some of the issues related to developing such games and to inform the development of future design and evaluation methods. In our future work we intend to develop methods which will allow the evaluation of systems based around the themes highlighted earlier.

8. Acknowledgements

We thank our colleagues at the Collaborative Virtual and Augmented Environments Department at Fraunhofer FIT for their comments and contributions. We further wish to thank our project partners of the IPerG and IPCity projects for their ideas, cooperation, and support. IPerG (FP6-2003-IST-3-004457) and IPCity (FP6-2004-IST-4-27571) are partially funded by the European Commission as part of the 6th Framework. We also

170

Experiences of Evaluating Presence in Augmented Realities

acknowledge the work of the MEC consortium and others upon which the modified questionnaire was based.

9. References

Alexander, C. (1977). A Pattern Language. New York: Oxford University Press. Appleyard, D. (1970). Methods of Structuring A City. Environment and Behaviour, 2 (1), 110-117. Bailenson, J.N., Blascovich, J., Beall, A.C., & Loomis, J.M. (2001). Equilibrium revisited: Mutual gaze and personal space in virtual environments. Presence: Teleoperators and Virtual Environments, 15(6), 668-687. Benyon, D., Smyth, M., O’Neill, S., McCall, R., & Carrol, F. (2006). The Place Probe: Exploring a Sense of Place in Real and Virtual Environments: Journal of Presence: Teleoperator and Virtual Environments, 15(6), 668-687. Chalmers, M. (2003). Seamful Design and Ubicomp Infrastructure. Retrieved September 3, 2008 from: http://www.dcs.gla.ac.uk/~matthew/papers/ubicomp2003HCISystems.pdf Ching, F.D.K. (1996). Architecture: Form, Space and Order (2nd ed.). New York: Wiley. Devlin, A. (1976) The Small Town Cognitive Map Adjusting to a New Environment. In G. T. Moore and R. Golledge (Eds.), Environmental Knowing New York (pp58-69), Stroudsburg, PA: Dowden, Hutchison & Ross Inc. Gibson, J.J. (1979). The Ecological Approach to Visual Perception. NJ: Erlbaum Hilldale. Gustafson, P. (2001). Meanings of Place: Everyday experience and theoretical conceptualizations. Journal of Environmental Psychology, 21, 5-16. Herbst, I., Braun, A, Broll, W., & McCall, R. (2008, September). TimeWarp: Interactive Time Travel with a Mobile Mixed Reality Game. Paper presented at Mobile HCI 2008, Amsterdam, The Netherlands. IJsselsteijn, W.A., De Ridder, H., Hamberg, R., Bouwhuis, D., & Freeman, J. (1998). Perceived depth and the feeling of presence in 3DTV. Displays, 18, 207-214. Lynch, K. (1960). The Image of The City. Cambridge, MA: MIT Press. Milgram, P., Takemura, H., Utsumi, A., & Kishino, F. (1994) Augmented Reality: A Class of Displays on the Reality-Virtuality Continuum. SPIE Telemanipulator and Telepresence Technologies, 2351, 282-292.

171

R. McCall, A-K. Braun

McCall, R., & Benyon, D. (2003). Navigation: Within and Beyond the Metaphor in Interface Design. In K. Hook, D. Benyon, & Munro A.J. (Eds.), Designing Information Spaces: The Social Navigation Approach (pp. 355-384). London: Springer-Verlag. Norberg-Schultz, C. (1971). Existence, Space, Architecture. London: Studio Vista. Relph, E.(1976). Place and Placelessness. London: Pion Books,. Tuan, Y.-F. (1977). Space and Place. Minneapolis: University of Minnesota Press. Vorderer, P., Wirth, W., Gouveia, F. R., Biocca, F., Saari, T., Jäncke, F., Böcking, S.,Schramm, H., Gysbers, A., Hartmann, T., Klimmt, C., Laarni, J., Ravaja, N., Sacau, A., Baumgartner, T., & Jäncke, P. (2004). MEC Spatial Presence Questionnaire (MEC-SPQ): Short Documentation and Instructions for Application. Report to the European Community, Project Presence: MEC (IST-2001-37661). Retrieved September 03, 2008, from http://www.ijk.hmt-hannover.de/presence

172

PsychNology Journal, 2008 Volume 6, Number 2, 173-188

Experience Design for Interactive Products: Designing Technology Augmented Urban Playgrounds for Girls Aadjan van der Helm , Walter Aprile and David Keyson Industrial Design Engineering Delft University of Technology (Netherlands) ABSTRACT Recent technological developments have made it possible to apply experience design also in the field of highly interactive product design, an area where involvement of non-trivial technology traditionally made it impossible to implement quick design cycles. With the availability of modular sensor and actuator kits, designers are able to quickly build interactive prototypes and realize more design cycles. In this paper we present a design process that includes experience design for the design of interactive products. The design process was developed for a master level course in product design. In addition, we discuss several cases from this course, applying the process to designing engaging interactive urban playgrounds. Keywords: Urban, Prototyping, Exercise, Sport, Serious Game, Gender, Obesity, Experience Design Paper Received 30/05/2008; received in revised form 30/07/2008; accepted 08/08/2008.

1. Introduction

The design activities we report on were carried out in the context of the Interactive Technology Design (ITD) course offered at Delft University of Technology, Master in Industrial Design Engineering, Design for Interaction. The designers were teams of students with some experience in user centered methods. The design problem was in the domain of urban playground facilities: traditional urban playgrounds have issues as the playgrounds don’t seem to attract children to the proper extent. The children stay at home playing computer games that deprive them of physical exercise and real world social skills. These problems are especially apparent in the disadvantaged urban

Cite as: Van der Helm, A., Aprile, W., & Keyson, D. (2008). Experience design for interactive products: designing technology augmented urban playgrounds for girls. PsychNology Journal, 6(2), 175 – 190. Retrieved [month] [day], [year], from www.psychnology.org. Corresponding Author: Aadjan van der Helm Faculty of Industrial Design, Room 10-2A-11 to 23, Landbergstraat 15, 2628 CE Delft, The Netherlands Phone: +31 15 2783029 E-mail: [email protected]

173

A. van der Helm, W. Aprile and D. Keyson

areas. In light of the health problems this causes, many initiatives have attempted to develop solutions (de Vries, Bakker, van Overbeek, Boer, Hopman-Rock, 005). Experience prototyping can be a useful tool for involving users in the design process at a level where they can contribute meaningfully to the project. We believe that it is particularly useful for design spaces where emotional involvement and enjoyment are of fundamental importance. Buchenau and Fulton give an operational definition of an experience prototype as “…any kind of representation, in any medium, that is designed to understand, explore or communicate what it might be like to engage with the product, space or system we are designing” (Buchenau & Fulton, 2000). The focus is “the experiential aspect of whatever representations are needed to successfully (re)live or convey an experience with a product, space or system” (ibid.) When involving children as well as other user groups in the design process, it is useful to clarify what is expected of them. User involvement can run from a brief prototype field validation to conceptual and brainstorming input in the ideation phase. (Druin, 2002) describes four different roles that children may have in the design of new technologies: user, tester, informant and design partner. In the project at hand, namely the design of interactive playgrounds for girls, we choose to involve children as informants. This means that their input was sought from the beginning of the design process; however there were no children operating as team members in the design teams. The choice of this specific role was dictated by our desire for early and profound user involvement in a project whose user population was initially quite unknown to the student design teams, balanced by the logistical issues of involving children in design work, and by the curricular necessity of producing a technology-based solution. When involving interactive technology in the design process it is important that it is ready-at-hand1 for the designers to use. Designers should be able to work with sensors, actuators and simulated product functionality in a sketchy manner (Buxton, 2007). They make many 3D sketches to study the complex interdependent relationships between products, users and context. This is made possible by using high-level programming tools and modular sensor systems (Hummels, Overbeeke, & Klooster, 2007). In this way design iteration can be performed quickly steering clear of complex and time consuming engineering work.

1

A tool that is ready-at-hand (zuhanden) enables the designer to concentrate on the content of his task and not on the technology tool that he is using. For example, for a typical user a computer mouse is readyat-hand. This concept, originally Heidegger’s, was applied to the design context and brought to the fore recently (Dourish, 2001).

174

Experience Design for Interactive Products

From a method point of view, the purpose of our work is to develop a system that solves a specific problem that we will detail below. Similar to (Jensen & Skov, 2005), our design approach has an engineering purpose, i.e. to develop a system that solves a specific problem. The design by experience approach applies the methods of field study, action research and, more importantly, builds on the notion of applied research. Applied research is defined as “…research where intuition, experience, deduction, and induction are used to analyze a specific research problem [through] trial and error based on the capabilities of the researcher” (ibid.) In this paper a detailed description of a design process developed for the Interactive Technology Design master course is presented. Student design examples of the resulting projects for urban interactive girl playgrounds are presented. Finally, conclusions are drawn about the application of the described design process.

2. Design Brief

Several Dutch municipalities have observed that the extent of physical activity of children in city playgrounds has reduced in recent years. This phenomenon has major health consequences for children and thus for the future adult population. In 2006, the Dutch Ministry of Health, Welfare and Sport commissioned TNO, a Dutch research organization, to study this problem and find the success factors that make urban playgrounds attractive to primary schoolchildren. The study involved observing children and measuring their energy expenditure at six playgrounds in various disadvantaged neighborhoods in big cities of the Netherlands (Bakker et al., 2008). The study delivered a set of recommendations of which TNO selected two for the design brief to the ITD students. Firstly, it was suggested to involve interactive technology in the urban playground facilities to provide a closer match to a generation of children accustomed to computer games and interactive products. Secondly, it was suggested to pay more attention to make the playgrounds attractive to girls because this group was underrepresented at the existing urban playground facilities. TNO’s design brief was used in the ITD course along with other briefs on very different topics. Apart from involvement in the launch of the assignment and in the final design reviews, staff from TNO has played an ongoing role in providing expert feedback. Students were instructed to consider many possible categories of design solutions: wearables, portable devices, cell phones or web-based services and of

175

A. van der Helm, W. Aprile and D. Keyson

course site installations. In instructing the students it was made clear that the fact that the design brief was related to a specific class of locations did not mean that the solution had to be an intervention on the physical space of a playground. Nonetheless, the majority of the student teams chose this type of solution. At the same time, students were instructed to use interactive technology in developing their concepts, as part of the course requirements. Although this is not a technology-driven project and no specific technology was imposed, one of the objectives of the course is to increase the students’ skills in the area of interactive prototypes: this requires, among other things, knowledge of and exposure to certain technologies. Lastly, we asked the students to focus on experience rather than on context, background and historical precedents.

3. Design Process

The experience driven design process was divided into 3 periods, the first lasting one month, with a focus on physical tinkering and making experiential prototypes in many short iterations. The second, also lasting one month, involved using interactive technology, and it required several iterations of slightly longer duration. Lastly, the third lasted two months, with a focus on involving children in two sequential user-tests. The threefold division of the design process was mainly based on educational goals so as to expose the students to using sensors and a software development tool. The changing of pace in the three periods, from short to longer iterations, reflects the progressive clarification of the concept and the increasing depth and fidelity of the experiential prototypes In the first period hands-on workshops were held with students to introduce the students to the interactive technology tools e.g. Phidgets and Max/MSP. Throughout the whole semester the students worked on their design brief making experiential prototypes, testing these, reflecting on the outcome and redesigning the concept. This process was repeated again and again in sync with each stage of the course. Throughout the entire semester in addition to working on their design brief, the students were provided with lectures focusing on aspects of interaction design. Faculty members as well as outside experts gave talks, for example on cultural aspects, child play, aesthetic aspects, and other interaction design cases.

176

Experience Design for Interactive Products

3.1 Physical Experiential Prototypes The first assignment for the students was to create a booklet. The approach was based on work by (Aprile, Boland, & Mirti, 2006). In this assignment the process considered most important was: understanding user needs, getting acquainted with where to find material, tools, and workspace, forming workgroups. The deliverable was a physical booklet for storing notes about the project or even better with some meaningful relationship with the design brief.

Figure 1. A collage project scrapbook with communication capabilities.

The students were then asked to design a concept and present it by including physical prototypes and video/audio prototypes in their presentation. The physical prototypes underwent several iterations, testing occurred with each group of students. To communicate the dynamical aspects of their concept we asked them to present a video prototype (Vertelney, 1989) or use playacting (Boess, Saakes, & Hummels, 2007) or Wizard-of-Oz techniques. The period finished with a plenary presentation and discussion of all projects. 3.2 Involving Interactive Technology The second period of the design process focused on the actual design of interactive prototypes. During the previous period the students were familiarized with Phidgets and Max/MSP: Phidgets is a modular system of electronic sensors and actuators that requires no technical skills for assembly and interfaces with Max/MSP; Max/MSP is a visual dataflow programming language that is easy to use for non-programmers. Students were also provided with a collection of Max/MSP-patches that helped them to interface the Phidgets and perform basic sensing and control tasks. The students were instructed to use these tools to develop their interactive prototypes.

177

A. van der Helm, W. Aprile and D. Keyson

The design phase concluded with plenary presentation and discussion in which each group first described their concept, followed by a demonstration and a short test by a student not from the presenting group. At the end of this part of the design process, students were asked to evaluate their work thus far and come up with a redesigned concept that would be delivered as an A5 postcard sent to the teaching staff, see Figure 2.

. Figure 2. The postcard format, applied in our lab in the past by Pieter Jan Stappers, enforces a concise and economical presentation of a concept.

One side of the postcard could be used to visually explain the concept, while only part of the other side could be used to explain the concept in words - the rest was for the stamp, recipient and address information. In addition the students had to send in a second postcard of a technical nature, featuring a diagram of the required technology on one side and a parts list on the other. The technical postcard forces students to assess realistically what it takes in terms of time, money and equipment to build their intended prototype and it is useful for the teaching staff to assemble a shopping list of parts and assess the load on the lab personnel. The staff worked on the principle of providing a technology array that the students could choose from: it was felt that the proper selection of the technology set to use, with its attendant limits, availability of local expertise and practical difficulties was an educational part of process.

3.3 Involving Users The third period began with feedback from the teaching staff on the delivered postcards to further refine the concepts and tune communication about the technical aspects of the concept. During the first two months of this phase user testing was conducted. This allowed the students to evaluate their interactive prototype with

178

Experience Design for Interactive Products

children, giving rise to a further iteration. We invited 8 girls in the age range of 8-12 who participated with consent from their parents and without compensation. The girls played 9 different games in groups of 2 for a period of 2 hours; we rotated the groups every 15 minutes so each game was played by at least 3 different groups. The students were then asked to provide a plan for building a prototype for the final user test at the end of the period. They were given a small budget and had to specify a building plan for a prototype that could withstand the children use for one afternoon. During the construction process the students were assisted on technical matters. The students were instructed to finalize the prototype one week in advance to allow for some testing and tuning. 3.4 Final Prototype To arrive at the final prototypes, two more technologies were introduced to the students: Flash and Arduino. Though no formal instruction was given in class about Flash or Arduino, lab assistance was provided for the students that wanted to use the technologies. Many students already had some level of knowledge of Adobe Flash (formerly Macromedia Flash), but none had worked with Arduino. Arduino is a physical computing platform that allows hobbyists and students to build simple interactive prototypes: it consists of an electronic board that includes an AVR microcontroller, and a simplified cross-platform IDE that hides the complexities of the AVR toolchain. The Arduino programming language is basically C. With Arduino and Flash on the stage concurrently with Max/MSP and Phidgets, the staff and the students engaged in a process of iterative fitting of the technology to the prototype according to the student’s abilities. This process included critical technology clinics, consisting of brief, 15-minute sessions focusing on specific prototype issues, moments of triage, during which it was decided which features of the concept would survive into the prototype and technological escalation, when the staff suggested that a simpler technology (like hacking a USB keyboard) was not sufficient any more, and that the group should escalate to something more powerful and complex. Iteration was absolutely necessary, because the students were building up their technological abilities as the course progressed, refining their tools while working on the prototype itself – in accordance with the principles of situated learning.

179

A. van der Helm, W. Aprile and D. Keyson

4. Evolution of the Design

In the following section four example student projects are presented. The projects have been chosen because they represent very different angles of attack to the same design brief, namely urban girls in the Netherlands not doing enough exercise. We are going to discuss a wearable device, a portable device, a stationary device and a site installation to show a variety of possible approaches.

4.1 S’Buzz, a Wearable The S’Buzz group chose to design a wearable product to support a non-verbal dance game of “follow the leader”. After the initial physical tinkering (see Figure 3), the team abandoned the idea of high resolution motion analysis and decided that the two players would communicate only through movement and haptic feedback through vibration. In this case, an explicitly low-resolution approach was chosen, based on the team technological expertise and on initial user testing that revealed that a significant level of enjoyment could be obtained with simple movement detection.

Figure 3. A very early prototype of S’Buzz still includes a screen that was later rejected.

The finished product will use shock sensors to detect the leader’s movements, and cell phone vibrators to convey game hints to the other player. This project underwent significant change during its development: initially the sensors and vibrators were mounted on the ankle and knee joints. Sensing was done via accelerometers and there was an idea to capture dance movements and approximate motion capture. Subsequent prototypes use simpler shock sensors instead of accelerometers, employ

180

Experience Design for Interactive Products

a more abstract representation of movements and move the knee sensors to the wrist to capture a broader range of body motion. The experiential prototype was tested on users through a Wizard of Oz prototype in order to validate the concept without making a substantial investment in a specific technology, see Figure 4.

Figure 4. Users from the target group found original ways to use the prototype, in this case by splitting the vibrating bands between two girls.

The final prototype of the S’Buzz project will use two Bluetooth Arduino boards communicating directly between themselves. 4.2 Nelson, a Portable Object The group that designed Nelson concentrated on adding emotional and relational aspects, normally associated with a pet or with a zoomorphic plush toy, to a very common object: a ball. The team considered that soccer, particularly in the version played inside the panna cages, is a fast, competitive and aggressive game. Nelson was designed to emphasize the social and relational aspects of play, and to remove the competence threshold that –it was found in early field studies - is one of the factors that prevents girls from wanting to use the urban sport facilities, see Figure 5. The field studies additionally show that the girls indeed go to playground areas but they don’t join the boys in the football games.

181

A. van der Helm, W. Aprile and D. Keyson

Figure 5. Nelson presents itself as a friendly/familiar object endowed with a playful, childish, behavior.

Acceleration and shock sensors inside Nelson allow it to sense whether it is being rolled around, thrown, spun or kicked. Limited computing abilities (a remote PC in the experiential prototype, a fully embedded Arduino system in the final version) allow it to retain state information and to guide the user through a progression of more complex ball-playing activities. Nelson expresses its state and communicates with its users through a color LED and cute nonverbal sounds. By exploiting the child’s aptitude for make-believe play and for imbuing objects with life-like attributes (see Figure 6), Nelson bases its interface on emotions: you don’t play to reach a high score, you play to make Nelson happy! 4.3 Leo, Urban Furniture The Leo groups set out with the clear targets of attracting girls to the playground and improving their soccer skills. It also stated that forcing the girls to play with the boys was not an objective. After some user research, the group decided to go for an unisex solution and to shift the focus from winning to playing. LEO (Light Emitting Objects) presents itself as a blobby, tree-like addition to the playground. It stands outside of the formal game areas. The skin of Leo is covered with impact sensors: the sensors can be activated by ball impact but also by a hand hitting the surface, see Figure 7.

182

Experience Design for Interactive Products

Figure 6. Nelson depends critically on evoking a strong emotional response.

Figure 7. Leo final prototype: its large physical form and large array of sensors permit easy and satisfying multiuser interaction.

The behaviors proposed are all intentionally simple and generic: the group is not prescribing a specific game or even a mode. Players can e.g. decide to play collaboratively at slapping Leo to turn on all its lights in the shortest time: or that the game consists of one team trying to turn on all the lights, and another one trying to turn them off.

183

A. van der Helm, W. Aprile and D. Keyson

One fixed behavior is that Leo starts in a dark mode, with all the lights turned off. This was done in accordance with user testing that revealed that turning the lights on was associated with “growth” and “making the object beautiful” and thus created positive emotional connotations. Leo has no additional interface elements such as buttons: it can be reset or sent into another game mode by sending the ball through portions of it like a hoop or a tube built into the base.

4.4 AudioPlay, A Site Intervention The AudioPlay group decided to concentrate on the 4-12 years old age group. As at this age fantasy play is a strong attractive, the group decided to explore imaginary environments such as the Jungle, the Waterworld and the Forest. From the beginning, this group decided to concentrate on acting on the space that is outside of the existing panna cage or football court, in order not to create conflict with other users of the inside space. Because of the well-known technological and logistical difficulties with displaying images, even at low quality, in outside locations, the group decided to work strictly with sound; in a similar vein, it was decided to go for a concept that could stand the use, abuse and extra-intentional use that any public installation must face. The initial concept was, in abstract term, a sensitive surface containing a grid of floortile sized pressure sensors or switches connected to a processing unit that can output spatial sound through a system of four speakers, see Figure 8. This would be a platform for at least three different games, one based on chasing frogs, one where the player crosses a stream using stepping stones, and a third one based on performing a Native American rain dance. This concept remained stable through the evolution of the project: although though a process of progressive narrowing in scope only the frogs-squashing game was developed, the hardware and part of the software remain generic enough that the other games could be implemented as well, see Figure 9. The possibility of upgrading the software and introducing new and seasonal games to a stable hardware platform is an important feature in a concept that implies the building of rather costly and stable infrastructure in a public area. In other words, this project addresses the issue of content and makes possible to retain a feeling of novelty and excitement.

184

Experience Design for Interactive Products

Figure 8. Early user testing: the physical structure of the board is undefined and only two of the four planned speakers are present, but useful user observations can already be made.

Based on user input and observation, scoring was left out intentionally: girls can enter and exit a game of Frogs at any moment, and more than one girl can be playing at the same time. This was done in accordance with the perceived user group reluctance at engaging in excessively rigid, performance-based games.

Figure 9. The final prototype being tested hard by typical users. The game can be played by one to several girls at the same time.

185

A. van der Helm, W. Aprile and D. Keyson

5. Conclusions

In comparing the design by experience approach to more formal top-down methods, in which for example user requirements are formalized and then translated into design concepts via a task model, the ED (Experience Design) method appears to be particularly well suited to the design of products in which the sense of creativity and engagement are central to the product. Given the wide range of design possibilities afforded by interactive technology when applied to physical environments such as an urban playground, and the inherent resulting design complexity, the ED method offers the possibility to rapidly develop and test ideas via a rapid iterative trial and error approach, accompanied by actual experience. This very process appears to further lead to new ideas that may have otherwise not been conceived. Underlying the approach is the use of rapid prototyping tools such as Max/MSP and modular hardware kits such as Phidgets. Research is currently being conducted towards developing a middleware toolkit that will further enhance the reusability of software-hardware combinations, while still striving to maintain the freedom of a creative and experimental design space.

Figure 11. The BeatBox installation features a physical interface to sampling and remixing capabilities.

Although the mentioned high-level tools have proven beneficial to the early stages of the design process allowing the designers to quickly iterate over design ideas, it turns out that applying these tools to later stages of the design process is sometimes problematic for concepts that require e.g. small form factors and wireless communication. To build experiential prototypes for these types of concepts one must make a technology shift from easily programmable, familiar, personal computer platforms to the more difficult class of embedded platforms. This takes the designer out of the loop and involves engineers with a consequential loss of designer influence.

186

Experience Design for Interactive Products

One may conclude that, despite the demonstrated appeal of interactive playgrounds, current design practices may cause the lack of widespread real-world implementations. In this context the current paper aims to pave a new way that may assist urban designers to think and design “out of the box”.

Several points can be made about the structure and the various stages of the outline design process. The postcards worked well to force the students to be brief and clear about their concept. The technical version was a good tool to tune the communication between teaching staff and students. Finishing the prototype one week in advance of the user test gave the students an excellent opportunity to play with the prototype they had been working on for the whole semester.

6. Acknowledgments

We are indebted to Caroline Hummels: she started ITD together with the first author in 2005 and her ideas are still present throughout the structure of the course. We wish to thank all the students of the ITD 2008 course for their enthusiasm and hard work. We want to thank the girls who participated in the user tests and their parents (who provided us written permission to publish the pictures of the user tests). We also want to thank Tinus Jongert from TNO, who initiated the collaboration in ITD this year, for his involvement and practical help.

7. References Aprile, W., Boland, B., & Mirti, S. (2006). Interaction Design Primer. Milan: Postmediabooks. Bakker, I., Vries, S. I. d., Boogaard, C. M. H. v. d., Hirtum, W. J. E. M. v., Joore, J. P., & Jongert, M. W. A. (2008). Playground van de toekomst. Succesvolle speelplekken voor basisscholieren. Leiden: TNO Kwaliteit van Leven. Boess, S., Saakes, D., & Hummels, C. (2007 February). When is role playing really experiential?: case studies. 1st International Conference on Tangible and Embedded Interaction February 15-17, Baton Rouge, Louisiana, USA.

187

A. van der Helm, W. Aprile and D. Keyson

Buchenau, M., & Fulton, S. J. (2000). Experience prototyping. 3rd conference on Designing interactive systems: processes, practices, methods, and techniques, August 17 - 19, New York City, NY, USA. Buxton, W. (2007). Sketching user experience – Getting the design right and the right design. san Francisco, CA: Morgan Kaufmann. de Vries, S.I., Bakker, S.I., van Overbeek, K., Boer, N.D. & Hopman-Rock, M. (2005). Kinderen in prioriteitswijken: lichamelijke (in)activiteit en overgewich. Leiden: TNO Kwaliteit van Leven. Dourish, P. (2001). Where the Action Is: The Foundations of Embodied Interaction. Cambridge, MA: MIT Press. Druin, A. (2002). The role of children in the design of new technology. Behavior and Information Technology, 21(1), 1-25. Hummels, C., Overbeeke, K. C. J., & Klooster, S. (2007). Move to get moved: a search for methods, tools and knowledge to design for expressive and rich movementbased interaction. Personal and Ubiquitous Computing, 11(8), 677-690. Jensen, J. J., & Skov, M. B. (2005 June). A review of research methods in children's technology design. 4th International Conference for Interaction Design and Children June 8-10, Boulder, Colorado, USA. Vertelney, L. (1989). Using video to prototype user interfaces. SIGCHI Bull., 21(2), 5761.

188

PsychNology Journal, 2008 Volume 6, Number 2, 189 – 201

Decoding Cognitive States from fMRI Data Using Support Vector Regression Maria Grazia Di Bono Department of Developmental Psychology and Socialization, University of Padova (Italy)

and Marco Zorzi Department of General Psychology and Center for Cognitive Science, University of Padova (Italy)

ABSTRACT In this paper we describe a method based on Support Vector machines for Regression (SVR) to decode cognitive states from functional Magnetic Resonance Imaging (fMRI) data. In the context of the Pittsburgh Brain Activity Interpretation Competition (PBAIC, 2007), three participants were scanned during three runs of 20-minute immersion in a Virtual Reality Environment (VRE) where they played a game that engaged them in various search tasks. A set of objective feature ratings was automatically extracted from the VRE during the scanning session, whereas a set of subjective features was then derived from each individual experience. The aim of the present study was to explore the feasibility of the SVR approach in the case of an extremely complex regression problem, in which subjective experience of participants immersed in a VRE had to be predicted from their fMRI data. The proposed methodology was modeled as a multiphase process: a pre-processing phase, based on a filter approach, for fMRI image voxel selection, and a prediction phase, implemented by nonlinear SVR, for decoding subjective cognitive states from the selected voxel time series. Results highlight the generalization ability of nonlinear SVR, making this approach particularly interesting for real world application of Brain Computer Interface (BCI). Keywords: Brain Computer Interfaces, Signal Processing, fMRI Data, Multivariate Analysis, Support Vector Machine. Paper Received 04/09/2007; received in revised form 15/08/2008; accepted 20/08/2008.

1. Introduction

Recent advances in brain imaging and machine learning provide the foundations for the development of Brain-Computer Interfaces (BCI) based on functional Magnetic Resonance Imaging (fMRI) (Weiskopf et al., 2004). In the last decade, fMRI has

Cite as: Di Bono, M. G., & Zorzi, M. (2008). Decoding Cognitive States from fMRI Data Using Support Vector Regression. PsychNology Journal, 6(2), 191 – 203. Retrieved [month] [day], [year], from www.psychnology.org. Corresponding Author: Maria Grazia Di Bono Department of Developmental Psychology and Socialization, University of Padova Via Venezia 12, 35131 Padova, Italy E-mail: [email protected]

189

M. G. Di Bono, M. Zorzi

become the most widely used non-invasive technique for investigating human brain functions. The main problem addressed by fMRI studies is to correlate neural signals temporally changing in different cortical areas to certain events (task conditions) opportunely encoded in the experimental paradigm. A BCI based on fMRI needs to interpret the relationships between neural signals from fMRI data and the subjective experience of scanned participants, in order to perform a sort of mind reading and to translate the predicted mental states into actions. Conventional fMRI data analysis techniques are based on the statistical Parametric Approaches (e.g. General Linear Model

GLM) which correlate external regressors

(task conditions) with the activity in specific brain regions, generating brain maps of localised activity (Friston et al., 1995). Traditional statistical methods measure the activity from many thousands of voxels in the brain images, analysing each voxel time series separately and comparing two or more task conditions at each voxel location. Recent methods, belonging to the class of Multivariate Analysis (Non Parametric Approaches), have the potential to improve our understanding of the complex pattern of brain activity measured by fMRI. These approaches are based on boosting the weak information available at each voxel location by a simultaneous analysis of hundreds or thousands of voxels to predict specific cognitive states. Many pattern recognition methods have been employed as multivariate techniques for fMRI data analysis. Machine learning techniques based on artificial neural networks (Chuang, Chiu, Lin, & Chen, 1999; Voultsidou, Dodel, & Herrmann, 2005) or different clustering algorithms (Meyer & Chinrungrueng, 2003; Liao, 2005; Heller, Stanley, Yekutieli, Rubin, & Beniamini 2006; Chen, Bouman, & Lowe, 2004; Chen, H., Yuan, Yao, Chen, L., & Chen, W., 2006) have been employed for time series data analysis in different domain applications, among which fMRI data analysis. Other methodologies, such as independent component analysis (ICA), have also been used for processing fMRI data (Hu et al., 2005; Meyer-Baese, Wismueller, & Lange, 2004). One of the most widely used Machine Learning techniques for fMRI data analysis are Support Vector Machines (SVM), which are kernel-based methods designed to find functions of the input data that enable both classification and regression (Vapnik, 1995). In particular, SVMs classify data with different class labels by determining a set of support vectors, which are members of the training set, outlining a hyperplane in the feature space. SVM provides a mechanism to fit the hyperplane surface to the training data using a specific kernel function. SMV classifiers are well known for their very good

190

Decoding Cognitive States from fMRI Data Using Support Vector Regression

generalization ability and have been used in recent studies of fMRI data analysis (Cox & Savoy, 2003; Mitchell et al., 2004; Kamitani & Tong, 2005; Haynes & Rees, 2005; La Conte et al., 2005, Mourão-Miranda, Friston, & Brammer, 2007). However, most of the previous studies have focused on classification problems. In the case of regression problems, the goal is to find a functional shape for the function that can correctly predict new cases that the SVM has not been presented with before. This latter method is usually referred to as Support Vector Regression (SVR; see Smola & Schölkopf, 2004, for a review). Thus, our goal was to explore different techniques of feature selection and the feasibility of SVR in the case of an extremely complex regression problem whereby the subjective experience of participants immersed in a virtual reality environment (VRE) must be predicted from a set of fMRI data.

2. Materials and Methods

The VRE experiment was organized in the context of the Pittsburgh Brain Activity Interpretation Competition (PBAIC, 2007). The purpose of this competition was to infer subjective experience of the participants experiencing an immersion in a virtual reality environment, from a contextually gathered set of fMRI data. The VRE experiment, the gathered fMRI data and the proposed fMRI decoding method are described in the next sections. 2.1 Participants Fifteen subjects participated in this study. In particular, after a selection procedure made by the PBAIC staff (see PBAIC, 2007 for details), the only data available for the competition were relative to three subjects (age range: 20-26 years). 2.2 Procedure Participants were instructed to play a game in a virtual world during three runs of fMRI data acquisition. In the game they were paid by an anthropology department grant to gather information about urban people. In particular, they were visiting several times the virtual reality environment, outside and inside some specified places, and were instructed to collect as much as possible samples of toy weapons and fruits, in a predefined order; moreover, they had to take pictures of people with piercings and

191

M. G. Di Bono, M. Zorzi

avoid an aggressive dog. Participants were also informed and asked to keep in mind that any money obtained in the game corresponded to an earning in real life. The study was completed over a period of four days. During the first day, participants watched a 13-minute video for a first phase of familiarization with the VRE and completed a battery of questionnaires, implicit association tests for assessing the ingroup/outgroup and canines perception, the level of anxiety and sickness, the sense of direction, the computer familiarity scale (see PBAIC, 2007 for details). During the second day, participants played the virtual reality game outside the scanner and every 2 minutes they were asked to rate their level of sickness. At the end of the session they completed three questionnaires for assessing the level of simulator sickness and the level of presence and comfort during the navigation. In the third day, subjects were asked to perform search tasks during three 20 minute runs of the game inside the scanner. As in the previous day, participants were asked to rate every 2 minutes their level of sickness. During the navigation of the virtual world, a set of target feature ratings was gathered for a total of 13 required and 10 optional features. In particular, some features were obtained through software loggings of subject actions in the virtual world, soundtrack analysis and eye-tracking based analysis of video from each run, and were referred to as objective features (e.g., Hits: whenever subjects correctly picked up fruit or weapon or took pictures of persons with piercings; Instructions: whenever task instructions were presented; Search people; Search weapons; Search fruit; Faces: whenever subjects looked at faces of pierced or unpierced people). The other features were referred to as subjective features, such as the level of arousal or valence (how positive or negative is the environment), and they were assigned by each participant on the last day of the study while watching a playback of the own actions in the virtual world. Figure 1 shows a screenshot of the virtual world and the behavioural time vector ratings of multiple categories representing what participants perceived/experienced during the navigation of the virtual world. For the first two runs, videos of the subject’s path through the virtual reality environment along with 20 minutes of continuous fMRI data and target feature ratings were provided, whereas for the third run the ratings were not provided.

192

Decoding Cognitive States from fMRI Data Using Support Vector Regression

Figure 1. Screenshot of the virtual world (a); illustration of behavioral time vector ratings of multiple categories (b).

The purpose of the competition was to predict feature rating vectors for the third segment. 2.3 fMRI Dataset 3T EPI fMRI data from three subjects in three runs were downloadable from the Pittsburgh Brain Activity Interpretation Competition web site (PBAIC 2007). Twenty minutes of continuous functional data, consisting of 704 of 64x64x34 image volumes, were available for each participant in each run. The acquired images were motion corrected, slice time corrected, detrended and spatially normalized. The fMRI data of the first two runs were used for the learning phase, whereas the last run was used as test set for the prediction of the related ratings. 2.4 fMRI Decoding Method The proposed fMRI decoding method is modelled as a multiphase process (preprocessing phase, prediction phase) as shown in Figure 2. In the pre-processing phase we first extracted only those voxels belonging to the brain cortex, by using the respective masks available for each subject. Then, for each subject, all brain voxel time series were normalized to the same mean intensity and temporally filtered. We then performed the voxel subset selection based on a filter approach. For each feature rating, convolved with the canonical Hemodynamic Response Function (HRF), we computed the correlation with each voxel time series in the image volumes, separately for each subject and run. Then we selected only those voxels showing a correlation that was significant at the 0.05 level (r > 0.45, p< 0.05). The subsets extracted for the first two runs, used as training set, were merged to form the final set of voxel time series.

193

M. G. Di Bono, M. Zorzi

Figure 2. Architecture of the system.

In order to validate the methodology, we initially used only the first run for training and the second one for testing the method. In particular we developed two different approaches for the pre-processing phase. In the first approach we used the subset of voxels extracted by the correlation filter directly as input to SVM. In the second approach we clustered the selected voxels using hierarchical clustering and k-means algorithms into 100 clusters, then we extracted the centroid of each defined cluster and used it like a super-voxel as input to SVM for the prediction phase. After this initial step, dedicated to the validation of the methodology, we selected the pre-processing approach that provided the best results with the first two runs. Thus, in the prediction phase, we used run 1 and run 2 of the same subject as training data to predict, in the test phase, the feature ratings for the third run. For each subject each feature was predicted separately. 2.5 SVM Regression (SVR) Support Vector Machines (SVM) were developed by Vapnik (1995) to solve classification problems, but recently SVM have been successfully extended to regression and density estimation problems (Smola & Schölkopf, 2004, for a review). Suppose we have training data

{(x1 , y1 ), (x2 , y 2 ),..., (x N , y N )}

M

×

, in the -

insensitive Support Vector Regression (SVR) technique (Vapnik, 1995) the goal is to find the function f ( x ) that has at most

deviation from the actually obtained target y i

for all the vectors of observation xi in the training data, and at the same time is as flat as possible. In nonlinear SVR, the input vector x is first mapped onto a high

194

Decoding Cognitive States from fMRI Data Using Support Vector Regression

dimensional feature space using some fixed nonlinear mapping, and then a linear model is constructed in this feature space as:

f ( x ) = w, ,

where

(x )

+b , w M

denotes the dot product in

M

,b

(1)

(x )

, w is the weight vector,

is the

nonlinear mapping function, and b is the bias. However, one also want to allow for some errors, thus, analogously to the soft margin loss function adopted by Vapnik (1995), one can introduce (non-negative) slack variables

i

,

* i

, i = 1,..., N to measure the deviation of training samples outside the

-

insensitive zone. Thus the problem can be modelled as a convex optimization problem that, in the nonlinear case, can be formulated as:

min

1 2 w +C 2

n i =1

(

i

+

* i

w,

yi

) subject to

( xi )

w, i

,

* i

( xi )

b

+

* i

+b

yi

+

i

(2)

0

In most cases, problem (2) can be easily solved in its dual formulation, that provides the key to extend SVR to nonlinear cases. The solution of the dual convex problem can be expressed as:

f (x ) = where

i

,

* i

N i =1

(

* i

i

)K (x , x ) + b

(3)

i

(

are the so called Lagrange multipliers and K x, x '

)

is the kernel

function. SVM algorithm only depends on dot products between patterns xi , hence it is

(

)

(x ),

sufficient to know the kernel function K x, x ' =

(x ) '

rather then

explicitly.

From the optimality constraints that are behind the dual problem definition, it is possible to derive w as a linear combination of the training patterns:

w=

N i =1

(

* i

i

)

( xi )

(4)

The difference from the linear case is that w can no longer given explicitly, whereas the flattest function f ( x ) , that has to be found in the feature space (not in the input space), can be expressed through the trick of the kernel function. Several functions, such as polynomial functions, radial basis functions (RBF), splines, hyperbolic tangent functions, can be used as kernel in SVR (Burges, 1998; Smola & Schölkopf, 2004). These functions have to satisfy the conditions of the Mercer’s theorem (Mercer, 1909), that is the conditions under which it is possible to write k ( x, x')

195

M. G. Di Bono, M. Zorzi

as a dot product in some feature space. In particular, translation invariant kernels

k ( x, x ' ) = k ( x

x') , that are proved be admissible kernels, are widespread, among

which one of the most widely used is the RBF kernel that can be written as:

k ( x, x ' ) = e

x x' 2

2

2

(5)

It is well known that SVM generalization performance (estimation accuracy) depends on a good setting of meta-parameters C,

and the kernel parameters (Burges, 1998;

Smola & Schölkopf, 2004). The parameter C determines the trade off between the model complexity and the degree to which deviations larger than optimization formulation. Parameter

controls the width of the

used to fit the training data. The value of

are tolerated in -insensitive zone,

can affect the number of support vectors

used to construct the regression function. The bigger , the fewer support vectors are selected. Hence, both C and

-values affect model complexity, but in a different way.

3. Results and Discussion

In developing the decoding method, we tested and tuned the SVR using run 1 for all subjects as training and run 2 as test set. We then applied our method for predicting the feature ratings of the third run after training on the voxels and ratings of the first two runs. All the algorithms used here were developed in Matlab 7.0.1 (R14), by using the SVM toolbox (Gunn, 2007) for developing regression algorithms and the tools of NIfTI (ANALYZE) MR image (Shen, 2005) for fMRI volume visualization. In the pre-processing phase, all brain voxel time series were normalized to the same mean intensity (subtracting their mean and dividing by their standard deviation) and temporally filtered, by using a running average filter with window size = 7 (Figure 3). As mentioned in section 2.4, we tested two different approaches of SVR input preparation, one based on a correlation filter for voxel selection and the other based on voxel selection followed by feature extraction. In particular, in the first approach we extracted for each subject the set of voxels showing a correlation with each feature rating (convolved with the canonical HRF) of run 1 that was significant at the 0.05 level (r > 0.45, p< 0.05) and merged them to obtain the final set. We then used the coordinates of the extracted voxels for selecting the same set of voxels from run 2. SVR was finally employed for predicting the feature ratings of the second run using the voxel time series and the target ratings of the first run as training set. In the second

196

Decoding Cognitive States from fMRI Data Using Support Vector Regression

approach, we applied a clustering step in the pre-processing phase based on hierarchical clustering and k-means algorithm, but we did not find any improvement in run 2 feature rating prediction. In contrast, we observed a general degradation of the prediction performance. We suggest that this deterioration could be due to a loss of the original distributed information which was compressed in the clustering phase. Thus, for the prediction of the target feature ratings of the third run we did not apply any clustering procedure or feature extraction.

Figure 3. Example of voxel time series after normalization (in blue) and temporal filtering (in red).

Figure 4. Selected voxels (subject 14) after computing the correlation with the convolved feature ratings (all features) on run 1 and run 2 and selecting the intersection between the two runs.

197

M. G. Di Bono, M. Zorzi

After the validation of the method, we used it for predicting target feature ratings of the third run. We therefore extracted two different subset of voxel time series, applying the correlation filter for both run 1 and run 2 with their respective feature ratings, and considered only the intersection between them as the final set to be used for selecting the voxels in the third run (Figure 4). SVR was then employed to predict the feature ratings for run 3, using voxels and ratings of the first and second run as training set. As explained in section 2.4.1, in the prediction phase the choice of the regularization constant C, the kernel function and its parameters was a fundamental key for obtaining good generalization performance. We explored different sets of critical parameters, empirically obtaining the best results using C=2 and the Exponential Radial Basis function (with

=6) as nonlinear kernel.

Table 1 shows the results obtained with the first pre-processing approach for the prediction of the target feature ratings relative to the third run. The prediction scores are expressed in terms of standardised correlation coefficients, and, at least for the third run, were computed directly by the Experience Based Cognition (EBC) Project staff that had the relative target feature ratings. In particular, the scoring algorithm adopted by the EBC team was based on two steps. In the first step, the Pearson correlation coefficient r between the predicted feature and the observed subject rating was computed for each feature. In the second step, the Fisher transformation, that makes the scores normally distributed, was applied to each correlation calculated in the previous step, according to:

z' =

1 !1+ r log 2 1 r

(12)

The obtained predictions, at least for a subset of features, reached a good correlation with the target ones. A consistency across subjects can be noted with respect to the features that are more reliably predicted.

Subject 1

Subject 13

Subject 14

Body

0.2421

0.4178

0.3438

Velocity

0.4554

0.5197

0.7126

Hits

0.3035

0.4339

0.3959

Instruction

0.5453

0.7957

0.7842

Faces

0.2611

0.3595

0.5871

Table 1: The best feature rating predictions, expressed in terms of correlations, achieved on run 3 for all subjects. The best three correlations for each subject are shown in bold font.

198

Decoding Cognitive States from fMRI Data Using Support Vector Regression

In conclusion, the aim of this study was to explore the feasibility of SVR in the case of an extremely complex regression problem, in which subjective experience of participants immersed in a VRE had to be predicted from their fMRI data. We used a decoding method modeled as a multiphase process: a pre-processing phase, based on a filter approach, for fMRI image voxel selection, and a prediction phase, implemented by nonlinear SVR, for decoding subjective cognitive states from the selected voxel time series. Results are quite good, at least for a subset of feature ratings. The emphasis of SVM/SVR on generalization ability makes this approach particularly interesting for realworld applications of BCI (Weiskopf et al., 2004; Piccione et al, 2006; Sitaram et al., 2007; Piccione et al., 2008), in particular when the amount of training data is limited and the input space has a high dimension (as in the case of fMRI data). Planned extensions to this work include the evaluation of different feature extraction techniques to combine with SVM/SVR or the use of embedded methods, in the context of nonlinear kernels, for voxel selection and ranking, in order to extract a more compact and informative set of voxels and to further increase the prediction accuracy.

4. Acknowledgements We thank the Experience Based Cognition (EBC) Project team for providing the fMRI data of the PBAIC 2007. Finally, we would like to thank three anonymous reviewers for their helpful suggestions. This study was supported by a grant from the Cariparo Foundation.

5. References

Chen, S., Bouman, C. A., & Lowe, M. J. (2004). Clustered components analysis for functional MRI. IEEE Transactions on Medical Imaging, 23, 85-98. Chen, H., Yuan, H., Yao, D., Chen, L., & Chen, W. (2006). An integrated neighbourhood correlation and hierarchical clustering approach of functional MRI. IEEE Transactions on Biomedical Engineering, 53, 452-458. Chuang, K. H, Chiu, M. J., Lin, C. C., & Chen, J. H. (1999). Model-free functional MRI analysis using Kohonen clustering neural network and fuzzy C-means. IEEE Trans Med Imaging, 18 (12), 1117-1128.

199

M. G. Di Bono, M. Zorzi

Cox, D. D., & Savoy, R. L. (2003). Functional magnetic resonance imaging (fMRI) ”brain reading”: detecting and classifying distributed patterns of fMRI activity in the human visual cortex. Neuroimage, 19, 261-270. Burges, C. J. C. (1998). A tutorial on support vector machines for pattern recognition. Data Mining and Knowledge Discovery, 2(2), 121-167. Friston, K. J., Holmes, A. P., Worsley, K. J., Poline, J. P., Frith, C. D., & Frackowiak, R. S. J. (1995). Statistical Parametric Maps in Functional Imaging: A General Linear Approach. Human Brain Mapping, 2 (4), 189-210. Gunn, S. R. (2007). Support Vector Machines for Classification and Regression. SVM toolbox

[Computer

software].

Retrieved

August

27,

2007,

from

http://www.isis.ecs.soton.ac.uk/isystems/kernel/svm.zip. Haynes, J. & Rees, G. (2005). Predicting the orientation of invisible stimuli from activity in primary visual cortex. Nature Neuroscience, 8, 686-691. Heller, R., Stanley, D., Yekutieli, D., Rubin, N., & Beniamini, Y. (2006). Cluster-based analysis of FMRI data. NeuroImage, 33, 599-608. Hu, D., Yan, L., Liu, Y., Zhou, Z., Friston, K. J., Tan, C., & Wu, D. (2005). Unified SPM– ICA for fMRI analysis. NeuroImage, 25, 746-755. Kamitani, Y., & Tong, F. (2005). Decoding the visual and subjective contents of the human brain. Nature Neuroscience, 8, 679-685. LaConte, S., Strother, S., Cherkassky, V., Anderson, J., & Hu, X. (2005). Support vector machines for temporal classification of block design of fMRI data. Neuroimage, 26, 317-329. Liao, T. W. (2005). Clustering of Time Series Data - A Survey. Pattern Recognition, 38 (11), 1857-1874. Meyer, F. G., & Chinrungrueng, J. (2003). Analysis of event-related fMRI data using best clustering bases. IEEE Transactions on Medical Imaging, 22, 933–939. Meyer-Baese, A., Wismueller, A., & Lange, O. (2004). Comparison of two exploratory data analysis methods for fMRI: unsupervised clustering versus independent component analysis. IEEE Transactions on Information Technology in Biomedicine, 8, 387-398. Mercer, J. (1909). Functions of positive and negative type and their connection with the theory of integral equations. Philosophical Transactions of the Royal Society, London, A209, 415-446.

200

Decoding Cognitive States from fMRI Data Using Support Vector Regression

Mitchell, T., Hutchinson, R., Niculescu, R. S., Pereira, F., Wang, X., Just, M., & Newman, S. (2004). Learning to Decode Cognitive States from Brain Images, Machine Learning, 57, 145-175. Mourão-Miranda, J., Friston, K. J., & Brammer, M. (2007). Dynamic discrimination analysis: a spatial-temporal SVM. Neuroimage, 36(1), 88-99. PBAIC

(2007).

Retrieved

August

27,

2007,

from

http://www.ebc.pitt.edu/2007/competition.html. Piccione, F., Giorgi, F., Tonin, P., Priftis, K., Giove, S., Silvoni, S., Palmas, G., & Beverina, F. (2006). P300-based brain computer interface: Reliability and performance in healthy and paralised participants. Clin Neurophysiol, 117(3), 531537. Piccione, F., Priftis, K., Tonin, P., Vidale, D., Furlan, R., Cabinato, M., Merico, A., & Piron, L. (2008). Task and Stimulation Paradigm Effects in a P300 Brain Computer Interface Exploitable in a Virtual Environment: A Pilot Study. PsychNology Journal, 6(1), 99-108. Shen, J. (2005). NIfTI (ANALYZE) MR image tool [Computer software]. Retrieved August 27, 2007 from http://www.mathworks.com/matlabcentral/. Sitaram, R., Zhang, H., Guan, C., Thulasidas, M., Hoshi, Y., Ishikawa, A., Shimizu, K., & Birbaumer, N. (2007). Temporal classification of multichannel near-infrared spectroscopy signals of motor imagery for developing a brain-computer interface. Neuroimage, 34(4), 1416-1427. Smola, A., & Schölkopf. B. (2004). A tutorial on support vector regression. Statistics and Computing, 14, 199-222. Vapnik, V. (1995). The Nature of Statistical Learning Theory. New York: SpringerVerlag. Voultsidou, M., Dodel, S. & Herrmann, J., M. (2005). Neural Networks Approach to Clustering of Activity in fMRI Data. IEEE Transactions on Medical Imaging, 24, 987–996. Weiskopf, N., Mathiak, K., Bock, S. W., Scharnowski, F., Veit, R., Grodd, W., Goebel, R., & Birbaumer, N. (2004). Principles of a brain-computer interface (BCI) based on real-time functional magnetic resonance imaging (fMRI). IEEE Transactions on Biomedical Engineering, 51 (6), 966-970.

201

202

PsychNology Journal, 2008 Volume 6, Number 2, 203-216

Contrasting the Effectiveness and Efficiency of Virtual Reality and Real Environments in the Treatment of Acrophobia Carlos M. Coelho

, Carlos F. Silva , Jorge A. Santos , Jennifer Tichon and Guy Wallis

University of Queensland (Australia)

University of Aveiro (Portugal)

University of Minho (Portugal)

ABSTRACT Previous studies reported good results in using virtual reality for the treatment of acrophobia. Similarly this paper reports the use of a virtual environment for the treatment of acrophobia. In the study, 10 subjects were exposed to three sessions of simulated heights in a virtual reality (VR) system, and 5 others were exposed to a real environment. Both groups revealed significant progress in a range of anxiety, avoidance and behaviour measurements when confronted with virtual as well as real height circumstances. Despite VR participants experiencing considerably shorter treatment times than the real-world subjects, significant improvements were recorded on the Behavioural Avoidance Test, the Attitudes Toward Heights Questionnaire and the Acrophobia Questionnaire. These results are suggestive of a possible higher effectiveness and efficiency of VR in treating acrophobia.

Keywords: Acrophobia, heights, virtual reality, fear, treatment. Paper Received 18/11/2007; received in revised form 02/06/2008; accepted 17/07/2008.

1. Introduction

In 1995, Rothbaum and collaborators (1995a) presented the first clinical application of a virtual reality (VR) system to acrophobia. The success of this study motivated the authors to continue their research. In the same year a more extensive study with 20 university students also revealed the effectiveness of treatment via VR. These first Cite as: Coelho, C.M., Silva, C.F., Santos, J.A., Tichon, J., & Wallis, G (2008). Virtual and Real Environments for Acrophobia Desensitisation. PsychNology Journal, 6(2), 205 – 218. Retrieved [month] [day], [year], from www.psychnology.org. Corresponding Author: Carlos M. Coelho School of Human Movement Studies University of Queensland, Level 5, Building 26, St Lucia QLD 4072 Australia Phone: +61 7 33656106 E-mail: [email protected]

203

,

C.M. Coelho, C.F. Silva, J.A. Santos, J. Tichon, G. Wallis

exploratory studies had some limitations related to the absence of a comparison group, who had been exposed to standard treatment. A more recent study (Emmelkamp, Krijn, Hulsbosch, de Vries, Schuemie, & van der Mast, 2002) reproduced the places used in real exposure in a virtual environment. Three weekly one-hour sessions were applied to 33 acrophobic subjects (16 real-world and 17 VR). The VR treatment was as effective as real-world exposure in combating anxiety and avoidance. In contrast to the earlier studies, it was shown that the improvements were demonstrated not only in the self-report, Attitude Towards Heights Questionnaire (ATHQ), but also in the Behavioural Avoidance Test (BAT). These improvements were still present in a 6-month follow-up. A considerable number of studies have now demonstrated the effectiveness of exposure provided by VR systems in the treatment of acrophobia (see Krijn, Emmelkamp, Olafsson, & Biemond, 2004 for a review). Interestingly, Emmelkamp, Bruynzeel, Drost, and van der Mast's (2001) study unintentionally revealed a further benefit of VR sessions for acrophobia, namely, that improved effects were gained more quickly. In their study, all participants received VR treatment in the first two sessions as a first treatment. The subsequent real-world exposure did not lead to a significant improvement in the ATHQ or in the BAT. Since the first treatment was in VR for all participants, the research design unexpectedly created a ceiling effect, leaving little space for improvement in subsequent treatment in the real environment. Therefore, there were more positive results than expected after only two treatment sessions of VR. Time effectiveness in VR is important, especially when using head-mounted displays (HMDs), which easily disrupt visuo-vestibular and proprioceptive signals (Durlach & Mavor, 1995; Emura & Susumu, 1998; Lawson, Graeber, Mead, & Muth, 2002; Kennedy, Jones, Stanney, Ritter, & Drexler, 1996) and cause motion sickness (Reason & Brand, 1975; Stanney, Mourant, & Kennedy, 1998; Kennedy, Stanney, & Dunlap, 2000). Kennedy, Stanney, and Dunlap (2000) recommended short and repeated VR exposures with an interval of a few days. This suggests that the effectiveness of VR treatment needs to be fast, so as to achieve positive results before the onset of motion sickness. The aim of the present study was to compare the effectiveness of real world exposure versus VR exposure in a between-group design of acrophobic patients with varied exposure times. VR treatments were restricted to an average exposure time of

204

Virtual and Real Environments for Acrophobia Desensitisation

approximately 22 minutes per session, while the real environment treatment group received approximately 50 minutes of exposure per session. In accordance with Emmelkamp et al.'s (2001) findings we expect to see a positive treatment outcome in VR and at least as quickly, if not more quickly, than when using traditional real-world exposure. We also expect to avoid severe motion sickness by implementing Kennedy et al’s (2000) recommendations.

2. Materials and Methods

2.1 Participants The participants of this study were individuals suffering from acrophobia who referred themselves for treatment after advertisements (e-mail and local newspaper) were posted around the university campus. Twenty-eight subjects were submitted to a screening process. Eight were excluded for not fulfilling the DSM-IV criteria of acrophobia. Two subjects were excluded because they showed fear of heights in the BAT but not in the VR environment. Another 3 participants abandoned treatment; one due to a long-term holiday and two for unknown reasons. Thus, there were 15 subjects without cardiac or vestibular problems, with normal or corrected-to-normal vision, and with a significant fear of heights, presenting a value of at least 5 on the Subjective Units of Disturbance Scale (SUDS) (Wolpe, 1982) in the last step of the BAT stairs (see the next section on methodological details). The participants comprised 5 men and 10 women, with ages ranging from 18 to 66 years old (mean 37). They all reported suffering a fear of heights over a period ranging from 18 to 55 years. Seven participants reported fear of heights for as long as they could remember. The average beginning age of fear was 5 years old. 2.2 Materials Our software, VRPhobias, was developed using Genes (Generic Environment Simulator), which is a simulator of virtual environments running on top of Performer. The virtual environment recreated the view from the balcony of a hotel. A gradual exposure to height was possible, as the subject rose from the 1st to the 8th storey of the building (total of 24 meters). The geometrical and pictorial characteristics of the façade and buildings facing of the hotel were reproduced in the virtual environment as accurately as possible. The VR scenario did not have moving agents (e.g., people or vehicles) and differed slightly from the real environment in visual detail, resolution,

205

,

C.M. Coelho, C.F. Silva, J.A. Santos, J. Tichon, G. Wallis

colours

and

dynamic

response to movement.

The

software specified

the

corresponding height in floors and the equivalent height to the real perspective of the subject in the hotel (see Figure 1). In our laboratory, participants were close to a balcony rail similar to those found in the hotel. The workstation was a Silicon Graphics, Octane MXE model. The helmet (HMD) was a V6 model — Virtual Reality Research Systems Inc. The system was able to generate the virtual scenario at a rate of about 20–24 frames per second. The tracker was an electromagnetic unit (FASTRACKTM, Polhemus Inc.).

Figure 1. View from the real world (left) and from the virtual reality system (right).

2.3 Instruments The

Kennedy,

Lane,

Berbaum,

and

Lilienthal

(1993)

Simulator

Sickness

Questionnaire (SSQ) was used only to guarantee the participants’ safety, as VR environments may provoke symptoms of cybersickness. The questionnaire was applied before and after the first session. Each time the subjects felt severe symptoms, they were invited, in subsequent sessions, to move their heads more slowly, and remove the helmet and rest after reducing the anxiety associated with a fear of heights. The SUDS (Wolpe, 1982) questionnaire asks respondents to quantify their present level of distress on a l0-point scale. SUDS was used approximately every 5 minutes during the treatment session to determine whether clients were able to proceed to higher levels of exposure or wait for habituation to occur. The SUDS was also used as a pre- and post-treatment measure of anxiety.

206

Virtual and Real Environments for Acrophobia Desensitisation

The anxiety subscale of the Acrophobia Questionnaire (AQ) (Cohen, 1997) was used. This questionnaire describes 20 situations with rating scales to assess anxiety (range 0–6) and avoidance (range 0–3), adding to a total score of between 0 and 180. This test has two subscales: one of anxiety (range 0–120) and another of avoidance (range 0–60). The ATHQ, developed by Abelson and Curtis (1989), includes six semantic scales, rated between 0 and 10, on which the subjects note down their attitudes towards high places. This scale provides for a measurement of two attitude variables: 1) cognitive assessment (good/bad, attractive/terrible, pleasant/unpleasant); and 2) danger assessment (safe/dangerous, non-threatening/threatening, harmless/harmful) (Table 1). The Behavioural Avoidance Test (BAT), which was also used, requests subjects to climb a staircase of 40 steps. The subjects are invited to climb five steps at a time and stop for about 10 seconds to explore the surrounding environment. During this pause, the SUDS test is administered. This procedure progresses until the participant is unable to climb any further or until the end of the stairs. The therapist accompanies the participant along the first 10 steps, in order to better explain the exercise, after which he observes the subject from the floor level. In order to compare the subjects in the behavioural tests, we created a measure of difficulty in heights, which we named the Level of Heights Difficulty (LHD). This measure, which had been used in our previous studies (e.g., Coelho, Santos, Silvério, & Silva, 2006), is the product of the number of sets of stairs that the subject can climb using the BAT, by the anxiety he/she manifests in SUDS. The LHD varies from “no difficulty” (value 0), which corresponds to climbing all sets of stairs (8 sets of 5 steps each) without anxiety (SUDS=0) to “maximum difficulty” (value 80), which corresponds to not climbing any stairs and experiencing the maximum disturbance (SUDS=10). The sets of stairs are counted backwards: 0–5 steps (1st set) = value 8; 6–10 steps (2nd set) = value 7; 11–15 steps (3rd set) = value 6 and so on until 36–40 steps (8th set) = value 1. For example, a subject who attained the 5th set, in this case with value 4, and obtained a SUDS=6, has a LHD=4x6 = 24. 2.4 Procedures Participants took part in 3 weekly sessions, conduced by a clinical psychologist. Before starting the first evaluation session, participants were informed about the exposure procedures, and gave their informed written consent. The overall study

207

,

C.M. Coelho, C.F. Silva, J.A. Santos, J. Tichon, G. Wallis

involved two different experiments. One experiment involved a group of acrophobics treated in a VR environment (n=10) and another group was treated within a real environment (n=5), with the visual characteristics of the VR environment being kept as close as possible to those of the real environment. Participants were not aware of the existence of two different treatment settings. Both assessment and treatment were free of charge. The therapist offered verbal orientation and encouragement to each participant and told him/her that he/she was capable of approaching the balcony, climbing to the various floors and reporting reduced values of subjective distress units. The participants were continuously instructed to look at the floor, explore the environment and stay as long as they could in each situation, until their anxiety diminished. The therapist could observe where each participant was in the virtual environment on a screen, and comment appropriately, as would be expected in a conventional exposure. When anxiety diminished (evaluated by means of SUDS, varying between 0 and 10), the therapist introduced the patient to a higher floor or encouraged the participant to approach the balcony rail. This process was then repeated floor by floor. In order to avoid motion sickness symptoms, participants were encouraged to stop and rest at the first signs of nausea or discomfort. When the anxiety levels diminished (assessed through SUDS, varying between 0–10), the therapist would introduce the patient onto a higher floor or would bring him or her closer to the railing of the balcony, and repeat the process. After about 30 minutes, the session would end; however, the abandonment at a moment of high anxiety was prevented, so it would not facilitate avoidance. The next session would start from where the last session had ended. The therapist’s comments were essentially identical to those expected in a real-world exposure; for example: “Can you try to take your hands off the balcony railing?”; “Would you like to get closer to the balcony?”; “We can now try to walk from one side of the balcony to the other”; “Everything is going well: your anxiety is diminishing by remaining in that situation”. The therapist would also question the subject about his or her thoughts and physical sensations. Participants were not encouraged to undertake exposure exercises outside the therapeutic sessions, and were also advised to avoid alcohol and to sleep normally before each session, in order to prevent increased susceptibility to motion sickness.

208

Virtual and Real Environments for Acrophobia Desensitisation

3. Results The results presented here refer to the SUDS during the behavioural exam (BAT), the AQ and the ATHQ. For the statistical analysis of each group the Wilcoxon test was used and for the comparison between the two groups, the Mann-Witney U test was applied. The group that participated in the treatment through VR exposure was called Group VR, and the group that participated in the treatment through real-world exposure Group R. 3.1 Real Environment Treatment Results Despite the low number of participants in Group R, statistically significant results were obtained for the LHD (Z=-2.032; p<0.05), the ATHQ (Z=-2,023; p<0.05), as well as the AQ (Z=-2.032; p<0.05) (Figure 2). These values are in agreement with the fairly welldocumented power of exposure therapies for the fear of heights (e.g., Abelson & Curtis, 1989; Baker, Cohen, & Saunders, 1973; Emmelkamp & Felten, 1985; Marshall, 1985; Spencer & Conrad, 1989; Williams, Dooseman, & Kleinfield, 1984; Williams, Turner, & Peer, 1985).

60.0

50.0

43.4

43.2

40.0 22.8 24.6

30.0

22.8

pre post

14.6 20.0

10.0

SD=22.4

SD=5.2

SD=6.8

SD=15.8

SD=7.4

SD=7.9

0.0 1

Figure 2. Median values comparing the Level of Heights Difficulty, Attitude Towards Heights Questionnaire and the Acrophobia Questionnaire, in pre- and post-test (Group R).

209

,

C.M. Coelho, C.F. Silva, J.A. Santos, J. Tichon, G. Wallis

3.2 VR Treatment Results Statistically significant results were obtained for the LHD (Z=-2.666; p<0.01), the ATHQ (Z=-2.703; p<0.01), as well as the AQ (Z=-2.094, p<0.05) (Figure 3) in Group VR. Again, the values are in close agreement with the known therapeutic power of VR exposure for the fear of heights treatment (e.g., Hodges et al., 1995; Rothbaum et al., 1995b; Emmelkamp et al., 2002).

60.0

47.7

47

50.0

33.6

40.0 26.7 30.0

pre

22.1

post

20.0 SD=12

10.0

SD=11.2

SD=9.3

5.8

SD=8.9

SD=13.3

SD=5.2

0.0

Figure 3. Median values comparing the Level of Heights Difficulty, Attitude Towards Heights Questionnaire and the Acrophobia Questionnaire, in pre- and post-test (VR Group).

3.3 Comparison of the Two Groups Before Treatment Comparing the two groups and considering the results obtained from the first assessment, there is no significant difference in AQ (U=19.0; p>0.05), ATHQ (U=19.0; p>0.05) and LHD (U=21.5; p>0.05). These results suggest that the groups were identical before treatment. 3.4 Comparison of the Differences Between the Two Groups In order to assess the differential effect of the two types of therapy, we analysed the differences between the values before and after treatment in each of the variables being studied for each therapeutic group (VR Group versus R Group). These variables were the SUDS, the LHD, the ATHQ and the AQ. The “pre–post” difference of medians was compared using the Mann-Whitney U test.

210

Virtual and Real Environments for Acrophobia Desensitisation

The VR and R groups show no pre-post treatment differences, according to the Mann-Whitney test (U=17.5; p>0.05) regarding the SUDS. Regarding the LHD differences, the groups cannot be significantly distinguished regarding pre-post differences according to the Mann-Whitney test (U=13.0; p>0.05) (Table 1). Considering the differences in the ATHQ, the VR and R groups show no pre-posttreatment differences according to the Mann-Whitney test (U=24.5; p>0.05), and the groups showed no difference regarding the AQ values (U=19.0; p>0.05).

LHD Median Amplitude

VR Group -15.5 37

R Group -5 23

Table 1. Differences between the Level of Heights Difficulty.

Comparing the two groups’ results obtained in the second assessment (post-test), no significant differences were shown in the AQ (U=17,0; p>0.05), ATHQ (U=21,5; p>0.05) and behaviour performance LHD (U=16,0; p>0.05). Overall results suggest that the groups were identical after treatment. We should stress that such similar therapeutic output was achieved despite the much shorter length of the virtual exposure compared to the real exposure (Figure 4). The average session time of the groups presented a significant difference (U= 0,000 p<0.01), being that the VR treatment took an average time of 22.3 minutes, much lower that that of the real-world treatment (51.7 minutes).

Average session time in minutes 70 60 50 40 30 SD=22.1

20 10

SD=6.5

0 1 group1 (virtual)

group2 (rea)

Figure 4. Comparing session times for Virtual and Real Treatment Groups.

211

,

C.M. Coelho, C.F. Silva, J.A. Santos, J. Tichon, G. Wallis

4. Discussion Overall, as indicated in prior research, our results suggest both the real-world and VR treatments were highly effective (e.g., Hodges et al., 1995; Rothbaum et al., 1995a; Rothbaum et al., 1995b; Emmelkamp et al., 2001; Emmelkamp et al., 2002). In the current study, VR treatment was determined to be at least as effective, if not more efficient that real-world exposure. VR offers a number of practical advantages over real-world exposure including: (i) better control of the situation by the therapist; (ii) avoidance of potential public embarrassment; (iii) maintenance of confidentiality; and (iv) comfort of the protective environment of the therapist’s office (Wiederhold & Wiederhold, 2005). In addition to these advantages our experience has been that VR provides a further advantage over traditional treatments by virtue of its ability to attract participants into treatment. The difference in numbers of participants between comparative groups in this project was due entirely to the fact that people were volunteering at twice the rate for the VR exposure group. It was much more difficult to find participants willing to volunteer for the real-world group. The VR treatment patients received less exposure time than the participants undergoing traditional treatment. Significantly, it was established that for the LHD, the VR Group presented a median of -15.5, while the R Group presented a median of -5 (Table 1), which means that the VR Group could climb more steps with less anxiety. One possible explanation for this large difference is the ability of the VR system to promote new visuo-vestibular skills in the participants (Whitney et al. 2005). Although the SUDS reported by participants throughout the sessions had decreased, indicating habituation, there was a temporary increase of the SUDS associated with approaching the balcony, physical movement, and increased height. Individual virtual environments may not produce exactly the same effects observed in this study. It is important when designing a suitable environment, that a range of realistic cues is offered, capable of making the subject feel threatened, and engendering in the subject the belief that their actions will have real consequences.

5. Limitations and Suggestions for Research

Comparison of the two treatment types suggests that there is an advantage for training in a virtual environment. It may be that if the steps used in the BAT had been

212

Virtual and Real Environments for Acrophobia Desensitisation

higher, a greater variation of results would have been obtained. Undertaking this modification in future research should achieve a change in the between and within group differences that results in statistical significance. It appears that limiting the number of steps to 40 provoked an undesirable ceiling effect. It is important to point out, therefore, that the absence of statistically significant differences between treatments might simply be due to a lack of experimental power (e.g., Agras & Jacob, 1981). Participant numbers were unevenly distributed. This difference was due to the comparative difficulty to recruit participants in enrolling the real world exposure group. Participants were informed by e-mail and local newspaper that VR was being used, free of charge, for the treatment of acrophobia. This personal preference of participants was difficult to manage. It is interesting that all study dropouts were applicants invited to participate in the real environment. While clearly resulting in a study limitation, this experience with participants also demonstrated unintentionally the potential of VR to attract people with phobias to treatment. A patient group is usually characterised by treatment avoidance (Boyd et al., 1990). Further studies are still needed to explore acrophobia. One particularly interesting future direction is a more detailed investigation of specific phobic triggers in height environments. VR systems offer exciting and novel opportunities for testing the role of these cues in a tightly controlled and independent manner (e.g., Loomis, Nlascovich, & Beal, 1999; Gaggioli, 2003). For example, head motion could be recorded during training, to provide a tighter causal link between self-motion and fear of heights.

6. Acknowledgements

This research was funded by a Bial Foundation Grant (project Bial 39/98) and by a Science and Technology Foundation Grant (ref. SFRH/BPD/26922/2006).

7. References

Abelson, J.L., & Curtis, G.C. (1989). Cardiac and neuroendocrine responses to exposure therapy in height phobics: Desyncrony within the “physiological response system”. Behaviour Research & Therapy, 27, 561-7.

213

,

C.M. Coelho, C.F. Silva, J.A. Santos, J. Tichon, G. Wallis

Agras, W.S., & Jacob, R.G. (1981). Phobia: Nature and Measurement. In Matig Mavissakalian

&

David

H.

Barlow

(Eds.),

Phobia:

Psychological

and

Pharmacological Treatment (pp.35-62). New York: The Guilford Press. Baker, B.L., Cohen, D.C., & Saunders, J.T. (1973). Self-directed desensitization for acrophobia. Behaviour Research and Therapy, 11, 79-89. Boyd, J.H., Rae, D.S., Thompson, J.W., Burns, B.J., Bourdon, K., Locke, B.Z., & Regier, D.A. (1990). Phobia: prevalence and risk factors. Social Psychiatry and Psychiatric Epidemiology, 25, 314-323. Coelho, C.S.M.; Santos, J.A.; Silvério, J.A.S; & Silva, C.F. (2006). Virtual reality and acrophobia: one year follow-up and case study. Cyberpsychology and Behavior, 9(3), 336-341. Cohen, D.C. (1997). Comparison of self-report and overt-behavioral procedures for assessing acrophobia. Behavior Therapy; 8, 17-23. Durlach, N.I., & Mavor, A.S. (1995). Virtual reality: Scientific and technological challenges. Washington, DC: National Academy Press. Emmelkamp, P.M.G., Bruynzeel, M, Drost, L., Van der Mast, C.A.P.G. (2001). Virtual Reality Treatment in Acrophobia: A Comparison with Exposure In Vivo. Cyberpsychology and Behavior; 4, 335-339. Emmelkamp, P.M.G., & Felten, M. (1985). The process of exposure in vivo: Cognitive and physiological changes during treatment of acrophobia. Behaviour Research & Therapy, 23, 219-223. Emmelkamp, P.M.G., Krijn, M., Hulsbosch, L., de Vries, S., Schuemie, M. J., van der Mast, C.AP.G. (2002). Virtual Reality Treatment versus Exposure In Vivo: A Comparative Evaluation in Acrophobia. Behaviour Research & Therapy, 40, 509516. Emura, S., & Susumu, T. (1998). Multisensor integrated prediction for virtual reality. Presence: Teleoperators & Virtual Environments, 7, 410-423. Gaggioli, A. (2003). Using Virtual Reality in Experimental Psychology. In G. Riva & C. Galimberti (Eds.). Towards CyberPsychology: Mind, Cognitions and Society in the Internet Age (pp.157-174). Amsterdam: IOS Press. Hodges, L.F., Kooper, R.; Meyer, T.C., Rothbaum, B.O., Opdyke, D., Graaff, J.J., Williford, J.S., & North, M.M. (1995). Virtual environments for treating the fear of heights. IEEE Computer, 28, 27-34.

214

Virtual and Real Environments for Acrophobia Desensitisation

Kennedy, R.S., Jones, M.B., Stanney, K.M., Ritter, A.D., & Drexler, J.M. (1996). Human factors safety testing for virtual environment mission-operation training. (Contract No. NAS9-19482, 6/96). Houston, TX: NASA Johnson Space Center. Kennedy, R.S, Lane, N.E, Berbaum, K.S, Lilienthal, M.G. (1993). Simulator sickness questionnaire: an enhanced method for quantifying simulator sickness. The International Journal of Aviation Psychology, 3, 203-20. Kennedy, R.S., Stanney, K.M., & Dunlap, W.P. (2000). Duration and exposure to virtual environments: sickness curves during and across sessions. Presence: Teleoperators and Virtual Environments, 9, 463-472. Krijn, M.; Emmelkamp, P. M. G.; Olafsson, R.P. & Biemond, R. (2004) Virtual reality exposure therapy of anxiety disorders: A review. Clinical Psychology Review, 24, 259-281. Lawson, B.D., Graeber, D.A., Mead, A.M., & Muth, E.R. (2002). Signs and symptoms of human syndromes associated with synthetic experiences. In K.M. Stanney (Ed.), Handbook of virtual environments, design, implementation, and applications, (pp.589-618). Mahwah: Lawrence Erlbaum Associates. Loomis, J.M., Nlascovich, J.J., & Beal, A.C. (1999). Immersive virtual environment technology as a basic research tool in psychology. Behavior Research Methods, Instruments & Computers, 31(4), 557-564. Marshall, W. L. (1985). The effects of variable exposure to flooding therapy. Behavior Therapy, 16, 117-135. Reason, J.T., & Brand, J.J. (1975). Motion sickness. London: Academic Press. Rothbaum, B., Hodges, L., Kooper, R., Opdyke, D., Williford, J., & North, M. (1995a). Virtual reality graded exposure in the treatment of acrophobia: a case report. Behavior Therapy, 26, 547-554. Rothbaum, B., Hodges, L., Kooper, R., Opdyke, D., Williford, J., & North, M. (1995b). Effectiveness of computer-generated (virtual reality) graded exposure in the treatment of acrophobia. American Journal of Psychiatry, 152, 626-628. Spencer, C.R., & Conrad, P.L. (1989). Treatment of acrophobia of an institutionalized adult with mental retardation. Mental Retardation, 27, 1-4. Stanney, K.M., Mourant, R.R., & Kennedy, R.S. (1998). Human factors issues in virtual environments: a review of the literature. Presence: Teleoperators and Virtual Environments, 7, 327-351.

215

,

C.M. Coelho, C.F. Silva, J.A. Santos, J. Tichon, G. Wallis

Whitney, S.L., Jacob, R.G., Sparto, P.J., Olshansky, E.F., Detweiler-Shostak, G., Brown, E.L., & Furman, J.M. (2005). Acrophobia and pathological height vertigo: indications for vestibular physical therapy? Physical Therapy, 85, 443-458. Wiederhold, B. K., & Wiederhold, M. D. (2005). Virtual reality therapy for anxiety disorders. Advances in evaluation and treatment. Washington: American Psychological Association. Williams, S.L., Dooseman, G., & Kleinfield, E. (1984). Comparative effectiveness of guided mastery and exposure treatments for intractable phobias. Journal of Consulting and Clinical Psychology, 52, 505-518. Williams, S.L., Turner, S.M., & Peer, D.F. (1985). Guided mastery and performance desensitization for severe acrophobia. Journal of Consulting and Clinical Psychology, 53, 237-247. Wolpe, J. (1982). The Practice of Behavior Therapy (3rd ed.). Oxford: Pergamon Press.

216