Sameer Ahuja Thesis

A Tale of Two Sites: An Explorative Study of the Design and Evaluation of Social Network Sites

Sameer Ahuja

Thesis submitted to the Faculty of the Virginia Polytechnic Institute and State University in partial fulfillment of the requirements for the degree of

Master of Science in Computer Science & Applications

Dr. Manuel A. P´erez-Qui˜ nones, Chair Dr. Andrea L. Kavanaugh Dr. Stephen H. Edwards

July 17, 2009 Blacksburg, Virginia

Keywords: Social Network Software, Usability Evaluation, Social Affordance, Design Copyright 2009, Sameer Ahuja

A Tale of Two Sites: An Explorative Study of the Design and Evaluation of Social Network Sites

Sameer Ahuja

(ABSTRACT)

Social Network Sites allow individuals to construct a public or semi-public profile within a bounded system, articulate a list of other users with whom they share a connection, and view and traverse their list of connections and those made by others within a system. Such sites are generally centered around a particular activity, such as maintaining social relationships or uploading user created content. Increasingly, niche domains such as education, healthcare and software development have been exploring the creation of social network sites centered around the activities of the domain. This has led to an increasing focus on the processes involved in designing and evaluating these sites. We argue that social network sites require a specialized focus in their design and evaluation on the social utility of the features on the site. We have created two social network sites for niche communities: Colloki, a conversation platform designed for members of local communities; and CATspace, a social repository of Computer Science assignments, designed for use by CS instructors and students. In this thesis, we describe the motivation, design and implementation of these two sites. We provide a formative evaluation of these two sites, wherein we evaluate the usability, and study the perceived social affordances of individual features across the two site. Finally, we discuss future work towards building a framework for evaluating the social utility of Social Network Sites at a formative stage.

biji

iii

Acknowledgments My advisor, Dr. Manuel A. P´erez-Qui˜ nones, has been a source of inspiration for me during the past two years. I thank him foremost for his wonderfully cheerful, supportive and flexible nature. I’ve been lucky to have worked at a close level with Dr. Andrea Kavanaugh and Dr. Steve Edwards in their respective areas of work. They both have a deep understanding of their research areas, and it was fascinating to learn from them for the period that we worked together. I thank both of them for their guidance, for their flexibility, and of course, for serving on my committee. Several people helped at different stages for this work to come through in the form that it has. Ricardo, Ben, Vineeta and Mathew, have all helped with the development of these prototypes at different stages. Dr. Pardha Pyla endured my old-age jokes and spent hours guiding me. Fellow colleagues - Joon Kim, Candida and Szu-Chia provided support and guidance; and Manas, my lab-mate for the better part this work, always kept things interesting at KWII 1132. Finally, I’d like to thank my parents for making the small and big sacrifices in their lives, so that I could achieve the small and big achievements of my life.

iv

Contents

1 Introduction

1

1.1

Defining Social Network Sites, Social Affordance and Social Actions . . . . .

2

1.2

Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

5

1.3

Problem Statement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

7

1.3.1

Why Evaluate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

8

Overview of the Thesis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

8

1.4

2 Related Work 2.1

2.2

10

Social Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

10

2.1.1

Tagging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

11

2.1.2

Activity Streams . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

12

2.1.3

User Profiles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

13

2.1.4

Comments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

14

2.1.5

Ratings and Votes . . . . . . . . . . . . . . . . . . . . . . . . . . . .

14

Social Actions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

15

2.2.1

16

Social Browsing and Search . . . . . . . . . . . . . . . . . . . . . . . v

2.3

2.4

2.2.2

Interaction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

16

2.2.3

Sharing Content . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

17

2.2.4

Collaboration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

17

Local Deliberation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

18

2.3.1

Low Scale . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

19

2.3.2

Automated relevance algorithms . . . . . . . . . . . . . . . . . . . . .

19

2.3.3

Citizen Participation . . . . . . . . . . . . . . . . . . . . . . . . . . .

19

2.3.4

Centralization of content . . . . . . . . . . . . . . . . . . . . . . . . .

20

Computer Science Assignment Repositories . . . . . . . . . . . . . . . . . . .

20

3 Prototypes 3.1

3.2

3.3

23

Colloki . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

24

3.1.1

Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

24

3.1.2

System Description . . . . . . . . . . . . . . . . . . . . . . . . . . . .

25

3.1.3

Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

30

CATspace . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

30

3.2.1

Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

32

3.2.2

System Description . . . . . . . . . . . . . . . . . . . . . . . . . . . .

32

3.2.3

Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

36

Social Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

42

3.3.1

Tagging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

42

3.3.2

Activity Streams . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

44

vi

3.3.3

User Profiles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

44

3.3.4

Comments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

46

3.3.5

Ratings and Votes . . . . . . . . . . . . . . . . . . . . . . . . . . . .

48

4 Methodology

49

4.1

Goals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

49

4.2

Experimental Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

50

4.2.1

Selection of Participants . . . . . . . . . . . . . . . . . . . . . . . . .

50

4.2.2

Usability Lab Setup

. . . . . . . . . . . . . . . . . . . . . . . . . . .

51

4.2.3

Evaluation Protocol . . . . . . . . . . . . . . . . . . . . . . . . . . . .

51

Benchmark Tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

52

4.3.1

Colloki . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

52

4.3.2

CATspace . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

54

Perceived Social Affordance . . . . . . . . . . . . . . . . . . . . . . . . . . .

55

4.4.1

Simulated Community . . . . . . . . . . . . . . . . . . . . . . . . . .

56

4.4.2

Mapping social features to actions . . . . . . . . . . . . . . . . . . . .

56

4.3

4.4

5 Results

58

5.1

Overall Usability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

58

5.2

Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

62

5.2.1

Tagging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

62

5.2.2

Activity Streams . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

63

vii

5.2.3

User Profiles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

65

5.2.4

Comments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

67

5.2.5

Ratings and Votes . . . . . . . . . . . . . . . . . . . . . . . . . . . .

67

6 Discussion and Conclusions 6.1

70

Outcomes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

70

6.1.1

Articulation of connections . . . . . . . . . . . . . . . . . . . . . . . .

70

6.1.2

Activity Streams . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

71

6.1.3

Task Sets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

72

6.2

Evaluation Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

72

6.3

Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

75

Bibliography

83

A IRB Approvals

84

B Informed Consent Forms

87

C Pre-Evaluation Survey

94

D Colloki Tasks

98

E CATspace Tasks

101

F Colloki Post-Evaluation Questionnaire

104

G CATspace Post-Evaluation Questionnaire

112

viii

List of Figures 3.1

A sample topic page in Colloki . . . . . . . . . . . . . . . . . . . . . . . . . .

26

3.2

Colloki Data Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

31

3.3

CATspace Homepage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

33

3.4

Representation of information that the different CATspace views display. . .

34

3.5

Object model and basic design of CATspace . . . . . . . . . . . . . . . . . .

37

3.6

CATspace data model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

39

3.7

Interaction between CATspace and Facebook . . . . . . . . . . . . . . . . . .

40

3.8

A user profile on Colloki . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

45

3.9

A user profile on CATspace . . . . . . . . . . . . . . . . . . . . . . . . . . .

47

5.1

Participant responses for questions on overall usability . . . . . . . . . . . .

59

5.2

Participant responses for questions on tagging . . . . . . . . . . . . . . . . .

61

5.3

Participant responses for questions on activity streams . . . . . . . . . . . .

64

5.4

Participant responses for questions on user profiles . . . . . . . . . . . . . . .

66

5.5

Participant responses for questions on comments . . . . . . . . . . . . . . . .

68

5.6

Participant responses for questions on votes/ratings . . . . . . . . . . . . . .

69

ix

List of Tables 1.1

Selected definitions of social software / social computing . . . . . . . . . . .

3

4.1

Expected mapping of Social Features to Social Actions . . . . . . . . . . . .

55

x

Chapter 1 Introduction Mainstream Social Network Sites (SNS) have seen exponential growth in recent times. These sites have become a cultural phenomenon, having managed to attain massive number of participants and high engagement levels. Facebook is the prime example, and according to their statistics, more than 3.5 billion minutes1 are spent on the site each day. Teenagers (Age 12-17) and Gen Y (Age 18-32) are the most engaged demographics for SNS, with the most recent PEW study indicating that 65% of online Teenagers and 67% of online Gen-Yers use some form of Social Network Sites [39]. Seeing the growth of SNS, researchers in several domains are starting to explore the implications and possibilities of social network software in their respective domains. Researchers see promise in the software’s ability to engage users and build social connections. Several of these domains are ones that deal with building software for communities of people. Examples are software development [19], Healthcare [8], Education [58], Local Conversations [5], Knowledge Communities [26] and so on. Complementary to the research movement, startups and small companies are continuously developing and deploying beta or even alpha stage websites that are bringing social networking concepts to new domains, such as travel2 , 1 2

Facebook Press Release, http://www.facebook.com/press/info.php?statistics Dopplr, http://dopplr.com

1

Sameer Ahuja

Introduction

2

music3 and even research activities4 . Some of these softwares find immediate adoption and a viral growth from the start, while others are relegated to ‘dead pools’5 , contributing as prototypes to the global-scale social networking experiment.

1.1

Defining Social Network Sites, Social Affordance and Social Actions

Before we consider the definition of social network sites, it is important to provide the definition of social software that we base our context on, and the relationship that exists between social software and social network sites. The definition of social software varies across literature from different schools of thought and different research areas. These definitions vary in the level of abstraction and coverage. Some selected definitions of social software in literature are provided in Table 1.1. Fortunately, social network sites satisfy roughly all of these definitions of social software, and most schools of thought accept them as a kind of social software. boyd and Ellison narrow down the scope of SNS by providing a definition that separates them from other forms of social software [11]: We define social network sites as web-based services that allow individuals to (1) construct a public or semi-public profile within a bounded system, (2) articulate a list of other users with whom they share a connection, and (3) view and traverse their list of connections and those made by others within a system. The requirement for a social network site to be a web-based service appears to be a rather restricting technical constraint imposed by this definition, and it might be a consequence of Last.fm, http://last.fm Mendeley, http://www.mendeley.com; and CiteULike, http://www.citeulike.org 5 TechCrunch deadpool, http://www.techcrunch.com/tag/deadpool 3 4

Sameer Ahuja

Introduction

Table 1.1: Selected definitions of social software / social computing Source

Definition

Schuler, ’94 [59]

(Social Computing) Describing any type of computing application in which software serves as an intermediary or a focus for a social relation.

Wang et. al, ’07 [70]

Computational facilitation of social studies and human social dynamics as well as the design and use of ICT technologies that consider social context.

Forrester Research, ’06 [14]

A social structure in which technology puts power in individuals and communities, not institutions.

Tepper, ’03 [67]

Social Software refers to various loosely connected types of applications that allow individuals to communicate with one another, and to track discussions across the web as they happen.

Wikipedia6

In its most common usage, social software means a range of software programs which are often web-based, and which allow users to interact and share data with other users. However, it may also refer to the procedures of society carried out by people since before the invention of computers.

3

Sameer Ahuja

Introduction

4

the fact that the web is the most ubiquitous technology platform in the world with most social networking applications being written for it at the time of this writing. A social networking site can exist over any technological platform that provides convenient access to the intended users, it does not necessarily need to be a web based application7 . There are other aspects to the definition that are a subject of debate in the research community. David Beer [7] argues that by categorizing a broad range of sites as ‘social network sites’, the boyd-Ellison definition over-generalizes the classification and merges different kinds of sites together: those that have social networking as the central activity (‘social networking sites’), and those that center around some other activity (Such as video publishing in YouTube). While the definition is a work under construction, it is representative of our state of understanding of social network sites. Hence, for the rest of this writing, we will be using the boyd-Ellison definition of social network sites. Researchers have studied the social affordances of internet and online systems in the past [73]. Social affordance, itself, has been defined as the quality of artifacts in any space physical or virtual, which invite and facilitate social interaction among the participants in that space [36]. This definition can be limiting in the context of social network sites, as social interactions are but one kind of social actions. Rating a piece of content on a site, for example, is not an interaction between two users, yet, it can be construed to be a social action. Similarly, customizing one’s profile on a social network site is an individual activity, yet it can be driven by social incentives and lead to social consequences [10]. So for the purpose of this work, we broaden the definition, to define social affordance as: The quality of an artifact in any space which invites and facilitates social actions by the participants in that space. To define social actions, let’s consider the concept of social capital. Social capital has been given several definitions in several areas of research [4], and the research community is 7

An example of the same is location-based social networking applications such as Loopt (http://loopt.com)

that provide their interface via native applications on mobile devices.

Sameer Ahuja

Introduction

5

torn on whether it is a fashionable (and short-lived) term proposed as a cure-all for the maladies affecting contemporary communities, organizations and societies as a whole or whether it has more long-term strategic - theoretical as well as applicable - meaning for sociology and other social-science disciplines [3]. Sidestepping the arguments, let’s consider the definition provided by James Coleman [57]. Coleman’s definition suggests that social capital is anything that facilitates individual or collective action, generated by networks of relationships, reciprocity, trust, and social norms. Based on this definition, we can define social actions in the context of social network sites as: Actions that affect the social capital of the community using the site, and this would encompass the above-mentioned individual actions with social repercussions.

1.2

Motivation

We look at social network sites from the practitioner’s perspective. The overarching problem is ad-hoc implementation of social networking concepts. It is best defined by the questions that we do not yet have answers to: How to design the software for social network sites? How to do the requirements analysis? What are the design patterns to utilize? How to evaluate the user interface for social network sites? Without an understanding of these high-level concepts and a rigorous evaluation methodology, most decisions taken in the design and implementation of software for social network sites are determined by the ‘gut feeling’ of the practitioners. Social networking software has an inherent complexity that makes the design and implementation of social network sites a unique undertaking that might not be best served by traditional software engineering approaches. This complexity is sociological effects. The success and growth of such software stems from the continuous tango between technological advancement and incremental user participation, and the net product is neither cleanly

Sameer Ahuja

Introduction

6

social nor technological in nature [9]. Sometimes, the way that the user community utilizes the software can lead to advancements in the technology. An example of the same is the ‘@’ reply feature of Twitter8 . This feature lets users address other users of the system in their updates by prefixing their usernames with the ‘@’ symbol. This syntax came into being when the developers of the application realized that several users were having directed public conversations by following the @username convention to address other users in the system. The developers decided to enhance the application to better support this community-generated convention. On other occasions, technology can affect the social dynamics of the community, such as in the case of Facebook introducing the ‘News Feed’ feature. The feature allows people using the site to view a feed of the activity that people in their social network have performed on the site in the recent past. This feature met with mass resistance when it was introduced in 2007, but consequently, it was central in driving certain kinds of social interactions amongst users [12]. It is worthwhile to note that this tango between technological innovation and sociological effects is not always perfect. Technology can fail to advance with the scale and requirements of user community, as evidenced by the recent pulling of some Twitter features that the application’s architecture was not being able to support9 . Complementarily, the user community can react negatively to technological innovation, as experienced by Facebook when it rolled out the Beacon advertising platform in 200710 . 8

Twitter (http://twitter.com), is a microblogging service that lets people send and receive brief text

updates from a variety of devices and interfaces 9 Twitter had to remove an option that let users view their contacts’ replies to other people (http://www.wired.com/epicenter/2009/05/twitter-screw-up/) 10 Facebook Beacon generated so much controversy amongst users that all major advertisers decided to leave the platform (http://www.techcrunch.com/2007/12/03/more-facebook-advertisers-bail-from-beaconplus-new-concerns/)

Sameer Ahuja

1.3

Introduction

7

Problem Statement

Social Network software, then, is defined not only by its feature characteristics, but also by how the social context of the community utilizing the software affects its utilization [48]. This realization leads to a rethinking of the conventional answers to the problems specified above. Of special concern are the activities performed in software engineering before the software is first released to the community. For example, how to design the application architecture to support future features that cannot even be envisioned at the time of design? Or, how to perform formative evaluations of a social network site, at a stage where it has not been released and the user community has not started participating, and get a sense of how successful it would be when released for use by that community? In this thesis we specifically look at the problem of formative evaluation of social network sites. As part of this body of work, we design and develop two niche community social network sites, Colloki and CATspace; and then perform a formative evaluation of the two softwares with respect to their usability, and their social affordances. The research contributions of this piece of work are to provide: 1. Foundational work for a framework for formative evaluation of perceived social affordances of social network sites, provided via the case studies of Colloki and CATspace. 2. A characterization of SNS features and social actions, and a review of the current body of work on them. 3. Formative evaluation of the usability and perceived social affordance of Colloki and CATspace. This work is intended to be one the first in a series of efforts to formalize processes for the evaluation for social network sites, and towards the formalization of established user interface patterns for social network sites.

Sameer Ahuja

1.3.1

Introduction

8

Why Evaluate

But why do this evaluation at all? Or, in other words, why concern ourselves with trying to evaluate the social utility of the site at a formative stage, when clearly it is not possible to get a clear measurement of the utility of social features before the sociological effects associated with the community using the tool start playing their role. Formative evaluations of social network sites would help reduce the risk of being able to evaluate the social utility of the site only when the software has been released to the community and a substantial amount of activity has been performed on the site. In fact, fundamental issues with the features of the site might not even let it reach a state where the social capital reaches a stage that it can start bringing utility to the members of the community. Even when one considers the agile engineering approach of ‘release soon, release often’ [16], in the case of social network software being released to a community of users, it is risky to release the software too soon - depending on the context of the community; if the tool does not have the appropriate privacy and security tools, or the social features are not engaging enough; the users might go away disappointed with the software never to return. We theorize that formative evaluation of social network sites will help find fundamental problems with the way social features of the site are implemented. This also has a consequent benefit for releasing changes to an in-production site - such changes can be evaluated using a similar framework to formative evaluations so as to get a sense of the user response to the features to prevent situations where new features are met with strongly negative reactions [65].

1.4

Overview of the Thesis

This thesis is organized as follows. Chapter 2 reviews related work that has studied social affordance, social features of social network sites and social actions taken by users of SNS. The

Sameer Ahuja

Introduction

9

chapter also explores the past work on the two niche areas that the prototypes being studied are involved with; namely, local deliberation and Computer Science assignment repositories. Chapter 3 presents an overview of the two prototypes, providing detailed description of the structure and implementation of the two systems, along with a comparison of the manner in which the two systems utilize social features. Chapter 4 presents the methodology used for the formative evaluation of usability and perceived social affordance of the two prototypes. Chapter 5 presents the results of the study. Chapter 6 provides a discussion on the results of the study, the questions that arise from this work, and possibilities for future work.

Chapter 2 Related Work In this chapter, we discuss past work done in the area of characterizing social features and understanding the user incentives around those features. Complementarily, we discuss social mechanisms and actions that users of social network sites perform using the site’s features, and their effects on the system. We then discuss the two niche domains of local conversations and Computer Science (CS) assignment repositories, and provide examples and justifications wherever possible from past work of the utilization of social network software in the particular domains.

2.1

Social Features

Past work has looked at different aspects or features of social network sites. Different aspects of SNS have received varying amounts of focus by the research community. For these features, researchers have performed characterization of functional aspects, characterization of user incentives in play, metric studies of the usage of features on mainstream SNS websites, and so on. In this section we detail some of the significant work done in this direction. This list of social features is by no means complete, it is a set of the most relevant features in

10

Sameer Ahuja

Related Work

11

the context of this thesis, and is an intersection of the most popular social features and the social features that we have chosen to implement and study in our prototypes.

2.1.1

Tagging

Tagging is one of the features that has received the most attention from the research community, and its the only social feature that has a well defined framework for characterization. As defined by [48], web-based tagging systems let users annotate a particular resource, such as a web page, a blog post, an image, a physical location, or just about any imaginable object with a freely chosen set of keywords (“tag”). We are just beginning to understand how tagging works [29], its implications, and its possible uses [48]. Tags are even making their way to personal information management tools, such as Google’s Mail program. Marlow and colleagues [48] have studied tagging systems and articulated a framework for studying tagging systems. They provide two organizational taxonomies to characterize social network systems with support for tagging (they call them social tagging systems): 1. The user incentives at play: Future retrieval of tagged items, contribution and sharing, attracting attention, play and competition, self presentation, opinion expression. 2. The system design and attributes: Tagging rights, tagging support, aggregation of tags, type of object being tagged. The two taxonomies are complementary, because one of them represents the original intent of the designer of the system (System design and attributes) and can be determined at design time, while the other characterization (User incentives) is such that it can only be determined when the software is used by the community. User incentives are affected by personal needs and social interests. While the design of the system can play a significant role in forming user incentives, they are also affected by the social norms within the community using the tool, which is a function of the characteristics of the individuals involved.

Sameer Ahuja

2.1.2

Related Work

12

Activity Streams

Activity streams are flowing commentaries on users’ actions on the different sections of the site. They are also known as ‘news feeds’ or ‘lifestreams’, and were first made popular amongst social sites by Facebook in 2007. Most systems utilizing activity streams show the subset of the stream that is most relevant to the user’s current view, such as a user’s activity on his/her profile. Further, different implementations of activity streams can give preference to a user’s close friends’ on the site in the stream that he/she sees. Some systems let users delete items off their activity stream to hide some of their actions, while in other systems the user can only get rid of an activity stream item by undo-ing the action that caused it. Complementarily, systems can also provide filtering and grouping options for viewers to group or restrict some kinds of items or items from some people from showing up in their feed. Some systems also perform analysis on activity streams to present them in more useful ways. Friendfeed1 , for example, restricts similar activities by a particular user that are visible in the stream. The remaining activities are provided below the fold by clicking a link. This prevents one user from dominating the activity stream of events. Activity streams are an effective way of helping users discover content. In addition, they have been found to motivate contribution by new users to the system. Burke et. al [12] recently conducted a study using the logs of 140,000 new users on facebook to determine that activity streams (Called ‘News Feed’ in Facebook terminology) can be vital for motivating new users to contribute to the site. They found four social mechanisms at play that encourage new users to contribute: 1. Social Learning: Newcomers whose friends share more content on the system go on to contribute more content themselves. 2. Singling out: Newcomers who are singled out in content go on to contribute more content. 1

Friendfeed, http://friendfeed.com

Sameer Ahuja

Related Work

13

3. Feedback: Newcomers receiving more feedback on their initial content go on to contribute more content. 4. Distribution: Newcomers whose initial content is distributed widely go on to contribute more content. Streams are the central vehicle for the above-mentioned social mechanisms in social network sites. A complementary approach to explicit activity streams worth mentioning in this context is the one taken by Amazon.com. Amazon records users’ activity streams, but instead of letting other users view those streams, it instead uses algorithms to analyze those streams to provide a different construct to its users - ‘Social navigation’ [21]. When users perform actions on the site, the system provides recommendations for next actions based on its analysis of past actions of users. These are displayed on the site in the form of recommendations like “People who bought this book also bought...”.

2.1.3

User Profiles

User profiles are central to the concept of a Social Networking Site, as per the boyd-Ellison definition [11]. Profiles provide users their identity on the system, and aid in discovery of common interests between people and in articulating relationships. There is a strong identity management component attached to user profiles, as detailed by danah boyd in her ethnographic study of MySpace [10]. User profiles vary in the following three aspects: 1. Customizability of presentation: The customizability of MySpace pages via CSS, images, videos and audios is a central aspect to its success with the youth demographic [11]. Facebook, on the other hand, has strict controls in place over the display and layout of information on the profile.

Sameer Ahuja

Related Work

14

2. Information: Most SNS let users manipulate what information is provided on their profiles in the system. Some systems let users import information from other social network sites or websites into their profiles either as ‘widgets’2 , or as items of activities in the users’ activity streams3 . 3. Articulated relationships: Different systems articulate relationships on user profiles in different ways. Some systems such as facebook and MySpace have explicit two-way relationships (‘friendships’); others like twitter support one-way explicit relationships (‘followers’); while some systems articulate relationships between users via the content that they contribute to the system, hence highlighting other users that have liked the same content as the current user, or other users whose content the user subscribes to.

2.1.4

Comments

Most social network sites let users comment on content shared on the site. Some sites also provide the ability to comment on user profiles. Comments are the primary conversational medium on social network sites, and the comments of users on such systems can often express social relationships [6].

2.1.5

Ratings and Votes

Several social applications utilize ratings and votes as an organizational tool. Amongst the mainstream websites, digg.com places user votes at the center of the system, using them as a social information filtering system [46, 47]. Other systems use ratings for personal and social recommendations [30]. Ratings are also a fundamental component of reputation systems 2

Developer

platforms

for

both

MySpace

(http://developer.myspace.com)

and

Facebook

(http://developers.facebook.com) provide for development of widgets that can be displayed on user profiles 3 Friendfeed (http://friendfeed.com) is a service that lets users import their activities on other websites into their activity stream on friendfeed

Sameer Ahuja

Related Work

15

for users of social applications. These reputation systems hold immense utility in certain areas and they have been found to facilitate social interaction, drive trust and limit aversive behavior [37]. Ratings can also be used as input for other features on the site - such as featured content, activity streams, leader-boards, and so on. The semantics and presentation of voting/rating systems differ across sites. The manner in which these systems are implemented and presented can have a significant effect on user behavior. Studies of systems using ratings to derive recommendations for users have found that such recommendations can have a cyclic effect on the ratings the users give to content on the site [18] - that is, ratings help provide good recommendations, which encourage further rating. Rating systems can provide different levels of granularity - ranging from a binary “I like this” on facebook.com to 10 point ratings on imdb.com. Some systems also provide for negative ratings, which are different from flagging - negative ratings do not indicate content that is not acceptable in the system, but rather content that the users dislike. Systems that support negative ratings can be more susceptible to unfair use [20].

2.2

Social Actions

We’ve defined social actions in Section 1.1 as actions that affect the social capital of the community using the site. That is a broad definition and there is indeed a broad range of actions that qualify for the definition at varying levels of abstraction. Forming a taxonomy of social actions on social network sites is beyond the scope of this thesis. In this section, we discuss past work on a set of social actions that we utilize as a barebones framework for our evaluation of social features. Ellison et al. [25] suggest that these social activities on social network sites provide users with social capital. While in our definition and discussion we are looking at social actions from the perspective of helping build social capital, there are other incentives and motivations for users to perform these actions. Stafford et al. [62] provide a detailed study of “uses

Sameer Ahuja

Related Work

16

and gratifications” of Internet use, “uses and gratifications” referring to the motivations of specific uses and the satisfaction people gain from such use. The third form of gratification from Internet that [62] mention is as a social environment. This is especially relevant in social network sites, where the social environment aspect is exemplified in addition to the other gratifications.

2.2.1

Social Browsing and Search

Lampe et al. [43] discuss social browsing and searching in the context of Facebook, but the concepts are generalizable to the broader range of social network sites. Social browsing refers to the use of the site for the act of finding people who are not in the user’s social network. Social search refers to the use of social network site to get more information about people in the user’s (offline) social network. Different social network environments lend differently to the above actions - in the case of Facebook, researchers have found users to be more inclined towards the action of social search than social browsing [38].

2.2.2

Interaction

We include any kind of direct interaction between two or more individuals via the medium of the social network site in our definition of interaction. Social Network sites can support interaction in different forms, however, the primary means of interaction is generally comments and posts. Other forms include giving virtual gifts (facebook.com), and ‘nudging’ (twitter.com). These interactions have varying levels of ‘interestingness’, as determined by the three variables of participation in related themes, participant cohesiveness and theme diffusion [15]. They affect the social capital of the community in perhaps the most direct manner of all social actions - interactions are used to strengthen or maintain current relationships or forge new ones. A study of Facebook usage by Lampe et al. [43] found ‘keeping up with people’ to be the primary motivation of Facebook use for college students, and that

Sameer Ahuja

Related Work

17

interaction is one of the primary means for doing so. A study by Golder et al. [32] reveals how different forms of interaction are utilized for performing different functions on the social capital by users of a social network site - they report that while the vast majority of messages are sent to friends (90.6%), a large proportion (41.6%) is sent to friends outside of one’s local network - indicating that messaging was primarily being used for building new connections. In comparison, ‘pokes’ (a form of content-free interaction) were primarily exchanged within a network (98.3% of all pokes were within a network) to maintain or strengthen relationships.

2.2.3

Sharing Content

Several social network sites are centered around the activity of users sharing content YouTube and Flickr being the most popular mainstream examples of the same. Even on sites where sharing content is not the primary activity - such as Facebook or Twitter, content sharing is one of the most oft-performed actions. Extensive work has been done in the past to characterize the motivations for sharing content [64, 50, 71, 35]. Stoeckl et al. [64] characterize the motivations into the following categories: Enjoyment, Information dissemination, Personal documentation, Passing time, External economic incentives, and contact. As with other forms of social actions, sharing content affects the social capital in varying amounts and via different means. Users of social network sites use the actions of uploading content, and controlling its access, to actively maintain and manipulate their social networks [44].

2.2.4

Collaboration

The concept of working together towards a common goal, or Collaboration, has a cyclic cause-effect relationship with social capital - both are beneficial to each other [17]. An intriguing mainstream example of collaboration in online systems is provided by an immensely successful system that does not precisely fall into our definition of social network sites Wikipedia. Wikipedia is a the Web-based user-created encyclopedia that is a prominent

Sameer Ahuja

Related Work

18

example of a collaborative, user-generated content outlet [1]. Several forms of motivations play a role in users collaborating to build the site’s knowledge base. One study claims the most significant motivations for collaborating on Wikipedia content to be Fun, Ideology, Values, Understanding, Enhancement, Protective, Career and Social [52]. Collaboration also occurs in several manners on Wikipedia. A central issue of Wikipedia is conflicts that occur amongst contributors. While these are generally looked at in a negative light, the resolution of these conflicts becomes a central activity on the site that can lead to positive effects of establishing consensus and strengthening common values amongst the participants [41], hence affecting the social capital of the community in a direct manner.

2.3

Local Deliberation

Local action groups are crucial to create awareness and draw average citizens into dialogue about local issues and concerns in democratic life [40]. These groups are centerpieces of local democratic activity and act as an intermediary level of organization between the individual and the government [69]. Most voluntary organizations face the typical challenges of leadership burnout and limited resources [40]. There is growing evidence that information and communication technology aids in resolution of these problems and increases participation among the members of these voluntary organizations [40]. Often finding and participating in local conversations is hard to accomplish online. This is for several reasons, the most important of them being: 1. There are fewer resources devoted by the news agencies to local issues. 2. Online deliberation at the local level often times occurs in particular groups and it is difficult for online citizens to join and become active participants. 3. Social software systems that allow users to gather online and discuss their interests (e.g., Digg, Slashdot, and other similar sites) work in part due to the sociological

Sameer Ahuja

Related Work

19

effects of a large number of people participating and are not as effective when the social network is small.

2.3.1

Low Scale

While mainstream media and conversations typically involve hundreds if not thousands of participants, the most active local conversations involve at the most tens of people. With reference to the social effects mentioned in Section Section 1.2, the lower scale places more emphasis on the technology to leverage the limited sociological effects of people using the technology.

2.3.2

Automated relevance algorithms

Local news and events have limited visibility in mainstream media, and any such existence has limited references to it from local or mainstream media. Current state of the art algorithms for search and discovery depend heavily on reverse-link counts [54]. This places locally relevant items at a lower priority for such algorithms.

2.3.3

Citizen Participation

Past work [61, 40] talks of the three user classes of leaders, active citizens and lurkers. Theories on online deliberation have proposed the 1% rule: That only 1% of all users of an online system contribute content to the system regularly, around 9% contribute content intermittently, and 90% of the users do not contribute content at all4 .

To engage all kinds of users, any system that is based on user generated content should 4

Jakob

Nielsen,

Participation

Inequality:

Encouraging

(http://www.useit.com/alertbox/participation inequality.html)

More

Users

to

Contribute

Sameer Ahuja

Related Work

20

provide appropriate features and means for all of the above mentioned user-classes to perform their typical tasks. This means providing affordances for the social actions that facilitate both lurking and participation, and providing incentives for participation, such as leaderboards [33].

2.3.4

Centralization of content

A typical characteristic of local conversations is that they are distributed in nature and often occur serendipitously [66]. There are several dimensions to why people prefer to post their opinions at some places and not at others. People might use a medium because they are familiar to it (such as forums), or because of features like private access (listservs), or just because they feel that the medium is relevant for them to tell their story (Such as blogs for personal opinions [51]). That is not to say that there isn’t work being done to build centralized systems for collaborative deliberation [42]. However, we envision an ideal local conversation system to be one that does not force people to one centralized conversation location, but rather provides support for authors to selectively syndicate their opinions and conversations onto the central system, and to consume content from the system in a similar manner. It is with this understanding that we first built VizBlog [66], a visualization tool that picked up conversations occurring in the local blogs of a region. Colloki is an extension of the work we did with VizBlog, that has the ambitious long-term goal of providing a local conversation platform which supports syndication from other conversational mediums.

2.4

Computer Science Assignment Repositories

Computer Science programs have seen a growing trend of online tools for managing course related activities. Such tools include, but are not limited to online automated-grading systems[24] and online repositories for course syllabi [68]. However, online repositories for sharing programming assignments among instructors has not seen such progress. The most

Sameer Ahuja

Related Work

21

notable examples of such efforts are the Nifty Assignments Repository at Stanford University [2] and the Multimedia Educational Resource for Learning and Online Teaching (MERLOT) [72]. The Nifty Assignments repository has grown as an effort of professors from different universities that come together as part of a committee during the annual SIGCSE conferences (e.g. [55, 56, 2]). Although instructors may expect high quality programming assignments from this source, they suffer from a relatively slow refinement and improvement process propitiated by the lack of involvement of instructors other than the authors of the assignments. Also, the static nature of the list of assignments inhibits the CS education community to track the use of those assignments and the modifications and improvements made through time. MERLOT attempts to improve some of the limitations of repositories like the Nifty Assignments. MERLOT provides an online space where instructors may post their assignments where they peer-reviewed by the community that they support [60, 34]. Despite MERLOT’s moderate success, there is a need for online tools that adhere to the particular needs of programming assignments [23, 53]. Overall, while teachers struggle to find quality programming assignments online [13], none of the current NSDL collections help. This has led to a situation where most instructors do not look for new repositories actively, and find the current ones to be of limited use [49]. There are a couple of problems with existing systems that has led to this situation - One, the basic ‘library model’ that most of the current repositories are based on, does not provide any greater utility than a simple web search for the assignment. Secondly, there are little to no incentives for users to actively engage in building the repositories. Hence, most repositories fail to engage the majority of instructors. There is scope for an assignment repository to be much more than a digital library. Content brings with it conversations, collaboration and other social activities centered around the content; and the repository could provide the social constructs for performing those

Sameer Ahuja

Related Work

22

actions within the site. A community-based repositories might provide the opportunity for instructors to connect at a personal level to advance their careers and promote their views on CS education [22]. The online system could also foster the evolution of programming assignments towards focusing on more culturally-diverse and socially-relevant problems [45].

Chapter 3 Prototypes This section details the two prototype applications that were evaluated in the study, namely Colloki and CATspace. The two systems are being developed for two separate niche communities - Colloki is an open-source conversation tool meant for local communities, while CATspace is a repository for Computer Science assignments, developed primarily for the Computer Science instructor community. The two applications, however, have a couple of similarities that make them worthwhile to study together for this particular study - One, both the softwares are unique in their particular domains as they bring social networking concepts to their respective domains. They use similar features for doing this, however, these features are customized to suit their particular domains. Secondly, they are both in the same stage of development. Both the applications are in their first development iteration, and have a basic feature-set that is complete in terms of essential functionality.

23

Sameer Ahuja

3.1

Prototypes

24

Colloki

Local conversations online are an important means of building awareness and increasing participation amongst citizens. However, most local websites are restricted to using forums or collaborative blogs as the medium of conversation. Colloki is a local conversation system that aims to utilize a set of social and organizational features for a more effective conversational environment. In this section we provide a detailed description of Colloki’s system attributes and implementation.

3.1.1

Introduction

In Section 2.3, we discuss the importance of local conversation tools for local action groups. Now while while the mainstream web has seen explosive growth of social software systems in the past few years, local online deliberation systems are still using the traditional discussion forums and email listservs. We believe that online deliberation systems for small groups have very unique design challenges that separate them from mainstream systems, and hence mainstream social software systems do not translate very well to the local environment. Colloki is an online “local conversation hub” that we are developing in close collaboration with several civic organizations in Blacksburg, including Citizens First1 , a grassroots organization; and Blacksburg Electronic Village2 , which represents the web presence of many local community groups. This design aims to utilize social features in a local environment to provide what we believe is a more effective local conversation medium. 1 2

http://citizensfirstforblacksburg.org http://www.bev.net

Sameer Ahuja

3.1.2

Prototypes

25

System Description

Content Organization: Topics, Tags and Ratings Colloki lets users explicitly manage their online content for organizational and social purposes. Personal accounts on Colloki support the creation of a social network, currently by providing means for informal connections formed by commenting or rating others’ content, and in the future by having people identify their friends online. These articulated relationships are the centerpieces of social network software [11] and are instrumental in building the social capital [25] . We will use the social connectivity provided by such relationships to allow events and information to be shared by others. All of the content within the system is organized in sections of interest called Topics that are aggregation of particular subsets of information. Topics are usually a small number of significant issues that a local community is facing. Browsing one of these topics is like browsing a subsection of the newspaper (e.g. the sports section, or the stock market section). Each section will have different type of content depending on how it is defined and used by the citizens participating in it. Topics like Upcoming Town Elections, Revisions to Comprehensive Plan, Downtown Revitalization are examples of possible local issue labels to organize content. In our research we have found that local civic organizations tend to focus on a particular small set of issues. We have organized Colloki in an analogous fashion, so that local citizens can quickly identify the issues at hand. Community leaders have a significant role in defining initial sections. Citizens can suggest new topics, and if enough other citizens vote for the creation of that new topic, the new topic is created. Figure 3.1 shows a topic page in a test deployment of Colloki that some members of the Computer Science department in Virginia Tech have been using for conversations for a few weeks as of this writing. Tagging is another means of organizing content on the site that is orthogonal to the topic approach. A tag is a non-hierarchical term that reflects some aspect of a content item.

Sameer Ahuja

Prototypes

Figure 3.1: A sample topic page in Colloki

26

Sameer Ahuja

Prototypes

27

It is one of the most intriguing features of social software, and we are just beginning to understand the utility [48] and implications [28] of tagging. Users of Colloki can browse information within a topic via tags. The tag pages for a topic also provides content items from other topics that have the same tag assigned to them. Further, users can browse through the profiles of other users on the system based on the tags that those users have assigned to their content, hence aiding people in discovering content and people based on their interests. Finally, if a user likes a particular content item, she can provide her approval in the system via a thumbs up. Colloki then uses these ratings to decide how to sort items on the topic homepage. For example, if an item is very popular, it does not go off the first page of the topic, even when newer items have come up. We also provide interface controls to filter and sort the list based on different parameters. The system does not support negative ratings. However, a user can flag a post if she feels that the content is inappropriate for the site. Administrators can later un-flag or delete the post, as appropriate. To counter the lack of user recommended content and for the purpose of aggregating relevant content on the site, in the future we plan to build a bot user’ - A computer program that aggregates several sources of information - including blog search results for topic-specific terms and aggregated feeds from trusted local news websites and local blogs; filters them based on the set of keywords that the site administrator provides to the system; and then posts the most appropriate stories on the site. If the human users on the site vote on the story, it stays in the system and is cached and later archived. Otherwise, the system flushes these automatic posts on a regular basis. This helps keep the system updated with the latest relevant news items from around the web, and provides the users a constant flow of stories. Colloki provides a flexible browsing mechanism of stories and content so that people can easily navigate among the information that is being collected in the site.

Sameer Ahuja

Prototypes

28

User Contributed Content: Citizen Opinions Colloki lets users enter their opinions in rich text in the web based interface. In the future, it will have multiple ways for users to express their opinions. For example, opinions can be typed text, video postings, or even audio postings. We will support doing so from mobile devices, so if citizens want to stand at a particular location in town and file a story with images from the location, they have the freedom to do so. Further, we plan to provide means for bringing in opinions from other sources, blogs and email listservs. We understand that such kinds of automated import features would require selective import features - so the features would allow authors to bring in blog posts or listserv emails to Colloki only if the blog post contains certain keywords. Such content can also be auto-classified via the specification of topic in the body of the blog post or the email. Opinions are organized in sections of interest called topics’ (Figure 3.1) that are an aggregation of a particular subset of information. Topics are usually a small number of significant issues that a local community is facing. Topics like Upcoming Town Elections, Revisions to Comprehensive Plan, Downtown Revitalization are examples of possible local issue labels to organize content. Community leaders have a significant role in defining these sections. In our research we have found that local civic organization tend to focus on a particular small set of issues. They often have a community leader in charge of the issue. We are organizing Colloki in an analogous fashion, so that local citizens can quickly identify the issues at hand. Browsing one of these topics is like browsing a subsection of the newspaper (e.g. the sports section, or the stock market section). Each section will have different type of content depending on how it is defined and used by the citizens participating in it. Some might have more links, others might have more opinions, etc. With collaboration from the local town, town officials could use appropriate tags for communications and Town Council agendas so that information is automatically classified into the appropriate section of Colloki.

Sameer Ahuja

Prototypes

29

User Recommended Content: Links Colloki will provide users with easy ways to share content from the web, in the form of links. Links are a flexible form of user recommended content (Figure 3.1). In spirit, they are similar to the bookmarks on del.icio.us, the Posted items’ feature of facebook.com and the news stories posted on digg.com. A link can be a news story, an image, a video, or any other form of web content that a user recommends to other users of colloki. This can be done by submitting it on the site; or more conveniently, by using a bookmarklet or a browser extension. Other users can then vote’ positively or negatively on the items. Hence, organization is brought to the section using the same sum of the parts effect that systems like digg.com utilize, but the initial posting of the story is done by community leaders.

User Contributed Content: Local Deliberation Beyond the top level organization of opinions and stories around ‘Topics’, all online participants are allowed to comment on each other’s contributions. We will allow any content item to have a series of comments or posts following it, in a manner similar to blog comments. This supports discussion and deliberation by citizens as a response to the postings of community leaders. We will develop an easy way for cross referencing, to support people posting comments that easily link to other stories/comments within the site.

Promoting real and virtual world conversations: Activity streams Activity streams (Latest Activity’ in Figure 3.1) are flowing commentaries on users’ actions on the different sections of the site. They were first made popular amongst social sites by Facebook in 2007. Activity streams can be customized to give preference to a user’s close friends’ on the site in the stream that he/she sees. They are an effective way of helping users discover content that they might be interested in. Colloki provides an activity stream of the site’s activity at a system level, and topic-level streams on the topic homepages.

Sameer Ahuja

Prototypes

30

Content Syndication: Feeding the Web 2.0 User’s content can be syndicated out to other services (e.g., their photos of Montgomery County or their comments on revisions to the Town Plan). The goal is to provide as many ways to get information out to people as possible. We allow information in Colloki to be reused by other systems, thus giving way to a wider network of information about local issues. We provide RSS feeds by sections, recent stories, particular users, tags, and any other organization scheme in the system. This allows users to be alerted when new information is available in whichever way they find most appropriate. In the future, we will also provide options to send updates via emails, SMS, Twitter, and subscription to calendar feeds using popular applications like Google Calendar, Apple’s iCal and Microsoft Exchange.

3.1.3

Implementation

Colloki is based on the Ruby On Rails framework. Its architecture, hence, is standard Model View Controller. The base data model is shown in Figure 3.2. The key entities are users and stories. Stories are individual items of content on the site - a story is either a link or a post. Other tables support features of the application such as tagging (Taggings, Tags), comments, activity items, and topics. In the figure, a directed arrow represents a foreign key relationship.

3.2

CATspace

CATspace is a social web-based repository for Computer Science assignments that allows instructors to share, discover, and iterate on programming assignments. The objective of the project is to facilitate the reuse of these assignments and promote their evolution and iterative enhancement through social networking capabilities. CATspace is built as a Face-

Sameer Ahuja

Prototypes

31

Stories Taggings ID tagged_id tagged_type tag_id

Tags

Votes

ID Title Description URL Kind Views Count Popularity Topic_id user_id

ID vote story_id user_id

Users ID login email crypted_password salt session token session expiration activation code activated at realname website bio location twitter_id delicious_id friendfeed_id linkedin_url facebook_url reset_code

Topics ID Titile Description user_id

ID title

Comments ID body user_id story_id

Activity_Items ID Activity story_id user_id

Figure 3.2: Colloki Data Model

Sameer Ahuja

Prototypes

32

book application using the Facebook Developer Platform 3 . This allows us to leverage the social graph of Facebook for the social features of the application. We expect that the ubiquity of Facebook will facilitate the appropriation and acceptance of CATspace in the CS education community by providing a space where instructors may collaborate, regardless of their institutional affiliations.

3.2.1

Introduction

CATspace is a web-based repository that allows instructors to share, discover, and improve programming assignments. The tool supports computer science instructors in developing, refining, and discovering programming assignments in multiple ways. Instructors can add assignments to the repository with descriptions and solution code files, provide feedback to other assignments, contribute to assignments they are co-authors on, and find new assignments through tags, recommendations, and social graphs. Through the use of this tool instructors should be able to minimize the time investment in developing new programming assignments and provide long term contributions to the computer science educators community by adding or providing feedback for assignments. The process of contribution, reuse, and refinement of programming assignments should result in the evolution of higher quality content for programming courses and help to mitigate the duplication of efforts.

3.2.2

System Description

As mentioned in Section 3.2.1 the goals of CATspace are sharing, discovering, and improving assignments. In this section we describe the decisions and reasoning behind the different sections of our user interface. An example screenshot of CATspace is available in Figure 3.3 The main driver of our interface design are the two collections of data that CATspace is 3

Facebook Developer Platform, http://developers.facebook.com

Sameer Ahuja

Prototypes

Figure 3.3: CATspace Homepage

33

Sameer Ahuja

Prototypes

34

Search People

Assignments

My CATspace Home

USERS

ASSIGNMENTS

Figure 3.4: Representation of information that the different CATspace views display. primarily concerned with, assignments and users. Access to this information is provided in two distinct methods, one, is a depth oriented search and the other, is a breadth oriented search. By the term depth oriented search we mean that one collection is completely searchable and viewable without regards to the other collection, this is illustrated in Figure 3.4 by the People and Assignments views. These views provide the users the ability to search and order the different assignments and users within the system. However, where the more interesting aspects of our design reside are in the breadth views, illustrated in Figure 3.4 by My CATspace and Home. Since our application resides within Facebook, we have access to a rich social map for each of our users and can inspect the relationships between them. These relationships give CATspace the ability to make more intelligent suggestions to our users, by combining information about the relationships between users and the assignments that different users have expressed interest in.

Sameer Ahuja

Prototypes

35

Home The Home view is a representation of all of the pertinent information that CATspace can determine about the user through leveraging the social graph and expressed interests. There are sections that show recent activity about tags and people that he/she has expressed interest in. Also, there is a section that shows CATspace’s recommended assignments based on interests the user has specified. Home also shows the user other users with similar interests so that they can grow the social map. Our hope is that this page will encourage our user base to make discoveries of new colleagues and assignments based on expressed interests and social data.

Assignment(s) The Assignments view has multiple purposes, search results, ordered lists, and individual assignments are all displayed in this view. The search results and ordered list functions of the view are meant to provide users of the system with a more traditional way to discover assignments. In this view the user finds assignments they might be interested in without the help of the social graph. The single assignment mode of the assignments view is meant to be the hub of all activity for an assignment. This is where the editing of the assignment’s metadata, source files, and various other fields takes place. This editing is provided with as little navigation as possible and fields are editable directly on the page, by using ajax technologies. This is also where a user that is interested in the assignment ultimately navigates to, as such downloading and providing feedback for the assignment are both done here.

People The People view, similar to the assignment view, servers dual purposes as well. In this view, there are lists of all users that are on CATspace and lists of your Facebook friends that are

Sameer Ahuja

Prototypes

36

using CATspace, this enables users to leverage existing connections and possibly find more. Also, the view of an individual user’s profile is displayed in this view, outlining all of their latest activity on CATspace and displays some of the users in their social map.

My CATspace The My CATspace view is a window of all of the activity associated with a user’s assignments, any assignments that this user is an author for and any feedback associated with these assignments are displayed here. This view is meant to house any of the activities of the user on CATspace, for instance, this is the jumping off point of creating assignments as well as displaying any information about already created assignments and their associated activity.

3.2.3

Implementation

The architecture of CATspace strictly follows the MVC design pattern, as seen in Figure 3.5, and is implemented using Ruby on Rails. In CATspace views, each page of the application has its own class. Since some of the views of CATspace are more interactive than others, some of the views have a further breakdown of classes into sections. In the controllers there is a one-to-one mapping of classes, except for the application controller which is the root controller class that other controllers inherit from. Each action in the controller maps to a view, and generally utilizes more than one model. For example, the “show assignment” action would utilize the assignment model to display the assignment, and the user model to authenticate the user. The models correspond to data sources. The data sources can be database tables, web services, files on the server, or a mixture of these. For example, the user model comprises of a table in the database, and also Facebook’s web services. The assignment model abstracts a database table and a collection of files in the web server’s filesystem. We utilize some other Ruby on Rails mechanisms such as partials in the view layer that

Sameer Ahuja

Prototypes

37

CATspace Layouts

Views Home index

Users profile / my catspace preferences

Assignments index search

show/edit

new

tag

AJAX Response

Activity Items

Application IFrame

AJAX Views

Partials Assignment List People List Activity Item

Controllers Home

Users

Activity Items

Assignments

Application

Models Facebook Users

Assignment

Facebook Publisher

Comment

Activity Item

Authorship

Libraries & Helpers Application Helper JavaProps Read/Write Attachment Handling (Plugin) Spawn Processes (Plugin)

Externals ZIP (RubyZip Gem)

Facebook Web Services API

Catspace Database

Figure 3.5: Object model and basic design of CATspace

Sameer Ahuja

Prototypes

38

provide common modular functionality available to all the views in CATspace.

Data Model The data model for CATspace is shown in Figure 3.6. The key entities are users and assignments. Other tables support features of the application such as tagging (Taggings, Tags), comments, activity items, and authorship roles. In the figure, a directed arrow represents a foreign key relationship. As one might notice, the user model is devoid of any personal information. This is because the personal information is maintained by Facebook and retrieved within our application via Facebook’s developer APIs.

Facebook Integration From a user perspective, facebook forms a proxy between him/her and the application. From the application’s perspective, Facebook App Platform environment proxies user requests. There are two key ways for the application to interact with Facebook in this model (Figure 3.7): 1. Use FBML encoding to respond to user queries. Facebook interprets the FBML response and replaces it with the appropriate information before responding to the user. 2. Use Facebook APIs independent of FBML to query for user data. Developing the application within Facebook’s framework provided both benefits and unique challenges. The primary benefit, as mentioned previously, is that the application is able to leverage the rich social graph of Facebook. However, developing for Facebook is not without difficulties. Some of the difficulties our project ran into include having to implement an ajax and animation javascript library as we could not use existing open source libraries such as prototype.js; and having to add separate session authentication for file handling, because

Sameer Ahuja

Taggings ID tagged_id tagged_type tag_id

Tags ID title

Comments ID body user_id assignment_id

Activity_Items ID Activity assignment_id user_id

Prototypes

{

Assignments ID Title Synopsis Rating Attachment_Name License Language Topics Goals Potential Info_URL Lang_version course est_experience est_time est_size ... user_id

Users ID FB_ID session_key session_expires ROLE ROLE pref_comment_notify pref_tags_follow pref_email pref_download_notify pref_rate_notify pref_tags_notify (...)

Figure 3.6: CATspace data model

39

Authorship ID Role Order assignment_id user_id

Sameer Ahuja

Prototypes

40

Facebook App Platform

User

CATspace

HTTP

APIs

HTTP, FBML

User

Figure 3.7: Interaction between CATspace and Facebook Facebook, at the time of writing the application, did not support returning zip mime type while responding via its proxy.

Edit-in-place for Assignments One of the most important goals for CATspace is to make creation and editing of assignments fairly simple and fast. Long and tedious editing interfaces discourage users from actively contributing to the system. CATspace provides assignment authors a dynamic assignment editing interface that leverages AJAX for providing a WYSIWYG experience. For the fields of assignment title, description and the several metadata fields (e.g., license, license URL, language etc.), the fields are by default represented as standard text on the page. However, when the user hovers the mouse over these fields, the visual representation of the field is modified to represent highlighting by adding a background color and a border. If the user then clicks on the field, then the field converts into an actual HTML text field filled with the value that was being represented in text before. This text field is followed with a couple of buttons to save the new value of the field or to cancel the edit action. The user can then

Sameer Ahuja

Prototypes

41

modify content and click save. This sends an AJAX request to the back-end and if a success response is received, the HTML text field and the buttons vanish, and are replaced by the original text representation of the field, except that the text content is changed to the new value. The tags list is represented as links in its standard form on the page. Hence, in this case, instead of overloading the hover action, a small edit tags link is provided next to the tag list. The user can click it to convert the the list into an editable comma separated list inside a text field. This form of editing provides a richer, faster user experience than a standard edit page for the assignment: 1. Faster: When an author is gradually building up his or her assignment and filling it in with information, then in most cases he or she is just looking to edit one field at a time - for example, a metadata item. Editing via an edit-in-place field is faster, because lesser data is sent and received from the server for an AJAX request to update a field than in the case where the user navigates to an edit page and submits a form wherein all the fields are sent again to the back-end regardless of whether they have been updated or not. 2. Asynchronous: Since AJAX calls are asynchronous in nature, they do not affect the rest of the page elements during their processing. Hence, the user can actually go and start editing another field while the save request of the field he/she just edited is still being processed. Contrast this to an edit form for the assignment page - in which case the user would have to wait both for the edit page to initially loaded and then wait for the data to be stored to the database before he/she can return to the assignment page. 3. Reduces the chance of data loss: A related advantage is that by the use of edit-inplace fields, the assignment’s state is stored at an atomic level - whenever the author

Sameer Ahuja

Prototypes

42

edits a field, that property of the assignment is updated in the database. Contrast this to server-side generation of pages, in which case the user would go to a separate page to edit all the fields. If the user is editing multiple fields on the form and for some reason the browser session is lost before (s)he can save the form, then all the updated information would be lost.

3.3

Social Features

Both of the above applications utilize social features. The Related Work chapter discusses past work on these social features. This section characterizes and compares the specific implementations of these social features across these two applications.

3.3.1

Tagging

Both Colloki and CATspace use tags in similar ways. However, there are subtle differences. To characterize the tagging aspects of these two systems, we use the framework defined in [48]. As detailed in Section 2.1.1, the user incentive characterization can only be determined for a functional system that is being used by the community. Hence, we characterize the two systems only in terms of system attributes, as these tools do not have an active user base in their respective communities yet. 1. Tagging Rights: Both Colloki and CATspace are restricted to self-tagging, that is, a user can only tag resources that she creates. 2. Tagging Support: Both Colloki and CATspace have blind tagging support. They do not (yet) provide features to suggest tags to users while tagging resources. Suggestive tagging can benefit a tagging system by helping it converge to a folksonomy [31]. 3. Aggregation: Being self-tagging systems, both Colloki and CATspace follow the

Sameer Ahuja

Prototypes

43

bag-model for tag entry. In other words, a tag is a property of a resource and a user. However, there is a difference in the way Colloki and CATspace implement aggregation. While in Colloki, two users can have duplicate tags for the same resource for some types of resources (namely, links); in CATspace, each resource (assignment) is unique and only one user can tag it. 4. Type of Object: In Colloki the objects that users can tag are posts and links, as defined in Section 3.2.2. In CATspace, the objects are CS assignments that instructors upload. 5. Source of Material: CATspace allows only resources that are supplied by the participants. Colloki’s resources can be from the participants (posts, links to participants’ content), or could be any web resource (links). 6. Resource Connectivity: CATspace provided linked connectivity between resource via tags; while Colloki provides both linked and grouped connectivity between resources. Colloki provides grouped connectivity via the concept of ‘Topics’ which group related posts and links together (See Section 3.1.2). 7. Social Connectivity: Both CATspace and Colloki provide linked social connectivity independently and with respect to tags. Users are connected when they use the same tag to tag their resources (and by other means, which do not involve tags). Both the systems also provide grouped user connectivity. Both the systems use means to connect users by forming grouping, like top contributors in Colloki, and recommended users in CATspace. Given the above characteristics, we expect tagging on these two systems to provide the navigational constructs that provide affordance for the social actions of social browsing, social search and sharing content. As the system’s usage in the community increases, we also expect users to find affordance for collaboration.

Sameer Ahuja

3.3.2

Prototypes

44

Activity Streams

Both Colloki and CATspace implement activity streams in a fairly similar manner. On CATspace, However, there are some differences. In Colloki, activity streams are shown on the homepage and the topic pages. In CATspace, these are more widely available - The homepage, the user profile, and assignment page; all show activity streams. CATspace also provides the ability to delete items from a user’s activity stream. In Colloki the only way to remove an activity stream is to undo the activity (Deleting a comment or un-voting an item). These are conscious design decisions taken based on the context of the application. CATspace exposes more information as part of the activity stream - the actions shown on the stream are publishing assignments, commenting, rating, tagging and editing of published assignments. Instructors might sometimes want to remove items that they feel might mislead viewers, or are just not important enough to be published to the stream. In Colloki, all actions that are published on the stream (Creation of posts/links, commenting, voting) are public in nature, and removing items from the stream goes against the very public and open nature of the system. There are no filtering/grouping features in either application yet. Based on our design choices, we expect activity streams to be an engaging means to navigate across the content on the system. Hence the social actions that we expect the feature (as implemented in the two systems) to provide strong affordance for social search and social browsing.

3.3.3

User Profiles

User profiles, as mentioned in Section 6.1.2, are central to social network sites. These two applications are no different. Colloki’s profiles are shown in Figure 3.8, and CATspace’s user profiles are shown in Figure 3.9. Both applications use the profiles to showcase the content that the user has contributed to the system, and provide the users ability to provide

Sameer Ahuja

Prototypes

Figure 3.8: A user profile on Colloki

45

Sameer Ahuja

Prototypes

46

additional information about themselves. Colloki has several fields in the profile that let the user provide name, a short bio, an avatar image, location, and link to website. In addition, users can provide their usernames on other social systems (LinkedIn.com, Twitter.com, Facebook.com, Friendfeed.com, and delicious.com) that are converted to icons on the user profile that link to the user’s profile(s) on those systems. In addition, the profile provides a showcase of the latest content and comments by the user, and a tag cloud of the user’s tags. CATspace is different because it builds over facebook’s social graph. It pulls the user’s profile photo from Facebook and shows it in the profile along with the user’s name. The user can supply their affiliation separately in the CATspace application. There are no other custom fields, the rest of the profile is a showcase of the user’s assignments on CATspace (Figure 3.9) User profiles are central information sources for people, so by definition we expect them to strongly support the actions of social browsing and social search; and provide good affordance for sharing.

3.3.4

Comments

Both Colloki and CATspace have fairly straightforward implementations of commenting. The feature is designed to essentially facilitate conversations, hence we expect it to strongly support interaction. Comments also help in discovery of people’s thoughts and opinions, hence we foresee benefits in the social actions of social search and social browsing. By providing a means to conversation, we also expect users to find comments to be beneficial for collaboration.

Sameer Ahuja

Prototypes

47

Figure 3.9: A user profile on CATspace (Picture used with the permission of Dr. Stephen Edwards, 2009)

Sameer Ahuja

3.3.5

Prototypes

48

Ratings and Votes

Colloki and CATspace implement user ratings/votes in very different ways. In Colloki, users can ‘vote’ for an item that they have not created. Voting is an atomic activity and has a binary score - you either vote or you do not. It also provides users the ability to un-vote a piece of content after they have voted for it. An activity stream item is generated when users vote for an item, and the only way to remove that item is to un-vote the content. Hence, there is no way to secretly vote for a piece of content. votes are used to derive the sort order on the default view of topic pages, hence they are central to the organizational structure of Colloki. CATspace provides users a 5-star rating system. Users can change their rating of an assignment, but once having rated an assignment, cannot un-rate it. Rating activity does not show up on the activity stream of the system. These ratings are then used to provide one of the sorting orders in the assignment views. In the future, we will also use ratings to provide leaderboards - listings of the top rated assignments and the top rated users (via association) in the system. Ratings and votes provide building of the social capital via community-powered organization of content and via reputation scores that lead to user interface artifacts such as leaderboards. These kind of social mechanisms are out of the scope of our current study, however, in the actions that we are measuring in our study, we expect users to find ratings and votes helpful in finding relevant content, an activity could then lead to benefits for social search and social browsing.

Chapter 4 Methodology This chapter presents the methodology used for performing the evaluation study for the two tools. This study was conducted as a usability evaluation along-with pre and post evaluation surveys. We had ten participants in total, five in each of the individual studies. All participants were graduate students at Virginia Tech. All five participants of CATspace study worked in the Computer Science Department, while the participants of Colloki study were from other departments across campus. Each participant was given a short Pre-Evaluation survey that comprised of questions on internet usage and academic background.The participant were then provided a short video demonstration of the site they were evaluating, and then asked to perform a set of tasks on that site.This was followed by a comprehensive postevaluation questionnaire on the site, that had questions on usability and perceived social affordance.

4.1

Goals

The main goals of this study is to perform a formative evaluation of the two sites. This evaluation, in addition to evaluating the usability of the two sites, should explore how the

49

Sameer Ahuja

Methodology

50

the perceived social affordance of the features for the social actions of social browsing, social search, interaction, sharing and collaboration matched the design expectations of the features of CATspace and Colloki, and the outcomes thereof.

4.2 4.2.1

Experimental Setup Selection of Participants

We recruited 10 graduate students from the town of Blacksburg as participants for our evaluation. The evaluation used two different groups of 5 users each for the two applications. For Colloki, we recruited graduate students studying at Virginia Tech, with the additional condition that the participants need to be studying in a department other than Computer Science. This is because the benchmark tasks of Colloki can be simpler for people who have contextual information about the Department of computer Science at Virginia tech, and its members. To avoid any such circumstances, we made sure that none of the participants had contextual information about the Department that would help them in the tasks. For CATspace, we recruited graduate students studying at Virginia Tech, with the additional condition that the participants need to studying in the Department of Computer Science. This is because we felt that the graduate student community in the Computer Science department provides a good approximation of the kind of community that we expect to be using CATspace when it is deployed in the wild. Both the user groups were invited via individual emails to participate in the study. The participants received a gift coupon worth $5.50 each in return of their participation.

Sameer Ahuja

4.2.2

Methodology

51

Usability Lab Setup

The usability evaluation was setup in a closed lab in the Knowledgeworks II building of the Corporate Research Center at Virginia Tech. This room was not accessible to the general public when the study was in session. The evaluator and the participant were both present in the room at the same time. Once the tutorial video was completed, communication between the evaluator and the participant occurred only when the participant had questions or comments. In general, there was no communication between the evaluator and the participant, during the time of the actual tasks. The setup included the following: 1. Computer with Internet access: A computer with Internet connection was used for the evaluation. Shortcuts to the prototype and the demo video were provided on the desktop of the computer system. 2. Screen capture software: A screen capture software was used to record the users’ on-screen activity. These recordings helped us find critical incidents in the evaluations.

4.2.3

Evaluation Protocol

The study detailed here is formative in nature. The purpose of formative evaluation is to validate or ensure that the goals of the instruction are being achieved and to improve the instruction, if necessary, by means of identification and subsequent remediation of problematic aspects [74]. According to [63], “When the cook tastes the soup, that’s formative; when the guests taste the soup, that’s summative.” Formative evaluation is intended as a basis for improvement. Both the studies (CATspace and Colloki) were approved by IRB at Virginia Tech, please see Appendix A for the letters of approval. When the participants arrived in Room 1132 at Department of Computer Science Building, Knowledge Works II they were greeted by

Sameer Ahuja

Methodology

52

the evaluator, and the purpose of the evaluation was explained to them, along with the evaluation setup, the procedure, confidentiality of data and the approximate duration of the evaluation. They were then handed the Informed Consent Forms to read and sign if they were willing to participate in the evaluation. Once the participants signed the Consent forms they were instructed to keep one copy for their records. Then, we asked them to fill out an online pre-evaluation survey. This survey included a short demographic survey and a survey that asked the participants questions about the depth and the breadth of their usage of social network sites. After completing the survey, each participant was asked to view a video demonstration of the software. The video goes through the main features of the software that the users would be using in their evaluations. Following the video demo we handed the participants a sheet that contained a set of predesigned benchmark tasks. These were a set of tasks derived from the scenarios, and are described in Section 4.3. We then reminded the users that they are not going to be timed on any of the tasks. Following these tasks, we asked the participants to fill out a post-evaluation questionnaire. The post-evaluation questionnaire is designed to provide feedback about the tool and the evaluation procedures. This questionnaire form helped us get information about the participants experience using the tool with respect to usability, and the perceived social affordance of the tool’s features. It also provided space for free-form feedback by the participants.

4.3 4.3.1

Benchmark Tasks Colloki

For the tasks, the users were asked to assume that they had just joined the Computer Science department at Virginia Tech as a graduate student and were interested in learning

Sameer Ahuja

Methodology

53

more about the department’s student community. 1. Use the following credentials to log on to the site Username: participant Password: participant 2. Browse through the list of links and posts in the topic ‘Local Restaurants’. Is there a restaurant you know in the area that is missing? Add a link to the website of that restaurant to the topic along with some description and tags. (You can use web search or other tools to lookup local restaurants and/or their websites.) 3. Go to the ‘Graduate Life’ topic, and view the most recent links in that topic. Vote for the link that you like the most. (You can click through on the links and visit the sites to decide your favorite link). Now, leave a comment on the link mentioning why you like it. 4. Which is the oldest post in the topic ‘Graduate Life’ ? 5. Which one of the five topics listed on the homepage do you think is the most popular amongst the slurp community? 6. Which one of the five topics listed on the homepage do you think has been the most active recently amongst the slurp community? 7. From the department website, you know that Sameer Ahuja works in the same research group that you are interested in. Find him in Slurp. Once you’ve found him, find out two people that he has had the most interactions with (via comments) on slurp. 8. Manuel A. P´erez-Qui˜ nones is the professor who heads the research group that you are interested in. His user id on Slurp is ‘mapq’. His top tag on Slurp represents his primary research area: ‘PIM’. Name one more user in the system whose top tag is ‘PIM’.

Sameer Ahuja

4.3.2

Methodology

54

CATspace

For the tasks, the users were asked to assume the role of a fictional professor in Computer Science, who teaches the introductory CS class and is interested in the topics of algorithms, arrays and linked lists. They were logged into the application as that user. 1. Go to your CATspace preferences page and add the tags ‘CS1’ and ‘arrays’ to your list of tag subscriptions. 2. You are looking for an assignment for your upcoming class on two dimensional arrays. Find an assignment that’s tagged both ‘CS1’ and ‘2d array’. 3. Find which of your Facebook friends are already using CATspace. Name one of these friends. 4. Name two other people that the person you named in the last question has as co-authors on his/her assignments. 5. ‘Stephen Edwards’ is a very well known CS instructor working in your area of teaching who had earlier mentioned to you that he is already on CATspace. Find his profile on CATspace. How many assignments has Stephen published on CATspace? Which is the latest assignment that he’s published? What’s his top rated assignment? 6. Is ‘CS1’ amongst the top tags for Stephen? Find another who has ‘CS1’ as one of his/her top tags. 7. You decide to contribute to the CATspace community by adding one of your own assignments. Create a new assignment with the title ‘Single dimensional arrays’ and tag it with tags ‘CS1’ and ‘1d arrays’. Upload the assignment.zip file available on the desktop to this assignment’s file. 8. After making these changes, you realize that the assignment also has a question on linked lists. Edit the assignment’s title to ‘Single dimensional arrays and linked lists’,

Sameer Ahuja

Methodology

55

Table 4.1: Expected mapping of Social Features to Social Actions Social

Social

Interact-

Sharing

Collabor-

Browsing

Search

ion

Content

ation

Tagging

High

High

High

Good

Activity Streams

High

High

User Profiles

High

High

Comments

Good

Good

High

Good

Ratings and Votes Good

and add ‘linked lists’ to the list of tags. 9. You’re now finishing up the assignment. Let’s make a final set of changes. Set the metadata property called ‘Course’ for this assignment to ‘CS1’. Also, update the readme.txt file in the assingment’s files - change the first line of the file from ‘Arrays’ to ‘Arrays and Linked Lists’. 10. Publish the assignment.

4.4

Perceived Social Affordance

The goal behind measuring perceived social affordance is to get a sense of whether users see the social utility of the software. This utility that users see comprises of software and social effects. In the evaluation, we control for the social effects by providing each user with exactly the same state of the system when they evaluate it. This state was represented by a simulated set of users, content and interaction. Below, we describe the guidelines we used for simulating the community around the software.

Sameer Ahuja

4.4.1

Methodology

56

Simulated Community

At this stage, the software does not have a community of people using the content. However, for a meaningful evaluation, the users need to be introduced to a state of the system with some content already present. For this purpose, we simulated a community of users and their content. The kind of community and content to simulate is contextual to the kind of application being built, but the following general guidelines were followed while building the communities: 1. The scale of the content on the site should be just enough to provide the users an indication of how features work. For example, in a listing of content, any content that falls onto the second page is extra. One should be careful to not place too much content on the site, as this might start affecting the user’s perception of the social utility to greater extents. As the general guidance, for all the central views in the prototypes, we filled the views with content enough to fill up the first fold/page. 2. The content should model the kind of content that members of the community are expected to post.

4.4.2

Mapping social features to actions

Table 4.1 shows the expected mapping of features to social actions. Each element in this grid is a mapping between a single social feature and a single action; and it represents how well the social feature is perceived to be for that particular social action. The social actions listed here are sufficiently abstract to be representative for a set of characteristic actions. Yet the actions listed are easily differentiated from each other. The values in this table have been filled based on our design expectations from the features, and the user responses were evaluated based on these expectations. To measure the individual values, we provide users statements for each grid element, and ask

Sameer Ahuja

Methodology

57

them to evaluate them on the Likert scale. For example, to evaluate the perceived affordance of tags for interaction in CATspace, we provide the following statement to the users: Tags will be helpful for me to interact with other people on CATspace. These questions are asked in the post evaluation questionnaire. The complete questionnaires are provided in Appendix F and Appendix G.

Chapter 5 Results In this section, we discuss the results from the evaluation study. We break down the results based on features, and provide results for both sites together to facilitate comparison. Given the low scale of the study (5 participants for each site), we’ve concentrated on results that show a clear opinion across participants, in terms of a unanimous agree or disagree on the likert scale questions. To provide a visual aid for understanding the results, we provide figures in each section that show the user responses to the likert-scale questions in a color-coded grid of responses, along with the averages. The questions that we asked the participants are labelled as A1 to A43 for the Colloki study, and B1 to B55 for CATspace participants.

5.1

Overall Usability

Overall, participants found the two sites easy to use (Questions A1, B1 in Figure 5.1). The participants also unanimously found the basic tasks of creation and editing of content (Questions A2, B2, B3, B4 in Figure 5.1) to be easy to use. However, for the two questions that we asked related to articulating relationships via tags (Questions A5, B7) or comments (Question A6); we found that the users clearly did not

58

Sameer Ahuja

Results

Figure 5.1: Participant responses for questions on overall usability

59

Sameer Ahuja

Results

60

find it easy to use the features in their current state to perform the task of articulating relationships. We consider that an important task because its a central task with respect to the definition of SNS that we are using, and also because it supports the social action of social browsing. Some of the free-form feedback users provided gives us a deeper understanding of why participants did not find these tasks easy to use. One of the participants of Colloki’s evaluation mentioned in the feedback: The Tagging thing was odd. I had to click on a tag which took me to a page that had all posts with the tag. Then from there I had to click on one of the posts, then go to its author and see if he or she had that tag in the tag cloud. That’s cumbersome. Another participant, this time from the CATspace evaluation, mentioned: I think this profile looks a little empty. You can have a list of users with similar stuff here, kindof like facebook friend list. Actually, you could show the person’s facebook friends (who are on CATspace) right here. The results and these qualitative feedbacks clearly indicate that the system needs to aid the users in articulation of relationships amongst users of the system, by providing explicit interface elements that showed the relationships we deem important, ideally in user profiles. An example of such an interface element is providing a list of users who have a similar set of top tags, on the user’s profile in the system.

Sameer Ahuja

Results

Figure 5.2: Participant responses for questions on tagging

61

Sameer Ahuja

5.2

Features

5.2.1

Tagging

Results

62

At a basic level, participants on both evaluations found tags easy to understand conceptually (Figure 5.2), and found it easy to create tags and navigate the systems based on tags. This assures us that there aren’t any conceptual issues with the implementation. However, the implementation of tagging fell one step short of providing users affordance to perform slightly more complex tasks, like finding people using similar tags. Participants on the Colloki study did not agree that finding similar people was easy (Question A5 in Figure 5.1). The CATspace users gave varied responses to the same question, with two strongly agreeing, two disagreeing, and one user was neutral (Question B7 in Figure 5.1). As discussed above, this task related to articulation of relationships on the network, and has direct impact on the social actions of social search and social browsing. User responses indicate that the system needs to explicitly articulate relationships based on tags in user profiles in CATspace. In Colloki, while tags are used for organization orthogonally to topics, we are not as concerned about relationships based on tags, as we are on relationships based on interaction. With respect to social actions, the perceived utility of tagging in Colloki for different social actions generally matched the design expectations. In the case of CATspace, however, a couple of social actions saw a wide variety in users’ opinions of them. Colloki participants generally agreed with the utility of tagging for all five social actions evaluated (Figure 5.2). The participants felt that tagging would be most helpful for effectively sharing content with other users on the site. Interaction got the least score as the action that tagging facilitates. These results are in accordance with the design goals for the tagging feature, as discussed in Section 3.3.1. In CATspace, however, the participant reactions were more varied. The three actions that we deemed most important for users to perceive in our design were social browsing, social

Sameer Ahuja

Results

63

search and sharing content. Participants generally strongly agreed in the utility of tagging for social browsing - the average likert scale score for the related question (B13) was 4.4, with a standard deviation of 0.89. However, the scores for sharing assignments were spread out across the participant - while one of the participants strongly disagreed with the assertion, another was neutral, and the rest strongly agreed with the statement that tagging would be helpful for sharing content. For the other two social actions that we had deemed that tagging would be useful for, participants were generally neutral to the utility of tagging for social search, but were more positive towards its utility for collaboration (Average score of 4.2, with standard deviation of 1.1)

5.2.2

Activity Streams

There were significant differences in the way that participants of the Colloki and CATspace studies evaluated activity streams. As visible from the responses to A16 in Figure 5.3, there were varied opinions on how intuitive and easy to understand activity streams were. These findings are in contrast to those of CATspace, where all of the participants responded positively (B18 in Figure 5.3). We discuss possible explanations to the same later in this section. In our design of activity streams, we had expected them to be useful for social discovery and social our expectations for activity streams as vehicles of social browsing and social discovery. Clearly, the participants did not see that utility (Responses to A18, A19, B20 and B21 in Figure 5.3). This cannot be attributed to the lack of conceptual understanding of activity streams - most of the participants were regular users of Facebook, a site where ‘News Feeds’ about friends is a central concept. The above seems to point that there might be issues in our design or implementation of activity streams. Surveying participant videos provides some hints - none of the participants noticed used the activity stream for navigation. Also, both in Colloki and CATspace, when users were asked to find information that was available in a straightforward manner within

Sameer Ahuja

Results

Figure 5.3: Participant responses for questions on activity streams

64

Sameer Ahuja

Results

65

activity streams (Task 6 in Colloki, Task 4 in CATspace), they instead chose other means to reach the information. This might indicate that activity streams were not visually prominent enough in the interfaces to be noticeable. In Colloki, another issue related to activity streams was that they were not ubiquitous - the users’ profiles did not have the stream of activities that those users had performed, and similarly the pages for individual posts and links did not show the activity performed on those items.

5.2.3

User Profiles

User profiles, as discussed before, are a primary component of social networking sites, and hence a critical feature for the system to get right. Participants found user profiles easy to understand, and they consistently saw its utility in both the systems. In terms of social actions, we expected user profiles to be useful primarily for social browsing and social search. The results were in accordance with our expectations - participants fairly unanimously considered user profiles to be helpful for social browsing and social search. Users of both Colloki and CATspace also found user profiles to be unanimously useful for another form of social action - interaction (Questions A41 and B41 in Figure 5.4). This is interesting, because user profiles on both the system did not provide any direct means for interacting with the person. They did, however, provide links to other places on the system, or outside of the system, where interaction with the user was possible. One possible explanation for the users to vote unanimously in this regard is that they brought their expertise of existing social network sites with them. When they thought ‘user profiles’, they thought a place to find about and interact with someone. Evidence for this conceptual thinking is provided by one participant’s feedback to CATspace’s user profiles [...] but where can I leave him a message? Do I have to go to the Facebook profile for that? How do I go to the Facebook profile from here? Participants thought of user interaction as one of the obvious things to do on a user’s profile.

Sameer Ahuja

Results

Figure 5.4: Participant responses for questions on user profiles

66

Sameer Ahuja

Results

67

This feedback has a consequence to our designs for the next iteration of the product: Knowing that people expect to have the social affordance of interaction on a user profile, we can build features that are directed towards that goal.

5.2.4

Comments

Both the applications had fairly straightforward implementations of commenting. We expected users to find commenting helpful primarily for interaction and collaboration. Participants on both studies agreed with us for interaction (Questions A27 and B28 in Figure 5.5). For collaboration, however, it was a different story - In both the studies, a couple of users were neutral about the utility of collaboration. We attribute this partly due to a fault in our study, that we discuss in Section 6.1.3. As detailed in Section 3.3.4, we rated social browsing and social search as secondary benefits of commenting - a person’s comment tells you something about the person, and in these two systems it is fairly simple to navigate to people’s comments. In the results, we found unanimous agreement on the affordance of commenting for social browsing in both the applications, but not social search.

5.2.5

Ratings and Votes

Ratings and votes bring community controlled organization to social network sites. The users on Colloki found the voting feature to be easy to use almost unanimously - one user was neutral. On CATspace, the feature was not completely developed - the user interface for making the ratings had a couple of bugs. Hence we did not have a task for rating an assignment, only for organizing the view based on already-existing ratings.

Sameer Ahuja

Results

Figure 5.5: Participant responses for questions on comments

68

Sameer Ahuja

Results

Figure 5.6: Participant responses for questions on votes/ratings

69

Chapter 6 Discussion and Conclusions This chapter is divided into three sections - first, we discuss the outcomes of the study for the two social network sites we evaluated. Second, we summarize and discuss the evaluation procedure. Finally, we discuss future work towards evaluation of social network sites.

6.1

Outcomes

Based on the results, we obtained the following main outcomes for the two social networking sites:

6.1.1

Articulation of connections

From the results, a fairly clear design flaw with both the applications was that they did not provide a direct means to articulate and navigate social relationships. The recommendation for Colloki is to either clearly articulate social relationships (formed via using the same tags or commenting on each others’ posts) on user profiles, and let users navigate those; or to provide a different, more straightforward concept of social connections - such as mutual relationships of facebook, or the unidirectional ‘follower’ relationships of twitter. 70

Sameer Ahuja

Discussion and Conclusions

71

For CATspace, the problem is more complex - The underlying platform of Facebook already provides explicit social connections, and there are implicit social connections formed by people using the same tags. Neither is clearly articulated on user profiles. We see utility in exposing both of the relationships as they facilitate different social actions - articulating facebook connections may help people find more about people they already have as social connections, hence facilitating social search; while articulating content-based relationships facilitates social browsing by listing users who have posted or are subscribed to similar tags. Hence, the first step will be articulating both on the CATspace user profiles. In the next evaluation, it might make sense to ask users to evaluate the two relationships for social affordances and to evaluate whether articulation of two kinds of relationships confuses the users in any manner.

6.1.2

Activity Streams

Users on both sites rated activity streams lower than we expected. They consistently used other means of navigation for finding relevant pieces of information. The key takeaway for us was to reset our expectations. Both Colloki and CATspace are content centric, and provided clear affordances for navigation. Activity streams are great for serendipitous discovery, however; when asked to perform tasks, users responded by utilizing the feature that most clearly provided the affordance for the task required - which in both cases was not the activity stream. This also raises an interesting question - are social affordances of features in a site a zero-sum game (that is, if add a feature that provides a certain social affordance for some action, does the affordance for that action decrease for the other features)? It may very well be true to a certain extent, and there is a clear analogue in conventional usability engineering to this situation - If you let the user do the same thing in several ways, you might end up confusing the user, causing overall perception of usability to drop. So its important to think of what the key features for social actions are, and hence in our expectations account for the reduction in perceived affordance, caused by having a feature facilitating a social action

Sameer Ahuja

Discussion and Conclusions

72

which is also facilitated by a more central feature to the application.

6.1.3

Task Sets

In our analysis of the results, we realized that the social action of collaboration received low scores on all social features. On further examination, we found that collaboration was the only social action that the users did not perform as part of their task scenarios. We think that this causes the ratings to distort, especially in this controlled environment. Hence, we decided to ignore all results for collaboration for any of the outcomes, and more importantly: we tuned the study for future instances to provide task scenarios that reflected on all the social actions that we are measuring.

6.2

Evaluation Procedure

Going back to the overarching problem that we established in Chapter 1: How to perform formative evaluations of a social network site, at a stage where it has not been released and the user community has not started participating, and get a sense of how successful it would be when released for use by that community? So how does the evaluation we performed in this thesis help in determining how successful the site would be in the community? In the context of social actions based evaluation, the question of ‘how successful is the site is going to be’ can be broken down into three constituent questions: 1. Do the users of the site perceive the features of the site to have social affordances for the actions that we expect the features to have social affordances for?

Sameer Ahuja

Discussion and Conclusions

73

2. Will the users, even when they do perceive the social affordances, actually perform the social actions? 3. Finally, if the users do perform these social actions, how would that relate to the central goals of the application? Formative evaluation gives us a good sense of the answer to the first question. The second question deals with user incentives and motivations, and the question is out of scope for formative evaluation, and the results obtained cannot predict usage. Finally, what does it mean for the application’s goals if the users perform the expected actions? The actions have been framed as positive in our work - that’s derived from past literature that has related these actions to the rise of social capital for the community. The rise in social capital is what connects the site’s goals to the social actions, and it brings along the social utility that social network sites hope to achieve. However, one cannot overrule the possibility of negative effects of certain social affordances of features, or certain social actions. Feedback from participants during formative evaluation can help catch some of these negative consequences at an early stage, but in essence this is something that relates to the second question above - it is very contextual to the community that ends up using the software. Following is a summary of the evaluation procedure. 1. Design features for the site while considering which social actions those features are expected to facilitate. With this information, construct the expectations matrix. The social actions listed in this thesis are not an exhaustive list, but they provide a good foundation of social actions to consider for most common contexts. 2. Perform an evaluation with a set of participants that are ideally a representative set from the community the site is intended for. This evaluation is designed to perform a formative usability and social affordance evaluation. As part of this evaluation, (a) Perform a pre-evaluation on the participants on their expertise with social network sites, and particularly, the social features that they are going to be evaluated on.

Sameer Ahuja

Discussion and Conclusions

74

(b) Provide a demonstration or a guided tour of the features to the participants. (c) Provide the participants with the prototype site. Simulate a community and content on the site, based on the guidelines provided in Section 4.4.1. Provide the users with a set of tasks to perform on the site. These tasks are to be derived as per standard usability engineering guidelines - from scenarios of usage. These scenarios should cover the primary task(s) of the application (Such as uploading and editing assignments in CATspace, adding links in Colloki); the primary social tasks of articulating social connections and navigating those connections to reach user profiles; and finally, tasks based on the expected social actions. The tasks are a means for not just measuring the task completion, but also to provide the users a sense of the social utility of the software. (d) Provide the participants with a post-evaluation questionnaire, that asks them three sets of questions - Likert scale questions on the usability of the features of the site, Likert scale questions on the perceived social affordance of the site, and finally ask them for free-form feedback. This last set can be substituted for by a short interview. 3. With these results, find strong mismatches in expectations and results of perceived social affordance. Such mismatches might mean one or more of three things - there might be a design issue with the software, there might be an issue with the study (For example, the tasks might be covering some social actions and not others), or it might mean that the expectation was wrong - that the users either would not perceive the utility with the software until the sociological effects start playing their role or that the feature was not as important for the particular social action. Any number of these three aspects might require tuning. Participants’ past experience with SNS, the critical incidents from the task performance, and the feedback from the participants can help provide more hints to root causes, and lead to recommendations.

Sameer Ahuja

6.3

Discussion and Conclusions

75

Future Work

This thesis is a first step towards studying the design and evaluation of social network sites from a practitioner’s perspective. There are some key areas where we see future work in the area. One, it would be interesting to extend the social actions based framework to measuring user behavior on these sites when they are released to the community, and see if there are any strong generalizable relationships between perceived affordances and actual use. Secondly, an interesting trend had started to emerge in this study, a trend that was out of scope for this study to explore, but merits further study. This trend was that the features appeared to have similar affordances across the sites - that is, the set of actions users users seemed to think a particular feature facilitated seemed to be similar across sites. This was largely due to the fact that since these two applications had the same designers, they had very similar implementations of features. However, it would be interesting to study affordances for social actions for particular features - If there seems to be a base ‘behavior’ for a feature, it might be worthwhile to start thinking of these features as design patterns with the set of default social actions being facilitated as a default property of the pattern. Finally, this study was run as a controlled formative study. By nature, its a low-cost, highturnaround approach. Another approach to consider for future work on formative evaluation of social network sites is a more longitudinal study, wherein a focus group uses the software for period of time and then performs an evaluation. Such studies for social software have been performed before [27], and its apparent that the participant responses might be more in-depth in a longitudinal study, given that they use the software for a much longer period. Also, it might be easier to evaluate some kinds of social actions - such as collaboration. In case of a longitudinal study, a group of users can be given a common task to be performed, whereas in the case of a controlled study it becomes harder to provide tasks that give a real sense of collaboration. However, longitudinal studies might face the issue of users linking the technology’s performance to the performance of the community of participants in the study. They are also costlier to perform in terms of resources and time.

Bibliography [1] User-generated content drives half of u.s. top 10 fastest growing web brands. Nielsen NetRating, 2006. [2] F. Ackerman, T. McGuire, T. Scott, and J. Peterson. Nifty assignments. Journal of Computing Sciences in Colleges, 24(1):257–262, 2008. [3] F. Adam and B. Roncevic. Social capital: recent debates and research trends. Social Science Information, Jan 2003. [4] P. Adler and S. Kwon. Social capital: the good, the bad, and the ugly. KNOWLEDGE AND SOCIAL CAPITAL: FOUNDATIONS AND APPLICATIONS, Eric L. Lesser, ed., Butterworth-Heinemann, Boston, MA, pages 89–115, 2000. [5] S. Ahuja, M. P´erez-Qui˜ nones, A. Kavanaugh, and C. Tauro. Colloki: Rethinking local conversations on the web. Proceedings of Directions and Implications of Advanced Computing - Conference on Online Deliberation, 2008. [6] N. Ali-Hasan and L. Adamic. Expressing social relationships on the blog through links and comments. Proceedings of the International Conference on Weblogs and Social Media, Mar 2007. [7] D. Beer. Social network (ing) sites. . . revisiting the story so far: A response to danah boyd & nicole ellison. Journal of Computer-Mediated Communication, Jan 2008.

76

Sameer Ahuja

Bibliography

77

[8] M. K. Boulos and S. Wheeler. The emerging web 2.0 social software: an enabling suite of sociable technologies in health and health care education. Health Information & Libraries Journal, Jan 2007. [9] D. Boyd. The significance of social software. BlogTalks reloaded: Social software research & cases, pages 15–30. [10] D. Boyd. Why youth social network sites: The role of networked publics in teenage social life. The John D. and Catherine T. MacArthur Foundation Series on Digital Media and Learning, Jan 2007. [11] D. Boyd and N. Ellison. Social network sites: Definition, history, and scholarship. Journal of Computer-Mediated Communication, Jan 2008. [12] M. Burke, C. Marlow, and T. Lento. Feed me: motivating newcomer contribution in social network sites. CHI ’09: Proceedings of the 27th international conference on Human factors in computing systems, Apr 2009. [13] A. Carbone, J. Hurst, I. Mitchell, and D. Gunstone. Principles for designing programming exercises to minimise poor learning behaviours in students. Proceedings of the Australasian conference on Computing education, Jan 2000. [14] C. Charron, J. Favier, and C. Li. Social computing. How Networks Erode Institutional Power, Jan 2006. [15] M. Choudhury, H. Sundaram, A. John, and D. Seligmann. What makes conversations interesting?: themes, participants and consequences of conversations in online social media. WWW ’09: Proceedings of the 18th international conference on World wide web, Apr 2009. [16] A. Cockburn. Agile software development. Jan 2002. [17] D. Cohen and L. Prusak. In good company: How social capital makes organizations work. Ubiquity, Jan 2001.

Sameer Ahuja

Bibliography

78

[18] D. Cosley, S. Lam, I. Albert, J. Konstan, and J. Riedl. Is seeing believing?: how recommender system interfaces affect users’ opinions. CHI ’03: Proceedings of the SIGCHI conference on Human factors in computing systems, Apr 2003. [19] E. Courses and T. Surveys. A study of collaborative software development using groupware tools. Information Technology: Research and Education, Jan 2003. [20] C. Dellarocas. Immunizing online reputation reporting systems against unfair ratings and discriminatory behavior. Proceedings of the 2nd ACM Conference on Electronic Commerce, Jan 2000. [21] A. Dieberger, P. Dourish, K. H¨o¨ok, and P. Resnick. Social navigation: Techniques for building more usable systems. interactions, 7(6):36–45, Dec 2000. [22] J. DiMicco, D. R. Millen, W. Geyer, C. Dugan, B. Brownholtz, and M. Muller. Motivations for social networking at work. pages 711–720, 2008. [23] S. H. Edwards, J. B¨orstler, L. N. Cassel, M. S. Hall, and J. Hollingsworth. Developing a common format for sharing programming assignments. ACM SIGCSE Bulletin, 40(4):167–182, 2008. [24] S. H. Edwards and M. A. Perez-Quinones. Web-cat: automatically grading programming assignments. pages 328–328, 2008. [25] N. Ellison, C. Steinfield, and C. Lampe. Spatially bounded online social networks and social capital. International Communication Association, Jan 2006. [26] T. Erickson and W. Kellogg. Knowledge communities: Online environments for supporting knowledge management and its social context. Sharing Expertise: Beyond Knowledge Management, Jan 2003. [27] S. Farnham, S. Kelly, W. Portnoy, and J. Schwartz. Wallop: designing social software for co-located social networks. Proceedings of the 37th Annual Hawaii International Conference on System Sciences, Jan 2004.

Sameer Ahuja

Bibliography

79

[28] U. Farooq, T. Kannampallil, Y. Song, and C. Ganoe. Evaluating tagging behavior in social bookmarking systems: metrics and design heuristics. Proceedings of the 2007 international ACM conference on Supporting group work, Jan 2007. [29] G. Furnas, C. Fake, L. Ahn, J. Schachter, S. Golder, K. Fox, M. Davis, C. Marlow, and M. Naaman. Why do tagging systems work? CHI ’06: CHI ’06 extended abstracts on Human factors in computing systems, Apr 2006. [30] S. Ghita, W. Nejdl, and R. Paiu. Semantically rich recommendations in social networks for sharing, exchanging and ranking semantic context. Lecture notes in computer science, Jan 2005. [31] S. Golder and B. Huberman. The structure of collaborative tagging systems. Journal of Information Science, Jan 2006. [32] S. Golder, D. Wilkinson, and B. Huberman. Rhythms of social interaction: Messaging within a massive online network. Communities and Technologies 2007: Proceedings of the Third International Conference on Communities and Technologies, Michigan State University, East Lansing, Michigan, June 28-30, 2007, Jan 2007. [33] F. Harper, S. Li, Y. Chen, and J. Konstan. Social comparisons to motivate contributions to an online community. Lecture notes in computer science, Jan 2007. [34] M. Hawkes, D. Talley, and J. Webster. An introduction to the multimedia educational resource for learning and online teaching (merlot). Journal of Computing Sciences in Colleges, 17(1):255–256, 2001. [35] N. V. House. Flickr and public image-sharing: distant closeness and photo exhibition. CHI ’07 extended abstracts on Human factors in computing systems, pages 2717–2722, 2007.

Sameer Ahuja

Bibliography

80

[36] J. Howarth, P. Pyla, B. Yost, and Y. Haciahmetoglu. Designing a conference for women entering academe in the sciences and engineering. Advancing Women in Leadership Online Journal, 23, 2007. [37] C. Jensen, J. Davis, and S. Farnham. Finding others online: Reputation systems for social online spaces. Proceedings of the SIGCHI conference on Human factors in computing systems: Changing our world, changing ourselves, Jan 2002. [38] A. Joinson. Looking at, looking up or keeping up with people?: motives and use of facebook. CHI ’08: Proceeding of the twenty-sixth annual SIGCHI conference on Human factors in computing systems, Apr 2008. [39] S. Jones and S. Fox. Generations online in 2009. PEW Internet and American Life Project, 2009. [40] A. Kavanaugh, H. Kim, M. P´erez-Qui˜ nones, and P. Isenhour. Models of government blogging: Design trade-offs in civic engagement. Communities and Technologies 2007: Proceedings of the Third International Conference on Communities and Technologies, Michigan State University, East Lansing, Michigan, June 28-30, 2007, Jul 2007. [41] A. Kittur, B. Suh, B. Pendleton, and H. Chi. He says, she says: Conflict and coordination in wikipedia. Proceedings of the SIGCHI conference on Human factors in computing systems, pages 453–462, 2007. [42] M. Klein, L. Iandoli, and I. Federico. Supporting collaborative deliberation using a largescale. Proceedings of Directions and Implications of Advanced Computing - Conference on Online Deliberation, 2008. [43] C. Lampe, N. Ellison, and C. Steinfield. A face (book) in the crowd: Social searching vs. social browsing. Proceedings of the 2006 20th anniversary conference on Computer supported cooperative work, pages 167–170, Jan 2006.

Sameer Ahuja

Bibliography

81

[44] P. Lange. Publicly private and privately public: Social networking on youtube. Journal of Computer-Mediated Communication, Jan 2008. [45] L. Layman, L. Williams, and K. Slaten. Note to self: make assignments meaningful. volume 39, pages 459–463, 2007. [46] K. Lerman.

Social networks and social information filtering on digg.

CoRR,

abs/cs/0612046, 2006. [47] K. Lerman. Social information processing in news aggregation. Internet Computing, IEEE, 11(6):16–28, 2007. [48] C. Marlow, M. Naaman, D. Boyd, and M. Davis. Ht06, tagging paper, taxonomy, flickr, academic article, to read. HYPERTEXT ’06: Proceedings of the seventeenth conference on Hypertext and hypermedia, Aug 2006. [49] S. Mitchell and W. Lutters. Assessing the value of computer science course material repositories. 19th Conference on Software Engineering Education and Training Workshops, CSEETW ’06, Jan 2006. [50] B. Nardi, D. Schiano, and M. Gumbrecht. Blogging as social activity, or, would you let 900 million people read your diary? Proceedings of the 2004 ACM conference on Computer supported cooperative work, Jan 2004. [51] B. Nardi, D. Schiano, M. Gumbrecht, and L. Swartz. Why we blog. Communications of the ACM, 47(12), Dec 2004. [52] O. Nov. What motivates wikipedians? Communications of the ACM, 50(11):60–64, 2007. [53] E. Orhun. Difficulties in integrating digital libraries into teaching: computer science teaching with the merlot collection. pages 311–316, 2006.

Sameer Ahuja

Bibliography

82

[54] L. Page, S. Brin, R. Motwani, and T. Winograd. The pagerank citation ranking: Bringing order to the web. 1999. [55] N. Parlante. Nifty assignments. pages 112–113, 2008. [56] N. Parlante, J. Cigas, A. B. Shiflet, R. Sooriamurthi, M. Clancy, B. Noonan, and D. Reed. Nifty assignments. volume 39, pages 497–498, 2007. [57] A. Portes. Social capital: its origins and applications in modern sociology. Annual review of sociology, Jan 1998. [58] B. Saulnier. “child is father to the man”: Social software in the is 2007 curriculum? Information Systems Education Journal, Jan 2007. [59] D. Schuler. Social computing. Communications of the ACM, Jan 1994. [60] L. A. Smith, E. T. Smith, and T. F. Melder. A taste of merlot: tutorial presentation. Journal of Computing Sciences in Colleges, 22(5):147–148, 2007. [61] M. Smith. Invisible crowds in cyberspace. Communities in cyberspace, Jan 1999. [62] T. Stafford, M. Stafford, and L. Schkade. Determining uses and gratifications for the internet. Decision Sciences, Jan 2004. [63] R. Stake. Standards-based & responsive evaluation. 2004. [64] R. Stoeckl, P. Rohrmeier, and T. Hess. Motivations to produce user generated content: differences between webloggers and videobloggers. Proceedings of the 20th Bled eConference: eMergence: Merging and Emerging Technologies, Processes, and Institutions., Jan 2007. [65] L. Story. Apologetic, facebook changes ad program. New York Times, Jan 2007. [66] C. Tauro, S. Ahuja, M. P´erez-Qui˜ nones, A. Kavanaugh, and P. Isenhour. Deliberation in the wild: a visualization tool for blog discovery and citizen-to-citizen participation. dg.o

Sameer Ahuja

Bibliography

83

’08: Proceedings of the 2008 international conference on Digital government research, May 2008. [67] M. Tepper. The rise of social software. netWorker, 7(3), Sep 2003. [68] M. Tungare, X. Yu, W. Cameron, G. Teng, M. A. P´erez-Qui˜ nones, L. Cassel, W. Fan, and E. A. Fox. Towards a syllabus repository for computer science courses. volume 39, pages 55–59, 2007. [69] S. Verba and N. Nie. Participation in america: Political democracy and social equality. Jan 1987. [70] F. Wang, K. Carley, D. Zeng, and W. Mao. Social computing: From social informatics to social intelligence. IEEE Intelligent Systems, Jan 2007. [71] Y. Wang and D. Fesenmaier. Assessing motivation of contribution in online communities: An empirical investigation of an online travel community. Electronic Markets, Jan 2003. [72] T. Weech. Merlot: Multimedia educational resource for learning and online teaching. Proceedings of the Informing Science + IT Education Conference, 2002. [73] B. Wellman, A. Quan-Haase, J. Boase, and W. Chen. The social affordances of the internet for networked individualism. Journal of Computer-Mediated Communication, Jan 2003. [74] C. Weston, L. McAlpine, and T. Bordonaro. A model for understanding formative evaluation in instructional design. Educational Technology Research and Development, Jan 1995.

Appendix A IRB Approvals

84

OfficeofofResearch Research Compliance Office Compliance InstitutionalReview Review Board Institutional Board

2000Pratt Kraft Drive, Suite 2000 (0497) 1880 Drive (0497) Blacksburg, Virginia 24061 Blacksburg, Virginia 24061 540/231-4991 Fax 540/231-0959 540/231-4991 Fax: 540/231-0959 [email protected] [email protected] E-mail: www.irb.vt.edu www.irb.vt.edu

DATE:

FWA00000572( expires 1/20/2010) IRB # is IRB00000667

June 23, 2009

MEMORANDUM TO:

Manuel A. Perez-Quinones Approval date: 6/23/2009 Continuing Review Due Date:6/8/2010 Expiration Date: 6/22/2010

FROM:

David M. Moore

SUBJECT:

IRB Expedited Approval: IRB # 09-567

“Evaluation of an Online System for Local Deliberation” ,

This memo is regarding the above-mentioned protocol. The proposed research is eligible for expedited review according to the specifications authorized by 45 CFR 46.110 and 21 CFR 56.110. As Chair of the Virginia Tech Institutional Review Board, I have granted approval to the study for a period of 12 months, effective June 23, 2009. As an investigator of human subjects, your responsibilities include the following: 1.

Report promptly proposed changes in previously approved human subject research activities to the IRB, including changes to your study forms, procedures and investigators, regardless of how minor. The proposed changes must not be initiated without IRB review and approval, except where necessary to eliminate apparent immediate hazards to the subjects. Report promptly to the IRB any injuries or other unanticipated or adverse events involving risks or harms to human research subjects or others. Report promptly to the IRB of the study’s closing (i.e., data collecting and data analysis complete at Virginia Tech). If the study is to continue past the expiration date (listed above), investigators must submit a request for continuing review prior to the continuing review due date (listed above). It is the researcher’s responsibility to obtain re-approval from the IRB before the study’s expiration date. If re-approval is not obtained (unless the study has been reported to the IRB as closed) prior to the expiration date, all activities involving human subjects and data analysis must cease immediately, except where necessary to eliminate apparent immediate hazards to the subjects.

2. 3.

4.

Important: If you are conducting federally funded non-exempt research, please send the applicable OSP/grant proposal to the IRB office, once available. OSP funds may not be released until the IRB has compared and found consistent the proposal and related IRB applicaton.

cc: File

Invent the F u t u r e V I R G I N I A

P O L Y T E C H N I C

VIRGINIA

I N S T I T U T E

U N I V E R S I T Y

POLYTECHNIC INSTITUTE An equal opportunity, affirmative

85

D S T A T E U N I V E R S I T Y AND A SNT A TE U N I V E R S I T Y action i n s t i t u t i o n

Office OfficeofofResearch ResearchCompliance Compliance Institutional InstitutionalReview ReviewBoard Board 1880 Drive (0497) 2000Pratt Kraft Drive, Suite 2000 (0497) Blacksburg, Virginia 24061 Blacksburg, Virginia 24061 540/231-4991 540/231-4991Fax: Fax540/231-0959 540/231-0959 E-mail: e-mail [email protected] [email protected] www.irb.vt.edu www.irb.vt.edu

DATE:

FWA00000572( expires 1/20/2010) IRB # is IRB00000667

July 8, 2009

MEMORANDUM TO:

Manuel A. Perez-Quinones Sameer Ahuja Stephen H. Edwards

FROM:

David M. Moore

SUBJECT:

IRB Amendment 1 Approval: Repository” , IRB # 09-577

Approval date: 6/24/2009 Continuing Review Due Date:6/9/2010 Expiration Date: 6/23/2010 “Evaluation of CATspace: A CS Assignment

This memo is regarding the above referenced protocol which was previously granted approval by the IRB on June 24, 2009. You subsequently requested permission to amend your IRB application. Since the requested amendment is nonsubstantive in nature, I, as Chair of the Virginia Tech Institutional Review Board, have granted approval for requested protocol amendment, effective as of July 8, 2009. The anniversary date will remain the same as the original approval date. As an investigator of human subjects, your responsibilities include the following: 1.

2. 3.

4.

Report promptly proposed changes in previously approved human subject research activities to the IRB, including changes to your study forms, procedures and investigators, regardless of how minor. The proposed changes must not be initiated without IRB review and approval, except where necessary to eliminate apparent immediate hazards to the subjects. Report promptly to the IRB any injuries or other unanticipated or adverse events involving risks or harms to human research subjects or others. Report promptly to the IRB of the study’s closing (i.e., data collecting and data analysis complete at Virginia Tech). If the study is to continue past the expiration date (listed above), investigators must submit a request for continuing review prior to the continuing review due date (listed above). It is the researcher’s responsibility to obtain re-approval from the IRB before the study’s expiration date. If re-approval is not obtained (unless the study has been reported to the IRB as closed) prior to the expiration date, all activities involving human subjects and data analysis must cease immediately, except where necessary to eliminate apparent immediate hazards to the subjects.

cc: File

Invent the F u t u r e V I R G I N I VAI R P Y T E CP O H LNYI TCE CI H N NSI T I T Y G IONLI A C I T U TI NES TUI TNUI TVEE R S AND

An equal opportunity,

affirmative

86

action

AS N U E N R I VS EI R T ADT ES T AUTNE I V T S YI T Y institution

Appendix B Informed Consent Forms

87

!"#$"%"&'()*+,-./%".'"%0,",1,-'&%2'0,&,-'1%"!-#0",+'' !!

"3456789'.53:83;'456'(<6;=>=?<3;:'=3'#8:8<6>@'(65A8>;:'"3B5CB=3D'/E7<3'0EFA8>;:'' !! !! "#$%&!'(!)*'+&,$-!./0%10$#'2!'(!02!'2%#2&!343$&5!('*!%',0%!6&%#7&*0$#'2! !! 82/&3$#90$'*:3;-!<*=!>021&%!?=!)@*&ABC1#D'2&3E!<*=!?26*&0!F0/02019GE!H05&&*!?G1+0=!! !! "G'(1#()0-')H',/"0'#-0-&#./I(#)J-.,'' !! "G#3!#3!02!&IJ%'*0$'*4!3$164!$'!&/0%10$&!0!,'5512#$4!K&7!0JJ%#,0$#'2!6&/&%'J&6!('*!31JJ'*$#29! %',0%!,'2/&*30$#'23E!,0%%&6!L'%%'M#=!"G&!J1*J'3&!'(!$G#3!*&3&0*,G!#3!$K'B('%6-!N2&!$'!J&*('*5!02! &/0%10$#'2!'(!$G&!1307#%#$4!026!3',#0%!0(('*602,&!'(!$G&!,'5512#$4!0JJ%#,0$#'2E!026!$K'E!$G&! 3$164!#3!J0*$!'(!0!%0*9&*!&(('*$!$'!$0M&!0!(#*3$!3$&J!$'K0*63!('*50%#A0$#'2!'(!J*',&61*&3!026!#331&3! #2!$G&!6&3#92!026!&/0%10$#'2!'(!,'5512#$4!3'($K0*&=! !! ""G'(#).-21#-0'' !! O#*3$E!$G&!J0*$#,#J02$!K'1%6!7&!9#/&2!0!3G'*$!)*&B./0%10$#'2!31*/&4!$G0$!K#%%!,'5J*#3&!'(! P1&3$#'23!'2!#2$&*2&$!1309&E!K&73#$&!1309&E!026!0,06&5#,!70,M9*'126=! ! H&,'26E!$G&!J0*$#,#J02$!K'1%6!7&!03M&6!$'!/#&K!0!3G'*$!/#6&'!6&5'!'(!'2&!'(!$G&!$K'!K&7! 0JJ%#,0$#'23=! ! "G#*6E!$G&!J0*$#,#J02$!K'1%6!7&!03M&6!$'!J&*('*5!0!3&$!'(!$03M3!'2!$G0$!0JJ%#,0$#'2=! ?!/#6&'!*&,'*6#29!'(!$G&!3,*&&2!K'1%6!7&!$0M&2!KG&2!$G&!J0*$#,#J02$!K'1%6!7&!J&*('*5#29!$G&! *&P1#*&6!$03M3=!Q'!016#'!K'1%6!7&!*&,'*6&6E!026!2'!*&,'*6#29!'(!$G&!J0*$#,#J02$!K'1%6!7&! $0M&2!KG#%&!J&*('*5#29!$G&!$03M3=! ! O'1*$GE!$G&!J0*$#,#J02$!K'1%6!7&!03M&6!$'!(#%%!0!J'3$B&/0%10$#'2!P1&3$#'220#*&!'2!$G&!0JJ%#,0$#'2! 703&6!'2!$G&!&/0%10$#'2!$03M3=! ! "G&!3$164!K#%%!7&!,'261,$&6!#2!R''5!2'=!SSTU!#2!F2'K%&69&!V'*M3!88!:<&J0*$5&2$!'(!L'5J1$&*! H,#&2,&;E!#2!$G&!L'*J'*0$&!R&3&0*,G!L&2$&*!0$!W#*9#2#0!"&,G=!"G&!066*&33!#3-!R''5!SSTUE!UUXU! F*0($!<*#/&E!Y%0,M371*9E!W?!UZX[S=!! ! "G&!3$164!K#%%!%03$!('*!2'!%'29&*!$G02!02!G'1*=! ! !! """G'#"0K0''

88

! "#$%&'&(#%&)*!&*!%+&,!$-,-#$'+!.&//!*)%!(/#'-!%+-!(#$%&'&(#*%!#%!0)$-!%+#*!0&*&0#/!$&,1!)2!(+3,&'#/! )$!(,3'+)/)4&'#/!+#$05!! !! !"#$%&'&(!)*$$ !! "#$%&'&(#%&)*!&*!%+&,!,%673!.&//!+-/(!&0($)8-!6*7-$,%#*7&*4!)2!9)'&#/!:22)$7#*'-;!#,!&%!#((/&-,!%)! &*%-$#'%&8-!#((/&'#%&)*,5!<%!.)6/7!#/,)!+-/(!($)8&7-!2)6*7#%&)*#/!.)$1!2)$!%+-!2)$0#%&)*!)2! 7-,&4*!($&*'&(/-,!#*7!-8#/6#%&)*!($)'-76$-,!)2!')006*&%3!,&%-,5!:77&%&)*#//3;!%+&,!-8#/6#%&)*! .)6/7!+-/(!6,!$-2&*-!=)//)1&;!#!/)'#/!')*8-$,#%&)*!%))/;!.+&'+!.-!&*%-*7!%)!$-/-#,-!.+-*!&%! $-#'+-,!#!0#%6$-!,%#%-;!#,!#!2$--!)(-*>,)6$'-!%))/!2)$!/)'#/!')006*&%&-,5! ! !! "#$&+)&')$,($-','./!).$-'0$1,'(!0&')!-2!).$$ !! ?+-!$-,6/%,!)2!%+&,!,%673!.&//!@-!1-(%!,%$&'%/3!')*2&7-*%&#/5!A)6$!.$&%%-*!')*,-*%!&,!$-B6&$-7!2)$! %+-!$-,-#$'+-$,!%)!$-/-#,-!#*3!7#%#!&7-*%&2&-7!.&%+!3)6!#,!#*!&*7&8&76#/!%)!#*3)*-!)%+-$!%+#*! (-$,)**-/!.)$1&*4!)*!%+-!($)C-'%5!!?+-!&*2)$0#%&)*!3)6!($)8&7-!.&//!+#8-!3)6$!*#0-!$-0)8-7! #*7!)*/3!#!,6@C-'%!*60@-$!.&//!&7-*%&23!3)6!76$&*4!#*#/3,-,!#*7!#*3!.$&%%-*!$-()$%,!)2!%+-! $-,-#$'+5!A)6$!(-$,)*#//3!&7-*%&2&#@/-!&*2)$0#%&)*!.&//!@-!1-(%!6*7-$!/)'1!#*7!1-3!&*!%+-!)22&'-! )2!D$5!E#*6-/!:5!"F$-G>H6&I)*-,;!J))0!K)5!LLMN;!O*)./-74-!P)$1,!<<5!!://!%+-!8&7-)!$-')$7&*4,! .&//!@-!7-,%$)3-7!)*'-!%+-3!+#8-!@--*!%$#*,'$&@-75!! !! D#%#!.&//!@-!,%)$-7!,-'6$-/3!#*7!.&//!@-!0#7-!#8#&/#@/-!)*/3!&*!%+-!')*%-Q%!)2!$-,-#$'+!! (6@/&'#%&)*,!#*7!7&,'6,,&)*5!K)!$-2-$-*'-!.&//!@-!0#7-!&*!)$#/!)$!.$&%%-*!$-()$%,!%+#%!')6/7!/&*1! 3)6!%)!%+-!7#%#!*)$!.&//!3)6!-8-$!@-!&7-*%&2&-7!#,!#!(#$%&'&(#*%!&*!%+-!($)C-'%5!://!7#%#!4#%+-$-7! .&//!+#8-!3)6$!*#0-!$-0)8-7!#*7!)*/3!#!6,-$!*60@-$!.&//!&7-*%&23!-#'+!6,-$!76$&*4!#*#/3,-,! #*7!#*3!.$&%%-*!$-()$%,!)2!%+-!$-,-#$'+5!! !! "!#$1,/3&'*-)!,'$$ !! R3!(#$%&'&(#%&*4!&*!%+&,!($)C-'%!3)6!.&//!@-!$-.#$7-7!#!')6()*!.)$%+!SN5N!#%!%+-!R#'1,%$--%,! J-,%#6$#*%;!/)'#%-7!#%!MTU!9!E#&*!9%;!R/#'1,@6$4;!V:!MWTXT5! !! "!!#$(4&&0,/$),$5!)604-5$$ !! A)6!#$-!2$--!%)!.&%+7$#.!2$)0!%+&,!,%673!#%!#*3!%&0-!2)$!#*3!$-#,)*!#*7!.&%+)6%!(-*#/%35!! !! "!!!#$-334,"-2$,($4&*&-416!! !! ?+&,!$-,-#$'+!+#,!@--*!#(($)8-7;!#,!$-B6&$-7;!@3!%+-!<*,%&%6%&)*#/!J-8&-.!R)#$7!2)$!($)C-'%,! &*8)/8&*4!+60#*!,6@C-'%,!#%!V&$4&*&#!")/3%-'+*&'!<*,%&%6%-!#*7!9%#%-!Y*&8-$,&%3;!#*7!@3!%+-! D-(#$%0-*%!)2!=)0(6%-$!9'&-*'-5!! !! 89

!"#$%&'(!)!%&*(+,$'-,%.*,!/!0!(!-,$! !! "!#$%&'()*+%,!)-*..!($!/)*(+0+/)(.!+'!(1+2!2(&3,4!)'3!"!5'$6!$7!'$!*.)2$'!"!0)''$(!/)*(+0+/)(.8!"!6+%%! 5../!(1.!)0(+#+(+.2!)'3!+'7$*9)(+$'!3+20&22.3!0$'7+3.'(+)%4!2+'0.!$(1.*2!6+%%!:.!/)*(+0+/)(+'-!+'! (1+2!*.2.)*018!! !! !! "#$%&'(!)!%&*(+,$%-'1!,,!.*$! !!! "!1)#.!*.)3!)'3!&'3.*2()'3!(1.!+'7$*9.3!0$'2.'(!)'3!0$'3+(+$'2!$7!(1+2!/*$;.0(8!!"!1)#.!1)3!)%%! 9,!<&.2(+$'2!)'26.*.38!!"!1.*.:,!)05'$6%.3-.!(1.!):$#.!)'3!-+#.!9,!#$%&'()*,!0$'2.'(!7$*! /)*(+0+/)(+$'!+'!(1+2!/*$;.0(8!!"7!"!/)*(+0+/)(.4!"!9),!6+(13*)6!)(!)',!(+9.!6+(1$&(!/.')%(,8!!"! )-*..!($!):+3.!:,!(1.!*&%.2!$7!(1+2!/*$;.0(8!! !! !! =====================================!!!!!!!!!!=========================! >+-')(&*.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!?)(.!! !! !! !! ==============================!! ! !!!!!!!!==========================! @)9.!A/%.)2.!/*+'(B!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!C$'()0(D!E1$'.FG9)+%!AH/(+$')%B!! !! !! >1$&%3!"!1)#.!)',!/.*(+'.'(!<&.2(+$'2!):$&(!(1+2!*.2.)*01!$*!+(2!0$'3&0(4!"!9),!0$'()0(D!! !! >)9..*!I1&;)!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!2)9..*8)1&;)J#(8.3&! "'#.2(+-)($*!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.K9)+%!! !! !! ?*8!L)'&.%!I8!EM*.NKO&+P$'.2!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!/.*.NJ#(8.3&!!! "'#.2(+-)($*4!Q)0&%(,!I3#+2$*!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.K9)+%!! !! !! ?*8!I'3*.)!R)#)')&-1!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!5)#)'J#(8.3&!!! "'#.2(+-)($*4!C$KC1)+*!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.K9)+%!! !! !! ?)#+3!L8!L$$*.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!STUKVWXKTYYX!F!9$$*.3J#(8.3&!!! C1)+*4!"Z[!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!\.%./1$'.F.K9)+%!!

90

!"#$"%"&'()*+,-./%".'"%0,",1,-'&%2'0,&,-'1%"!-#0",+'' !!

"3456789'.53:83;'456'(<6;=>=?<3;:'=3'#8:8<6>@'(65A8>;:'"3B5CB=3D'/E7<3'0EFA8>;:'' !! !! "#$%&!'(!)*'+&,$-!./0%10$#'2!'(!34"560,&-!4!37!455#829&2$!:&6'5#$'*;! !! <2/&5$#80$'*=5>-!?*@!A021&%!4@!)B*&CDE1#F'2&5G!?*@!7$&/&2!H@!.IJ0*I5G!709&&*!4K1+0@!! !! "G'(1#()0-')H',/"0'#-0-&#./I(#)J-.,'' !! "K#5!#5!02!&L6%'*0$'*;!5$1I;!$'!&/0%10$&!0!,'9912#$;!*&6'5#$'*;!'(!37!055#829&2$5G!,0%%&I! 34"560,&@!"K&!61*6'5&!'(!$K#5!*&5&0*,K!#5!$J'D('%I-!M2&!$'!6&*('*9!02!&/0%10$#'2!'(!$K&! 150N#%#$;!02I!5',#0%!0(('*I02,&!'(!34"560,&G!02I!$J'G!$K&!5$1I;!#5!60*$!'(!0!%0*8&*!&(('*$!$'!$0O&! 0!(#*5$!5$&6!$'J0*I5!('*90%#C0$#'2!'(!6*',&I1*&5!02I!#551&5!#2!$K&!I&5#82!02I!&/0%10$#'2!'(! ,'9912#$;!5'($J0*&@! !! ""G'(#).-21#-0'' !! P#*5$G!$K&!60*$#,#602$!J'1%I!N&!8#/&2!0!5K'*$!)*&D./0%10$#'2!51*/&;!$K0$!J#%%!,'96*#5&!'(! Q1&5$#'25!'2!#2$&*2&$!1508&G!02I!0,0I&9#,!N0,O8*'12I@! ! 7&,'2IG!$K&!60*$#,#602$!J'1%I!N&!05O&I!$'!/#&J!0!5K'*$!/#I&'!I&9'!'(!34"560,&@! ! "K#*IG!$K&!60*$#,#602$!J'1%I!N&!05O&I!$'!6&*('*9!0!5&$!'(!$05O5!'2!34"560,&@! 4!/#I&'!*&,'*I#28!'(!$K&!5,*&&2!J'1%I!N&!$0O&2!JK&2!$K&!60*$#,#602$!J'1%I!N&!6&*('*9#28!$K&! *&Q1#*&I!$05O5@!R'!01I#'!J'1%I!N&!*&,'*I&IG!02I!2'!*&,'*I#28!'(!$K&!60*$#,#602$!J'1%I!N&! $0O&2!JK#%&!6&*('*9#28!$K&!$05O5@! ! P'1*$KG!$K&!60*$#,#602$!J'1%I!N&!05O&I!$'!(#%%!0!6'5$D&/0%10$#'2!Q1&5$#'220#*&!'2!$K&!066%#,0$#'2! N05&I!'2!$K&!&/0%10$#'2!$05O5@! ! "K&!5$1I;!J#%%!N&!,'2I1,$&I!#2!:''9!2'@!SSTU!#2!V2'J%&I8&!W'*O5!
91

!! !"#$%&'&(!)*$$ !! "#$%&'&(#%&)*!&*!%+&,!,%-./!0&11!+21(!&3($)42!-*.2$,%#*.&*5!)6!7)'
!866)$.#*'29!#,!&%!#((1&2,!%)! &*%2$#'%&42!#((1&'#%&)*,:!;%!0)-1.!#1,)!+21(!($)4&.2!6)-*.#%&)*#1!0)$,(#'29!#!=)3(-%2$!7'&2*'2!#,,&5*32*%!$2(),&%)$/!#((1&'#%&)*9!0+&'+! 02!&*%2*.!%)!$212#,2!0+2*!&%!$2#'+2,!#!3#%-$2!,%#%29!#,!#!6$22!)(2*?,)-$'2!%))1!6)$!%+2! =)3(-%2$!7'&2*'2!&*,%$-'%)$,!')33-*&%/:! !! "#$&+)&')$,($-','./!).$-'0$1,'(!0&')!-2!).$$ !! >+2!$2,-1%,!)6!%+&,!,%-./!0&11!@2!<2(%!,%$&'%1/!')*6&.2*%
:!A)-$!0$&%%2*!')*,2*%!&,!$2B-&$2.!6)$! %+2!$2,2#$'+2$,!%)!$212#,2!#*/!.#%#!&.2*%&6&2.!0&%+!/)-!#,!#*!&*.&4&.-#1!%)!#*/)*2!)%+2$!%+#*! (2$,)**21!0)$<&*5!)*!%+2!($)C2'%:!!>+2!&*6)$3#%&)*!/)-!($)4&.2!0&11!+#42!/)-$!*#32!$23)42.! #*.!)*1/!#!,-@C2'%!*-3@2$!0&11!&.2*%&6/!/)-!.-$&*5!#*#1/,2,!#*.!#*/!0$&%%2*!$2()$%,!)6!%+2! $2,2#$'+:!A)-$!(2$,)*#11/!&.2*%&6&#@12!&*6)$3#%&)*!0&11!@2!<2(%!-*.2$!1)'+&,!$2,2#$'+!+#,!@22*!#(($)42.9!#,!$2B-&$2.9!@/!%+2!;*,%&%-%&)*#1!J24&20!R)#$.!6)$!($)C2'%,! &*4)14&*5!+-3#*!,-@C2'%,!#%!V&$5&*&#!")1/%2'+*&'!;*,%&%-%2!#*.!7%#%2!Y*&42$,&%/9!#*.!@/!%+2! D2(#$%32*%!)6!=)3(-%2$!7'&2*'2:!! !! !"#$%&'(!)!%&*(+,$'-,%.*,!/!0!(!-,$$ !!

92

!"#$%&'()*+%,")-*.."($"/)*(+0+/)(."+'"(1+2"2(&3,4")'3"!"5'$6"$7"'$"*.)2$'"!"0)''$("/)*(+0+/)(.8"!"6+%%" 5../"(1.")0(+#+(+.2")'3"+'7$*9)(+$'"3+20&22.3"0$'7+3.'(+)%4"2+'0."$(1.*2"6+%%":."/)*(+0+/)(+'-"+'" (1+2"*.2.)*018"" "" "" !"#$%&'()($%*'+,#$-&.(,,(/*#! """ !"1)#."*.)3")'3"&'3.*2()'3"(1."+'7$*9.3"0$'2.'(")'3"0$'3+(+$'2"$7"(1+2"/*$;.0(8""!"1)#."1)3")%%" 9,"<&.2(+$'2")'26.*.38""!"1.*.:,")05'$6%.3-."(1."):$#.")'3"-+#."9,"#$%&'()*,"0$'2.'("7$*" /)*(+0+/)(+$'"+'"(1+2"/*$;.0(8""!7"!"/)*(+0+/)(.4"!"9),"6+(13*)6")(")',"(+9."6+(1$&("/.')%(,8""!" )-*.."($"):+3.":,"(1."*&%.2"$7"(1+2"/*$;.0(8"" "" "" =====================================""""""""""=========================" >+-')(&*.""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""?)(."" "" "" "" =============================="" " """"""""==========================" @)9."A/%.)2."/*+'(B"""""""""""""""""""""""""""""""""""""""""""""""""""""""C$'()0(D"E1$'.FG9)+%"AH/(+$')%B"" "" "" >1$&%3"!"1)#.")',"/.*(+'.'("<&.2(+$'2"):$&("(1+2"*.2.)*01"$*"+(2"0$'3&0(4"!"9),"0$'()0(D"" "" >)9..*"I1&;)"""""""""""""""""""""""""""""""""""""""""""""""""""""""""2)9..*8)1&;)J#(8.3&" !'#.2(+-)($*"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""".K9)+%"" "" "" ?*8"L)'&.%"I8"EM*.NKO&+P$'.2""""""""""""""""""""""""""""""""/.*.NJ#(8.3&""" !'#.2(+-)($*4"Q)0&%(,"I3#+2$*"""""""""""""""""""""""""""""""""""""""".K9)+%"" "" "" ?*8">(.#.'"R8"G36)*32"""""""""""""""""""""""""""""""""""""""""""28.36)*32J#(8.3&" !'#.2(+-)($*4"C$KC1)+*""""""""""""""""""""""""""""""""""""""""""""""""""".K9)+%"" "" "" ?)#+3"L8"L$$*."""""""""""""""""""""""""""""""""""""""""""""""STUKVWXKTYYX"F"9$$*.3J#(8.3&""" C1)+*4"!Z[""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""\.%./1$'.F.K9)+%"" "" "

93

Appendix C Pre-Evaluation Survey

94

!"#$#%&'(&)*+,-.(#/)*+,,&*"#01-234-35678-93-:3;-;<7-6=7-53>>34?8@-47A-BCC>?DB6?38<E-!>7B<7-B99-B8:-36=7F-47A-BCC>?DB6?38<-B<-8779791!"#

%$!$&$'()% #"*$+%(%,(-%

.#%)$(+#% /01$%(%,(-%

!$&$'()% #"*$+%(% 2$$3%

401$%(%2$$3%

5('$)-%

6$&$'%

GBD7A33H1D3I-

-

-

-

-

-

-

)4?667F1D3I-

-

-

-

-

-

-

J:
-

-

-

-

-

-

+FH;61D3I-

-

-

-

-

-

-

G>?DHF1D3I-

-

-

-

-

-

-

K3;);A71D3I-

-

-

-

-

-

-

L7>?D?3;<1D3I-

-

-

-

-

-

-

L?@@1D3I-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

!"#$%&'''( )*+$,&$(,--(.$+/01(

-

95

!"#$%&'#()*+#,-#&.')/)')01#%&/0#2,3#405-,56!"#$%#&'!(!#)!*#+,,-./+&.$%(#01./2#+--#&'+&#+,,-345#67$%&!%&8#.%#&'.(# 7301'),*#.&*#60&*#&88#()*+1#,-#.,*'0*'9#8)(0#4%,',19#/)+0,19#631).9#:8,;#4,1'19#*0<1#&5').8019#0'."#=++#6,50# <0:1)'01#&1#*00+0+"# # !"#

%$!&'(")*% +,)#$)#%-"#&% ,#&$(%.$,./

%$0./,'1")*% 2,3(%,-)% +,)#$)#%

4,55$)#")*%,)% +,)#$)#%,(% .(,6"/$7%

8,#")*9%('#")*%,(% !"#$#%&'()*%+,%+%

:(,-7")*% *+-,./'( .(,6"/$7%

:(,-7")*% *+-,./'( +,)#$)#%

>&.0:,,(".,6#

#

#

#

#

#

#

?<)''05".,6#

#

#

#

#

#

#

@214&.0".,6#

#

#

#

#

#

#

A5(3'".,6#

#

#

#

#

#

#

>8).(5".,6#

#

#

#

#

#

#

B,3?3:0".,6#

#

#

#

#

#

#

C08).),31".,6#

#

#

#

#

#

#

C);;".,6#

#

#

#

#

#

#

#

#

#

#

#

#

#

#

#

#

#

#

#

#

#

#

#

#

#

#

#

#

#

#

#

#

#

#

#

#

#

#

#

#

#

#

#

#

#

#

#

#

#

#

#

#

#

#

#

#

#

#

#

#

#

#

#

#

#

#

#

#

#

!"#$%&'''( )*+$,&$(,--(.$+/01(

# % % % 96

!"#$%&'()*+,-./$&#'0*$., ! ! "#!$%&'!(((((((((!)*+#!! ! ! ,#!-&.'!((((((((((! ! ! /#!0112345678'!! ! !!!!((((!-529&85! !!!!((((!05:&*'!(((((((((((((((((((! ! !!!;63!5:&!<7??7@68%!A2&+5678#! ! ! B#!$*&!)72!4! ! !!!(((!C41:&?7*+!+529&85! !!!(((!D4+5&*+!+529&85! !!!(((!E:F!+529&85! !

97

Appendix D Colloki Tasks

98

!"#$#% ! ! "#$!%&'!(&#)*+'+!,*-.!%!,'/!/%0'+!%((1*2%-*#3!4#&!.#1+*35!2#3)'&0%-*#30!%3+!0.%&*35!1*360!%7#350-!7'7/'&0! #4!-.'!5&%+$%-'!0-$+'3-!2#77$3*-8!%-!-.'!9#7($-'&!:2*'32'!+'(%&-7'3-!#4!;*&5*3*%!<'2.=!<.'!%((1*2%-*#3!*0! !"##$%&'(#)*+,-&.#$"($&/"0!1&01$&(12&-!+'7#!)*+'#!#4!-.'!%((1*2%-*#3!/8!21*26*35!-.'!+'06-#(!0.#&-2$-!-*-1'+! '(#)*+&%$32,-&45!$&62)&1"7$&/"0!1$%&01$&78%$29&+#$"($&!#8!:&25&01$&%$(:02+&(12*0!)0&#";$##$%&<(#)*+<&02&2+$5& -.'!,'/0*-'>!%3+!('&4#&7!-.'!4#11#,*35!-%060!#3!-.'!0*-'?! ! ! @= A0'!-.'!4#11#,*35!2&'+'3-*%10!-#!1#5!#3!-#!-.'!0*-'!B! ! A0'&3%7'?!(%&-*2*(%3-! C%00,#&+?!(%&-*2*(%3-! ! ! D= =*2/($&01*2)>1&01$(0&2?U:(&"5%&+2(0(&85&01$&02+8!&@A2!"#&B$(0")*"50(C-&D(&01$*$&"&*$(0")*"50&62)&:52/&85& -.'!%&'%!-.%-!*0!7*00*35E!F++!%!1*36!-#!-.'!,'/0*-'!#4!-.%-!&'0-%$&%3-!-#!-.'!-#(*2!%1#35!,*-.!0#7'! +'02&*(-*#3!%3+!-%50=!G"#$!2%3!$0'!,'/!0'%&2.!#&!#-.'&!-##10!-#!1##6$(!1#2%1!&'0-%$&%3-0!%3+H#&!-.'*&! ,'/0*-'0=I! ! ! J= E2&02&01$&@E*"%)"0$&A8?$C&02+8!9&"5%&78$/&01$&&'#(%)*+*,(%-.,$#!*3!-.%-!-#(*2=!;#-'!4#&!-.'!1*36!-.%-!8#$!1*6'! -.'!7#0-=!G"#$!2%3!21*26!-.&#$5.!#3!-.'!1*360!%3+!)*0*-!-.'!0*-'0!-#!+'2*+'!8#$&!4%)#&*-'!1*36I=!K#,>!1'%)'!%! 2#77'3-!#3!-.'!1*36!7'3-*#3*35!,.8!8#$!1*6'!*-=! ! ! L= M.*2.!*0!-.'!'-/*#(%0'#(!85&01$&02+8!&@E*"%)"0$&A8?$CF&G*80$&62)*&"5(/$*&;$#2/-! ! NNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNN! ! ! !"#$%&'$("))"*+,-$./'0%+",01$200/3'$%&2%$4"/$&25'$6/0%$6"+,'7$%&'$8"39/%'#$:;+',;'$7'92#%3',%$2%$<+#-+,+2$ !"#$%&'%&%()&*+&,"%',+*"-,%&-*%&)"%.-,")"',"*%.-%/"&)-.-(%01)"%&21+,%,$"%*"3&),0"-,4'%',+*"-,%#100+-.,56! ! ! O= M.*2.!#3'!#4!-.'!4*)'!-#(*20!1*0-'+!#3!-.'!.#7'(%5'!+#!8#$!-.*36!*0!-.'!7#0-!(#($1%&!%7#350-!-.'!01$&(! 2#77$3*-8E! ! NNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNN! ! ! ! 99

!" #$%&$'()*'(+',$*'+%-*',(.%&/'0%/,*1'()',$*'$(2*.34*'1('5(6',$%)7'$3/'8**)',$*'2(/,'3&,%-*'9*&*),05' 32()4/,',$*'/069.'&(226)%,5:' ' ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;' ' ' <" =9(2',$*'1*.39,2*),'>*8/%,*?'5(6'7)(>',$3,'@32**9'A$6B3'>(97/'%)',$*'/32*'9*/*39&$'49(6.',$3,'5(6'39*' !"#$%$&#$'(!")(*!"'(+!,(!"(-./%0)(1"2$(34/56$(74/"'(+!,8(7!"'(4/#(#94(0$40.$',$3,'$*'$3/'$31',$*'2(/,' %),*93&,%()/'>%,$'C-%3'&(22*),/D'()'/069."'E),*9'5(69'3)/>*9/'8*0(>"' ' -:,$$%(;+/<:5&(/&$%(!'(4"(-./%0=(>>>>>>>>>>>>>>>(' ' F/*9'%1/'(+',>('6/*9/'$*'$3/'$31'2(/,'%),*93&,%()/'>%,$'C-%3'&(22*),/DG';;;;;;;;;;;;;;;;;;' ' ' H" I3)6*0'A"'J*9*K'L6%)()*/'%/',$*'.9(+*//(9'>$('$*31/',$*'9*/*39&$'49(6.',$3,'5(6'39*'%),*9*/,*1'%)"'M%/' /&$%(!'(4"(-./%0(!&(?,:0@A)(B!&(#40(#:C(4"(-./%0(%$0%$&$"#&(+!&(0%!,:%3(%$&$:%2+(:%$:=(?DEFA)(G:,$((4"$( ,4%$(/&$%(!"(#+$(&3&#$,(9+4&$(#40(#:C(!&(?DEFA)' ' ;;;;;;;;;;;;;;;;;;;;;' ' '

100

Appendix E CATspace Tasks

101

!"#$%&'()#"*+*) ! ! "#$%&'()!*%!'!+'(),--.!'&&/*('0*-1!02'0!/)0%!"3!*1%045(0-4%!&-%06!%2'4)6!(-77)10!-16!'18!4'0)!"3!'%%*917)10%:! !"#$%#&'$()*&(*#&%*+,(&-#.+&/0-#+&+1&(*#&$22"0)$(0+3&45&)"0)6037&(*#&-#%6(+2&%*+,()8(&(0("#-&9:;<%2$)#&-#.+=>& ;1()!<-5!2'=)!>'0(2)8!02)!=*8)-6!<-5!>*//!,)!&4-=*8)8!'!,4->%)4!>*18->!>*02!02)!'&&/*('0*-1!-&)1!*1!*0:! ! +-4!02)!?-//->*19!0'%.%6!<-5!>*//!,)!/-99)8!*10-!02)!'&&/*('0*-1!'%!'!?*(0*-1'/!&4-?)%%-4!*1!"-7&50)4!3(*)1()6! >2-%)!0)'(2)%!02)!*104-85(0-4*19!0'%.%!-1!02)!%*0)B! ! C: ?+&(+&5+8,&:;<%2$)#&2,#1#,#3)#%&2$7#&$3-&$--&(*#&($7%&9:@A=&$3-&9$,,$5%=&(+&5+8,&"0%(&+1&($7&%84%),02(0+3%>! ! ! D: @-5!'4)!/--.*19!?-4!'1!'%%*917)10!?-4!<-54!5&(-7*19!(/'%%!-1!0>-!8*7)1%*-1'/!'44'<%:!+*18!'1!'%%*917)10! (*$(B%&($77#-&4+(*&9:@A=&$3-&9C-&$,,$5=>&D,0(#&(*#&$%%073.#3(B%&(0("#&4#"+'>&! ! EEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEE! ! ! F: +*18!>2*(2!-?!<-54!+'(),--.!?4*)18%!'4)!'/4)'8:! ! EEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEE! ! ! H: G'7)!0>-!-02)4!&)-&/)!02'0!02)!&)4%-1!*1!I5)%0*-1!F!2'%!'%!(-J'502-4%!-1!2*%K2)4!'%%*917)10%:! ! EEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEE! ! ! L: 9@(#2*#3&E-'$,-%=&0%&$&/#,5&'#""&63+'3&:@&03%(,8)(+,!>-4.*19!*1!<-54!'4)'!-?!0)'(2*19!>2-!2'8!)'4/*)4! 7)10*-1)8!0-!<-5!02'0!2)!*%!'/4)'8!7'1
102

!" !"#$%&'(#)*+,-".#./0#.+1#.)-"#2+3#&.01/0,4#56,7#),+./03#8/+#/)"#$%&'(#)"#+,0#+2#/6"9/03#.+1#.)-":# # $$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$# # # %" &'(#)*+,)*#-'#+'.-/,0(-*#-'#-1*#234567+*#+'88(.,-9#09#7)),.:#'.*#';#9'(/#'<.#755,:.8*.-5"#2/*7-*#7#.*<# )""6-,*0,.#86./#./0#.6.;0#<&6,-;0#76*0,"6+,);#)33)="(#),7#.)-#6.#86./#.)-"#$%&'(#),7#$'7#)33)="(:#>1;+)7#./0# 755,:.8*.-"=,6#;6;0#)?)6;)@;0#+,#./0#70"A.+1#.+#./6"#)""6-,*0,.B"#26;0:# # # >" 3;-*/#87?,.:#-1*5*#+17.:*5@#9'(#/*7A,=*#-17-#-1*#755,:.8*.-#7A5'#175#7#B(*5-,'.#'.#A,.?*)#A,5-5"#C),-#-1*# )""6-,*0,.B"#.6.;0#.+#$&6,-;0#76*0,"6+,);#)33)="#),7#;6,A07#;6"."(C#),7#)77#$;6,A07#;6"."(#.'#-1*#A,5-#';#-7:5"# # # D" D+EB30#,+8#26,6"/6,-#E1#./0#)""6-,*0,.:#F0.B"#*)A0#)#26,);#"0.#+2#G/),-0":#&0.#./0#*0.)7).)#13+103.=#G);;07# $%+E3"0(#2+3#./6"#)""6-,*0,.#.+#$%&'(:#H;"+C#E17).0#./0#30)7*0:.I.#26;0#6,#./0#)""6-,*0,."B#;,A*5#E#+17.:*#-1*# ;,/5-#A,.*#';#-/0#26;0#23+*#$H33)="(#.+#$H33)="#),7#F6,A07#F6"."(:# # FG" H(0A,51#-1*#755,:.8*.-"# # # # # # # #

103

Appendix F Colloki Post-Evaluation Questionnaire

104

!"#$%&'()*($+",-.*/#$+",,(+0/-

1$0",2)34+#(20//-

4+#(20//-

5/*$0()-

620//-

1$0",2)3620//-

-

-

-

-

-

-

78-1)*09-:(#-/(#3-$"-*#/-

7-

;-

<-

=-

>-

-

-

-

-

-

-

;8-?-@"*,A-+$-/(#3-$"-B0/($/-9"#$#-"0-)+,C#-",1)*09-

7-

;-

<-

=-

>-

-

-

-

-

-

-

<8-?-@"*,A-+$-/(#3-$"-@+,A-$D/-)($/#$-9"#$#-(,A)+,C#-+,-1)*09-

7-

;-

<-

=-

>-

-

-

-

-

-

-

=8-?-@"*,A-+$-/(#3-$"-@+,A-$D/-E"#$-9"9*)(09"#$#-(,A-)+,C#-+,-1)*09-

7-

;-

<-

=-

>-

-

-

-

-

-

-

>8-?-@"*,A-+$-/(#3-$"-@+,A-*#/0#-:D"-D('/#+E+)(0-$"9-$(2#-

7-

;-

<-

=-

>-

-

-

-

-

-

-

F8-?-@"*,A-+$-/(#3-$"-@+,A-:D+BD-*#/0#-D('/B"EE/,$/!"#$"%&'("#)(%*+,"-#,),"&$!"./$0,-

7-

;-

<-

=-

>-

-

-

-

-

-

-

G8-?-@"*,A-+$-/(#3-$"-@+,A-$D/-$"9-B",$0+H*$"0#@"0-9(0$+B*)(0-$"9+B#-

7-

;-

<-

=-

>-

-

-

-

-

-

-

!"#

%$-

-

-

-

-

I8-?-@"*,A-+$-/(#3-$"-$(2-E3-9"#$#-(,A-)+,C#-

7-

;-

<-

=-

>-

-

-

-

-

-

-

105

!

"#$%&'()! *+,-'$..!

*+,-'$..!

/.0#$-(!

1'$..!

"#$%&'()! 1'$..!

23!4!5%0&6!+#!.-,)!#%!7$%8,.!9%,#,!-&6!(+&:,! 0,+&'!#-',!

;!


=!

>!

?!

!

!

!

!

!

!

;@3!A-',!8+((!7.!B.(950(!5%$!C.!#%!5+&6!9%,#,! -&6!(+&:,!$.(.D-&#!#%!C)!+&#.$.,#!

;!


=!

>!

?!

!

!

!

!

!

!

;;3!A-',!8+((!7.!B.(950(!5%$!C.!#%!5+&6!9.%9(.! 8+#B!,+C+(-$!+&#.$.,#,!-,!C.!

;!


=!

>!

?!

!

!

!

!

!

!

;<3!A-',!8+((!7.!B.(950(!5%$!C.!#%!5+&6!C%$.! -7%0#!9.%9(.!4!-($.-6)!:&%8!

;!


=!

>!

?!

!

!

!

!

!

!

;=3!A-',!8+((!7.!B.(950(!5%$!C.!#%!+&#.$-E#! 8+#B!%#B.$!9.%9(.!%&!"(0$9!

;!


=!

>!

?!

!

!

!

!

!

!

;>3!A-',!8+((!7.!B.(950(!+&!,B-$+&'!C)!9%,#,! -&6!(+&:,!8+#B!%#B.$!0,.$,!%5!"(0$9!

;!


=!

>!

?!

!

!

!

!

!

!

;?3!A-',!8+((!7.!B.(950(!+&!8%$:+&'!8+#B!0,.$,! %&!"(0$9!#%!$.-EB!,%C.!E%CC%&!'%-(!!

;!


=!

>!

?!

!

!

!

!

!

!

!"#"$%&'#%()(%*!!

!

!

!

!

!

;F3!4!5%0&6!$.E.&#!-E#+D+#)!(+,#!+�+#+D.!-&6! .-,)!#%!0,.!

;!


=!

>!

?!

!

!

!

!

!

!

;G3!H.E.&#!1E#+D+#)!8+((!7.!B.(950(!5%$!C.!#%! 5+&6!9%,#,!-&6!(+&:,!$.(.D-&#!#%!C)!+&#.$.,#!

;!


=!

>!

?!

!

!

!

!

!

!

106

!

"#$%&'()! *+,-'$..!

*+,-'$..!

/.0#$-(!

1'$..!

"#$%&'()! 1'$..!

234!5.6.&#!16#+7+#)!8+((!9.!:.(;<0(!<%$!=.!#%! <+&>!;.%;(.!8+#:!,+=+(-$!+&#.$.,#,!-,!=.!

2!

?!

@!

A!

B!

!

!

!

!

!

!

2C4!5.6.&#!16#+7+#)!8+((!9.!:.(;<0(!<%$!=.!#%! <+&>!=%$.!-9%0#!;.%;(.!D!-($.->)!E&%8!

2!

?!

@!

A!

B!

!

!

!

!

!

!

?F4!5.6.&#!16#+7+#)!8+((!9.!:.(;<0(!<%$!=.!#%! +&#.$-6#!8+#:!%#:.$!;.%;(.!%&!"(0$;!

2!

?!

@!

A!

B!

!

!

!

!

!

!

?24!5.6.&#!16#+7+#)!8+((!9.!:.(;<0(!+&!,:-$+&'! =)!;%,#,!-&>!(+&E,!8+#:!%#:.$!0,.$,!%
2!

?!

@!

A!

B!

!

!

!

!

!

!

??4!5.6.&#!16#+7+#)!8+((!9.!:.(;<0(!+&!8%$E+&'! 8+#:!0,.$,!%&!"(0$;!#%!$.-6:!,%=.!6%==%&! '%-(!!

2!

?!

@!

A!

B!

!

!

!

!

!

!

!"##$%&'(

!

!

!

!

!

?@4!D!<%0&>!+#!.-,)!#%!=-E.!6%==.&#,!

2!

?!

@!

A!

B!

!

!

!

!

!

!

?@4!G%==.&#,!8+((!9.!:.(;<0(!<%$!=.!#%!<+&>! ;%,#,!-&>!(+&E,!$.(.7-&#!#%!=)!+&#.$.,#!

2!

?!

@!

A!

B!

!

!

!

!

!

!

?A4!G%==.&#,!8+((!9.!:.(;<0(!<%$!=.!#%!<+&>! ;.%;(.!8+#:!,+=+(-$!+&#.$.,#,!-,!=.!

2!

?!

@!

A!

B!

!

!

!

!

!

!

?B4!G%==.&#,!8+((!9.!:.(;<0(!<%$!=.!#%!<+&>! =%$.!-9%0#!;.%;(.!D!-($.->)!E&%8!

2!

?!

@!

A!

B!

!

!

!

!

!

!

107

!

"#$%&'()! *+,-'$..!

*+,-'$..!

/.0#$-(!

1'$..!

"#$%&'()! 1'$..!

234!5%66.&#,!7+((!8.!9.(:;0(!;%$!6.!#%! +&#.$-<#!7+#9!%#9.$!:.%:(.!%&!"(0$:!

=!

2!

>!

?!

@!

!

!

!

!

!

!

2A4!5%66.&#,!7+((!8.!9.(:;0(!+&!,9-$+&'!6)! :%,#,!-&B!(+&C,!7+#9!%#9.$!0,.$,!%;!"(0$:!

=!

2!

>!

?!

@!

!

!

!

!

!

!

2D4!5%66.&#,!7+((!8.!9.(:;0(!+&!7%$C+&'!7+#9! 0,.$,!%&!"(0$:!#%!$.-<9!,%6.!<%66%&!'%-(!!

=!

2!

>!

?!

@!

!

!

!

!

!

!

!"#$%&

!

!

!

!

!

2E4!F!;%0&B!+#!.-,)!#%!G%#.!;%$!:%,#,!-&B!(+&C,!

=!

2!

>!

?!

@!

!

!

!

!

!

!

>H4!I%#.,!7+((!8.!9.(:;0(!;%$!6.!#%!;+&B!:%,#,! -&B!(+&C,!$.(.G-&#!#%!6)!+&#.$.,#!

=!

2!

>!

?!

@!

!

!

!

!

!

!

>=4!I%#.,!7+((!8.!9.(:;0(!;%$!6.!#%!;+&B!:.%:(.! 7+#9!,+6+(-$!+&#.$.,#,!-,!6.!

=!

2!

>!

?!

@!

!

!

!

!

!

!

>24!I%#.,!7+((!8.!9.(:;0(!;%$!6.!#%!;+&B!6%$.! -8%0#!:.%:(.!F!-($.-B)!C&%7!

=!

2!

>!

?!

@!

!

!

!

!

!

!

>>4!I%#.,!7+((!8.!9.(:;0(!;%$!6.!#%!+&#.$-<#! 7+#9!%#9.$!:.%:(.!%&!"(0$:!

=!

2!

>!

?!

@!

!

!

!

!

!

!

>?4!I%#.,!7+((!8.!9.(:;0(!+&!,9-$+&'!6)!:%,#,! -&B!(+&C,!7+#9!%#9.$!0,.$,!%;!"(0$:!

=!

2!

>!

?!

@!

!

!

!

!

!

!

108

!

"#$%&'()! *+,-'$..!

*+,-'$..!

/.0#$-(!

1'$..!

"#$%&'()! 1'$..!

234!5%#.,!6+((!7.!8.(9:0(!+&!6%$;+&'!6+#8! 0,.$,!%&!"(0$9!#%!$.-<8!,%=.!<%==%&!'%-(!!

>!

?!

2!

@!

3!

!

!

!

!

!

!

!"#$%&$'()*#"%

!

!

!

!

!

2A4!B!:%0&C!0,.$!9$%:+(.,!.-,)!#%!0&C.$,#-&C!

>!

?!

2!

@!

3!

!

!

!

!

!

!

2D4!E,.$!F$%:+(.,!6+((!7.!8.(9:0(!:%$!=.!#%!:+&C! 9%,#,!-&C!(+&;,!$.(.G-&#!#%!=)!+&#.$.,#!

>!

?!

2!

@!

3!

!

!

!

!

!

!

2H4!E,.$!F$%:+(.,!6+((!7.!8.(9:0(!:%$!=.!#%!:+&C! 9.%9(.!6+#8!,+=+(-$!+&#.$.,#,!-,!=.!

>!

?!

2!

@!

3!

!

!

!

!

!

!

2I4!E,.$!F$%:+(.,!6+((!7.!8.(9:0(!:%$!=.!#%!:+&C! =%$.!-7%0#!9.%9(.!B!-($.-C)!;&%6!

>!

?!

2!

@!

3!

!

!

!

!

!

!

@J4!E,.$!F$%:+(.,!6+((!7.!8.(9:0(!:%$!=.!#%! +&#.$-<#!6+#8!%#8.$!9.%9(.!%&!"(0$9!

>!

?!

2!

@!

3!

!

!

!

!

!

!

@>4!E,.$!F$%:+(.,!6+((!7.!8.(9:0(!+&!,8-$+&'!=)! 9%,#,!-&C!(+&;,!6+#8!%#8.$!0,.$,!%:!"(0$9!

>!

?!

2!

@!

3!

!

!

!

!

!

!

@?4!E,.$!F$%:+(.,!6+((!7.!8.(9:0(!+&!6%$;+&'! 6+#8!0,.$,!%&!"(0$9!#%!$.-<8!,%=.!<%==%&! '%-(!!

>!

?!

2!

@!

3!

! ! ! ! ! ! 109

! !"#$%&'()'$*'+,-.+$(+/$.,0'1$2'+'3-,)$.3$3'(,41')$.3$5&416$2'&.7#$8.4$9(+$(::$3'(,41')$,0(,$/.4$:.+;,$)''$-+$ "#$!%&'"(! ! !

!"#$%&"'

(")"*+$,'

)*++&,+!

!

-."&/&"0!1"2$*3'!

!

4533$,"&,+!

!

65"&,+!

!

7'$2!8259&%$'!

!

!"#$%!"#$%&'(&"'))"*&%+,-& !

!

!

!

!

!

! ! ::(!-2$!05;!.5,.$2,$*0'!"#*"!;'$2'!3&+#"!*=;'$!"#$!9$*";2$'!?25/&<$(! ! !

!"#$%&"'

-.//")$,'

)*++&,+!

!

-."&/&"0!1"2$*3'!

!

4533$,"&,+!

!

65"&,+!

!

7'$2!8259&%$'!

!

./0&1!"#$%&'(&"'))"*&%+,-& !

!

!

! 110

!"#$%&'()'$(**$(+,$-./'0$1-22'+.)$-+$3&405$/'0'#$6789'$108.8:4'$-;$;'(.40')<$(**8.8-+(&$;'(.40')$./(.$,-4$./8+9$ =8&&$>'$4)';4&<$(+*$)-$-+#?$$ $

111

Appendix G CATspace Post-Evaluation Questionnaire

112

!"#$%&'()*($+",-.*/#$+",,(+0/! ! !

"#$%&'()! *+,-'$..!

*+,-'$..!

/.0#$-(!

1'$..!

"#$%&'()! 1'$..!

!

!

!

!

!

!

23!415,6-7.!8-,!.-,)!#%!0,.!

2!

9!

:!

;!


!

!

!

!

!

!

93!=!>%0&?!+#!.-,)!#%!7$.-#.!-&!-,,+'&@.&#!%&! 415,6-7.!

2!

9!

:!

;!


!

!

!

!

!

!

:3!=!>%0&?!+#!.-,)!#%!@-&+60(-#.!?+>>.$.&#! -##$+A0#.,!%>!-&!-,,+'&@.&#!%&!415,6-7.!

2!

9!

:!

;!


!

!

!

!

!

!

;3!=!>%0&?!+#!.-,)!#%!-??!-&?!.?+#!>+(.,!%>! -,,+'&@.&#,!%&!415,6-7.!

2!

9!

:!

;!


!

!

!

!

!

!

<3!=!>%0&?!+#!.-,)!#%!+?.&#+>)!#B.!#%6!#-',!>%$! 6.%6(.!

2!

9!

:!

;!


!

!

!

!

!

!

C3!=!>%0&?!+#!.-,)!#%!>+&?!#B.!(-#.,#!-7#+D+#)!+&! 415,6-7.!

2!

9!

:!

;!


!

!

!

!

!

!

E3!=!>%0&?!+#!.-,)!#%!>+&?!6.%6(.!#B-#!B-D.! ,+@+(-$!#-',!

2!

9!

:!

;!


!

!

!

!

!

!

F3 =!>%0&?!+#!.-,)!#%!>+&?!6.%6(.!8B%!-$.!7%G -0#B%$,!%&!-,,+'&@.&#,!8+#B+&!415,6-7.!

2!

9!

:!

;!


!

!

!

!

!

!

113

!

"#$%&'()! *+,-'$..!

*+,-'$..!

/.0#$-(!

1'$..!

"#$%&'()! 1'$..!

23!4!5%0&6!+#!.-,)!#%!5+&6!7)!8-9.:%%;!5$+.&6,! %&!<1=,>-9.!

?!

@!

A!

B!

C!

!

!

!

!

!

!

!"#

%$!

!

!

!

!

?D3!4!5%0&6!+#!.-,)!#%!#-'!7)!-,,+'&7.&#,!

?!

@!

A!

B!

C!

!

!

!

!

!

!

??3!4!5%0&6!+#!.-,)!#%!:$%E,.!-,,+'&7.&#,! 0,+&'!#-',!

?!

@!

A!

B!

C!

!

!

!

!

!

!

?@3!=-',!E+((!:.!F.(>50(!5%$!7.!#%!5+&6! -,,+'&7.&#,!$.(.G-&#!#%!7)!+&#.$.,#!

?!

@!

A!

B!

C!

!

!

!

!

!

!

?A3!=-',!E+((!:.!F.(>50(!5%$!7.!#%!5+&6!>.%>(.! E+#F!,+7+(-$!+&#.$.,#,!-,!7.!

?!

@!

A!

B!

C!

!

!

!

!

!

!

?B3!=-',!E+((!:.!F.(>50(!5%$!7.!#%!5+&6!7%$.! -:%0#!>.%>(.!4!-($.-6)!;&%E!

?!

@!

A!

B!

C!

!

!

!

!

!

!

?C3!=-',!E+((!:.!F.(>50(!5%$!7.!#%!+&#.$-9#! E+#F!%#F.$!>.%>(.!%&!<1=,>-9.!

?!

@!

A!

B!

C!

!

!

!

!

!

!

?H3!=-',!E+((!:.!F.(>50(!+&!,F-$+&'!7)! -,,+'&7.&#,!E+#F!%#F.$!0,.$,!%5!<1=,>-9.!

?!

@!

A!

B!

C!

!

!

!

!

!

!

?I3!=-',!E+((!:.!F.(>50(!+&!E%$;+&'!E+#F!0,.$,! %&!<1=,>-9.!#%!$.-9F!,%7.!9%77%&!'%-(!!

?!

@!

A!

B!

C!

!

!

!

!

!

!

114

!

"#$%&'()! *+,-'$..!

*+,-'$..!

/.0#$-(!

1'$..!

"#$%&'()! 1'$..!

!"#$%#&'(#)*)#+!!

!

!

!

!

!

234!5!6%0&7!(-#.,#!-8#+9+#)!(+,#!+�+#+9.!-&7! .-,)!#%!0,.!

2!

:!

;!


=!

!

!

!

!

!

!

2>4!?-#.,#!18#+9+#)!@+((!A.!B.(C60(!6%$!D.!#%! 6+&7!-,,+'&D.&#,!$.(.9-&#!#%!D)!+&#.$.,#!

2!

:!

;!


=!

!

!

!

!

!

!

:E4!?-#.,#!18#+9+#)!@+((!A.!B.(C60(!6%$!D.!#%! 6+&7!C.%C(.!@+#B!,+D+(-$!+&#.$.,#,!-,!D.!

2!

:!

;!


=!

!

!

!

!

!

!

:24!?-#.,#!18#+9+#)!@+((!A.!B.(C60(!6%$!D.!#%! 6+&7!D%$.!-A%0#!C.%C(.!5!-($.-7)!F&%@!

2!

:!

;!


=!

!

!

!

!

!

!

::4!?-#.,#!18#+9+#)!@+((!A.!B.(C60(!6%$!D.!#%! +&#.$-8#!@+#B!%#B.$!C.%C(.!%&!G1H,C-8.!

2!

:!

;!


=!

!

!

!

!

!

!

:;4!?-#.,#!18#+9+#)!@+((!A.!B.(C60(!+&!,B-$+&'! D)!-,,+'&D.&#,!@+#B!%#B.$!0,.$,!%6!G1H,C-8.!

2!

:!

;!


=!

!

!

!

!

!

!

:<4!?-#.,#!18#+9+#)!@+((!A.!B.(C60(!+&!@%$F+&'! @+#B!0,.$,!%&!G1H,C-8.!#%!$.-8B!,%D.! 8%DD%&!'%-(!!

2!

:!

;!


=!

!

!

!

!

!

!

,-..$/#%&

!

!

!

!

!

:=4!G%DD.&#,!@+((!A.!B.(C60(!6%$!D.!#%!6+&7! -,,+'&D.&#,!$.(.9-&#!#%!D)!+&#.$.,#!

2!

:!

;!


=!

!

!

!

!

!

!

115

!

"#$%&'()! *+,-'$..!

*+,-'$..!

/.0#$-(!

1'$..!

"#$%&'()! 1'$..!

234!5%66.&#,!7+((!8.!9.(:;0(!;%$!6.!#%!;+&
=!

2!

>!

?!

@!

!

!

!

!

!

!

2A4!5%66.&#,!7+((!8.!9.(:;0(!;%$!6.!#%!;+&
=!

2!

>!

?!

@!

!

!

!

!

!

!

2D4!5%66.&#,!7+((!8.!9.(:;0(!;%$!6.!#%! +&#.$-E#!7+#9!%#9.$!:.%:(.!%&!51F,:-E.!

=!

2!

>!

?!

@!

!

!

!

!

!

!

2G4!5%66.&#,!7+((!8.!9.(:;0(!+&!,9-$+&'!6)! -,,+'&6.&#,!7+#9!%#9.$!0,.$,!%;!51F,:-E.!

=!

2!

>!

?!

@!

!

!

!

!

!

!

>H4!5%66.&#,!7+((!8.!9.(:;0(!+&!7%$C+&'!7+#9! 0,.$,!%&!51F,:-E.!#%!$.-E9!,%6.!E%66%&! '%-(!!

=!

2!

>!

?!

@!

!

!

!

!

!

!

!"#$%&'(

!

!

!

!

!

>=4!I-#+&',!7+((!8.!9.(:;0(!;%$!6.!#%!;+&
=!

2!

>!

?!

@!

!

!

!

!

!

!

>24!I-#+&',!7+((!8.!9.(:;0(!;%$!6.!#%!;+&
=!

2!

>!

?!

@!

!

!

!

!

!

!

>>4!I-#+&',!7+((!8.!9.(:;0(!;%$!6.!#%!;+&
=!

2!

>!

?!

@!

!

!

!

!

!

!

>?4!I-#+&',!7+((!8.!9.(:;0(!;%$!6.!#%!+&#.$-E#! 7+#9!%#9.$!:.%:(.!%&!51F,:-E.!

=!

2!

>!

?!

@!

116

!

"#$%&'()! *+,-'$..!

*+,-'$..!

/.0#$-(!

1'$..!

"#$%&'()! 1'$..!

!

!

!

!

!

!

234!5-#+&',!6+((!7.!8.(9:0(!+&!,8-$+&'!;)! -,,+'&;.&#,!6+#8!%#8.$!0,.$,!%:!<1=,9->.!

?!

@!

2!

A!

3!

!

!

!

!

!

!

2B4!5-#+&',!6+((!7.!8.(9:0(!+&!6%$C+&'!6+#8! 0,.$,!%&!<1=,9->.!#%!$.->8!,%;.!>%;;%&! '%-(!!

?!

@!

2!

A!

3!

!

!

!

!

!

!

!"#$%&$'()*#"%

!

!

!

!

!

2D4!E!:%0&F!0,.$!9$%:+(.,!.-,)!#%!0&F.$,#-&F!

?!

@!

2!

A!

3!

!

!

!

!

!

!

2G4!H,.$!I$%:+(.,!6+((!7.!8.(9:0(!:%$!;.!#%!:+&F! -,,+'&;.&#,!$.(.J-&#!#%!;)!+&#.$.,#!

?!

@!

2!

A!

3!

!

!

!

!

!

!

2K4!H,.$!I$%:+(.,!6+((!7.!8.(9:0(!:%$!;.!#%!:+&F! 9.%9(.!6+#8!,+;+(-$!+&#.$.,#,!-,!;.!

?!

@!

2!

A!

3!

!

!

!

!

!

!

AL4!H,.$!I$%:+(.,!6+((!7.!8.(9:0(!:%$!;.!#%!:+&F! ;%$.!-7%0#!9.%9(.!E!-($.-F)!C&%6!

?!

@!

2!

A!

3!

!

!

!

!

!

!

A?4!H,.$!I$%:+(.,!6+((!7.!8.(9:0(!:%$!;.!#%! +&#.$->#!6+#8!%#8.$!9.%9(.!%&!<1=,9->.!

?!

@!

2!

A!

3!

!

!

!

!

!

!

A@4!H,.$!I$%:+(.,!6+((!7.!8.(9:0(!+&!,8-$+&'!;)! -,,+'&;.&#,!6+#8!%#8.$!0,.$,!%:!<1=,9->.!

?!

@!

2!

A!

3!

!

!

!

!

!

!

117

!

"#$%&'()! *+,-'$..!

*+,-'$..!

/.0#$-(!

1'$..!

"#$%&'()! 1'$..!

234!5,.$!6$%7+(.,!8+((!9.!:.(;70(!+&!8%$<+&'! 8+#:!0,.$,!%&!=1>,;-?.!#%!$.-?:!,%@.! ?%@@%&!'%-(!!

A!

B!

3!

2!

C!

!

!

!

!

!

!

!"#$%"&'()*(+,&

!

!

!

!

!

224!D!7%0&E!#:.!;.%;(.!;-'.!.-,)!#%! 0&E.$,#-&E!

A!

B!

3!

2!

C!

!

!

!

!

!

!

2C4!6.%;(.!(+,#+&'!8+((!9.!:.(;70(!7%$!@.!#%! 7+&E!-,,+'&@.&#,!$.(.F-&#!#%!@)!+&#.$.,#!

A!

B!

3!

2!

C!

!

!

!

!

!

!

2G4!6.%;(.!(+,#+&'!8+((!9.!:.(;70(!7%$!@.!#%! 7+&E!;.%;(.!8+#:!,+@+(-$!+&#.$.,#,!-,!@.!

A!

B!

3!

2!

C!

!

!

!

!

!

!

2H4!6.%;(.!(+,#+&'!8+((!9.!:.(;70(!7%$!@.!#%! 7+&E!@%$.!-9%0#!;.%;(.!D!-($.-E)!<&%8!

A!

B!

3!

2!

C!

!

!

!

!

!

!

2I4!6.%;(.!(+,#+&'!8+((!9.!:.(;70(!7%$!@.!#%! +&#.$-?#!8+#:!%#:.$!;.%;(.!%&!=1>,;-?.!

A!

B!

3!

2!

C!

!

!

!

!

!

!

2J4!6.%;(.!(+,#+&'!8+((!9.!:.(;70(!+&!,:-$+&'!@)! -,,+'&@.&#,!8+#:!%#:.$!0,.$,!%7!=1>,;-?.!

A!

B!

3!

2!

C!

!

!

!

!

!

!

-))(,+."+*&/0(*1(+1$%23"&

!

!

!

!

!

CK4!D!7%0&E!+#!.-,)!#%!0&E.$,#-&E!:%8!#%!.E+#! #:.!E+77.$.&#!;$%;.$#+.,!%7!-&!-,,+'&@.&#!

A!

B!

3!

2!

C!

!

!

!

!

!

!

118

!

"#$%&'()! *+,-'$..!

*+,-'$..!

/.0#$-(!

1'$..!

"#$%&'()! 1'$..!

234!1,,+'&5.&#!.6+#+&'!7.-#0$.,!8+((!9.! :.(;70(!7%$!5.!#%!7+&6!-,,+'&5.&#,!$.(.<-&#!#%! 5)!+&#.$.,#!

3!

=!

>!

?!

2!

!

!

!

!

!

!

2=4!1,,+'&5.&#!.6+#+&'!7.-#0$.,!8+((!9.! :.(;70(!7%$!5.!#%!7+&6!;.%;(.!8+#:!,+5+(-$! +&#.$.,#,!-,!5.!

3!

=!

>!

?!

2!

!

!

!

!

!

!

2>4!1,,+'&5.&#!.6+#+&'!7.-#0$.,!8+((!9.! :.(;70(!7%$!5.!#%!7+&6!5%$.!-9%0#!;.%;(.!@! -($.-6)!A&%8!

3!

=!

>!

?!

2!

!

!

!

!

!

!

2?4!1,,+'&5.&#!.6+#+&'!7.-#0$.,!8+((!9.! :.(;70(!7%$!5.!#%!+&#.$-B#!8+#:!%#:.$!;.%;(.! %&!C1D,;-B.!

3!

=!

>!

?!

2!

!

!

!

!

!

!

224!1,,+'&5.&#!.6+#+&'!7.-#0$.,!8+((!9.! :.(;70(!+&!,:-$+&'!5)!-,,+'&5.&#,!8+#:!%#:.$! 0,.$,!%7!C1D,;-B.!

3!

=!

>!

?!

2!

! !"#$%&'()'$*'+,-.+$(+/$.,0'1$2'+'3-,)$.3$3'(,41')$.3$567)8(9'$2'&.:#$;.4$9(+$(<<$3'(,41')$,0(,$/.4$<.+=,$)''$ +&!#:.!(+,#4! ! !

!"#$%&"'

(")"*+$,'

D-''+&'!

!

E-#.,#!1B#+<+#)!

!

C%55.&#+&'!

!

F-#+&',!

!

G,.$!H$%7+(.,!

!

119

!"#$%&"'

(")"*+$,'

!"#$%"&'()*(+,&

&

-))(,+."+*&"/(*0(+0 $%12"&

&

!"#$%&'()*$+,$'+--'.$*/01! &

&

&

&

&

&

& & 345&-6"&7#8&2#+2"6+"/&19#8*&1+7&:17)&*;1*&8)"6)&.(,;*&198)"&*;"&<"1*86")&$6#=(/"/&#+&>-?)$12"@&-+7& 1//(*(#+1%&2#.."+*)&7#8&;1="&#+&$16*(28%16&<"1*86")@&!%"1)"&1//&*;".&9"%#:5& & &

!"#$%&"'

-.//")$,'

?1,,(+,&

&

-2*(=(*7&A*6"1.)&

&

>#.."+*(+,&

&

B1*(+,)&

&

C)"6&!6#<(%")&

&

!"#$%"&'()*(+,&

&

-))(,+."+*&"/(*0(+0 $%12"&

&

!"#$%&'()*$+,$'+--'.$*/01! &

&

&

&

&

&

& &

120

!"#$%&'()'$(**$(+,$-./'0$1-22'+.)$-+$345)6(1'$/'0'#$789:'$109.9;<'$-=$='(.<0')>$(**9.9-+(&$='(.<0')$./(.$,-<$ ./9+:$?9&&$@'$<)'=<&>$(+*$)-$-+#A$$ $

121