Taking The Case: An Evaluation Of Digital Case Study Libraries

Taking the Case: An Evaluation of Digital Case Study Libraries Yeong-Tay Sun Center for HCI at Virginia Tech 2202 Kraft Drive Blacksburg, Virginia 24060 540-231-7409 [email protected] Laurian C. Vega Center for HCI at Virginia Tech 2202 Kraft Drive Blacksburg, Virginia 24060 540-231-7409 [email protected] D. Scott McCrickard Center for HCI at Virginia Tech 2202 Kraft Drive Blacksburg, Virginia 24060 540-231-7409 [email protected]

Abstract: The field of Human-Computer Interaction (HCI) is intimately ingrained with the emerging development of educational resource tools. Case study libraries are one such tool that have proven to be of great interest due to their adaptability as learning resources. This study explores the ability of a collection of such case study libraries to support its users, gradually revealing useful heuristics information derived from design characteristics affecting the usability of each system. Though current results are preliminary, several interesting questions regarding usability issues have been raised that may warrant additional assessment.

1.

INTRODUCTION

Design, in general, is nebulous. In Human-Computer Interaction (HCI), we design first with the user in mind. However, even within the domain on HCI there is debate on the 'correct' way to start designing [5]. Therefore, there are many issues that teachers of design principles (the fundamental rules of interface design) can expect to address, from choices in development methodologies (such as usability engineering or goal-directed design) to learning objectives in the classroom. An emergent focus in education is on the design and implementation of useful learning tools that may help students excel in their understanding of specific topics. Methodologies, while indispensable for introducing an established curriculum from which to base course material and exercises, are necessarily complex due to their nature and invariably retain their own deficiencies such as timeliness, lack of regular feedback and lack of breadth [9]. In light of this, the introduction of tools that could sufficiently support and expedite the study of different concepts being taught in design courses today would be immensely useful, particularly to novices of those subjects [1]. The field of HCI, with its goals for improving interface design and implementation, is well-suited to the development of such tools intended for improving the learning curriculum. Out of the many different tools that have been proposed, the use of case study libraries have proven to be of some immediate interest. A case study involves a persistent analysis of a single event, possibly over some period of time. The basic structure of such a study provides relevant data with some degree of contextual information, while a subsequent analysis of the case's events eventually arrives at any number of conclusions. As a result, the reader of the case study will be able to better understand why these events happened as they did, and may also gain insight as to what elements

might warrant a more extensive analysis in future studies. Such cases lend themselves to both the generation and testing of hypotheses and, where appropriate, have proven themselves to be an immensely useful learning tool due to their ability to present problems in the form of an intuitive "story-like" narrative or structure [3]. Additional studies suggest that case-based approaches to learning promote key meta-cognitive skills including cognitive elaboration, error management, reflection, self-regulation, and transfer of knowledge [3]. Perhaps the most useful feature of a case study is the ability to simulate problems in a practical, realistic manner. This makes it easier to draw parallels to real-world situations and enhance the relevance and applicability of the material to both students and instructors alike. With a solid understanding of the underlying concepts of one case, one might be encouraged to make a logical leap of intuition to discover parallels between previously unassociated cases [4]. Obviously, a single use-case study would have severely limited applications in both scope and utility; thus we turn our attention to large collections of multiple use cases, and how such a resource can be beneficial to students in an academic setting. In the context of this discussion, we introduce an implementation of such a library called the Usability Case Study (UCS) which serves as a link between collections of relevant case studies and related artifacts [8]. A prevailing characteristic of each of these cases is their attention to the detailed documentation of the steps and artifacts generated during a project's conception. These may include requirements stages, task analysis, usermodeling, various forms of prototyping, evolutionary development, and usability evaluation methods involved in real-world instances of design examples, ultimately allowing for a prototyping of cases at various levels of abstraction [3]. In the original study, this tool was employed in the task of supporting specific topics of a usability engineering course over the length of a semester by illustrating the different aspects of a project's design process. By the semester's end, reactions towards the UCS were generally positive. In particular, it was reported that students, following a natural period of adaptation to the system, developed a more sophisticated understanding of usability engineering practices through their readings of homework assignments and participation in case-based activities [3]. The overall result was a better technical understanding of how human-centered software design works. It should be noted that this system was not intended as a replacement for regular lecturing or class sessions; rather, it was meant to support and enhance the comprehension of specific topics so that they might consume less time from other areas to be covered over the course's duration.

2.

BACKGROUND & RELATED WORK

Curiously enough, recent experiences with the UCS aimed at mimicking the findings of this research have produced results seemingly to the contrary [2]. In particular, anecdotal evidence indicates that these improvements in user comprehension have been difficult to reproduce under similar settings. An informal study, which also utilized the UCS to incorporate use cases into the design curriculum of a usability engineering class, was conducted over the better part of a regular semester course. At the semester's end, it was generally determined that user satisfaction with the system was somewhat lower than expected. In particular, it was remarked that the UCS proved to be overly intimidating to users who were unfamiliar with the system, thereby lengthening the time of adoption necessary before producing useful results [3]. Another negative report indicated that while many different pieces of a particular case study were present in some form, explanations of how each component fit with the others, or even explanations regarding an individual artifact's own significance, were minimal; that is, very little guidance was provided and it was ultimately up to the user to form their own interpretations. The effects of this might be negligible on students who were already familiar with a given concept or topic being studied, but proved to be an annoyance and possibly an unhelpful distraction to students who required further assistance. A related complaint was that the degree of detail present in the case studies, though organized in a relatively straightforward manner, nevertheless made it a daunting and time-consuming task to sift through an individual case study, thereby lessening user enthusiasm towards the system. In particular, a specific complaint voiced by the system's administrator claimed that the UCS did not adequately address certain stages of design that were key to developing a comprehensive understanding of the design process overall. Specifically, as we will also discuss later, the system allegedly failed to represent the inherent complexity present during the design process by oversimplifying the represented artifacts without adequately addressing the gradual progression in between. Further, the overall evolution of a project in which the initial brainstorming of system features are refined through elimination, feasibility testing, and requirements modifications was noticeably sparse, and so the representation might not be as true to life as could be expected. Consequently, this could have the unfortunate effect of misrepresenting the design process overall. One could further observe that many user sentiments regarding their experiences with the UCS actually coincided somewhat with the initially negative reactions expressed in the original study, though the end result was certainly different. It is

also interesting to note that despite the fact that both studies initially encountered user complaints, there was a fundamentally important distinction between the two groups. The complaints from the original experiment seemed to be aimed at inefficiencies and missing features in the system, or self-diagnostic issues. In contrast, students from the most recent study seemed to dismiss, or were genuinely baffled by the integration of the system as part of the curriculum in the first place; that is, they couldn't quite seem to grasp the relevance of the system in the context of what they were learning. While this experience has certainly formed a point of contention, it should be emphasized that at this time we are not attempting to make any specific assertions regarding the ability of the UCS to support its users (as there is insufficient evidence to make such a contention). Obviously, there are quite a few different plausible explanations for this phenomenon. The fault may lie with some previously undetected differences between the two groups exposed to the system, or it may be with the individuals conducting the studies. However, there is also the distinct possibility that the issue may lie within the fundamental design of the system itself. With these results as inspiration, there are several courses of action that we could take. For example, we could compile the lists of reactions and arguments from the system's users and try to incorporate these suggestions into the revision of a newer, better system. However, the problem with the UCS may be a problem in general with case study libraries. At this point in our research we would like to reassess case study libraries - in general - to generate and collect some heuristics for designing systems like the UCS. That is to say, based on information collected and analyzed on our own accord, we would like to have some assurance that the grievances of the users of the system are justified (and perhaps correlated in some manner). 2.1

Test Libraries

To this end, a brief evaluation of three different online case study libraries incorporating useful functions or distinguishing characteristics was conducted. Each of these libraries has satisfied our criteria for a successful implementation of a use case library: they have either been integrated into the regular learning curriculum in an academic setting, and/or they have been featured prominently as a section of a publicly accessible website with relatively high visibility for some period of time. The focus of each library in itself is not especially significant, but collectively they serve as a representative sampling of actual case study libraries currently in use. Despite their differing subject matters, each website shares a certain number of similarities as well as unique differences, both of which we would like to examine in detail. 2.1.1

Pathological Case Database

http://path.upmc.edu/cases.html This library incorporates a fairly large database with over 480 cases represented, providing entries from the last decade or so. Photographs or diagrams of pathological data (cultures, etc.) are depicted, as applicable or appropriate. Diagnoses compare impacted organs of a patient to healthy, normally functioning model. A wide variety of defects requiring different degrees and techniques of diagnosis are presented. An interesting navigational element is incorporated by splitting each case study into two parts. The first part includes all relevant details of a case, along with a preliminary diagnosis and other relevant information. At the bottom of each page is a link to the second half of the case study which presents a detailed "Final Diagnosis" of a patient's condition. Presumably, the separation of these two sections is to encourage critical thinking from the user by giving them the opportunity to speculate on their own diagnosis based on previously available information, rather than digesting all of the conclusions in the same motion. This library lacks powerful search and categorization tools. UCS Similarities: Complex Search w/ advanced parsing, Mainly sidebar-driven, High-level context provided as initial view, Standard template format, Multimedia, Well-structured artifacts, Consistent formatting, Real-world examples 2.1.2

TED Case Studies

http://www.american.edu/TED/class/all.htm

This is a 14-year archive of cases (consisting of two collections bi-annually) covering trade issues that affect the environment on a global scale; many countries are covered. Case studies are presented as a rather lengthy articles of text; multimedia is incorporated as applicable. The large amounts of information draw parallels to real-world scenarios where it may not be immediately obvious as to what is relevant and what isn't. Each case is generally divided into sections based on a unified template, while mnemonics are used for easy case identification. On closer inspection, there seems to be a great diversity among the authors/submitters for the cases included in this library. Most likely, this is a culmination of resources that was taken from local sources and adapted to the format of this library. Although the issues of quality control and content review inevitably come into play at some point, this is an excellent way of keeping the pool of case studies diverse and relevant. UCS Similarities: Key scenarios essential to case comprehension appear in their own section, along with relevant supporting materials, Standard template format, Multimedia, Artifact interpretation is open, Consistent formatting, Real-world examples 2.1.3

Case Law Database

http://www.cre.gov.uk/legal/casedatabase.html These case studies span a number of years, with landmark cases being clearly delineated. Key points of the law that factor heavily into the subsequent ruling of a case are outlined, while a summary of rulings appears towards the end; this includes related laws and arguments influencing a decision. Cases are divided into seven sections according to the most common groupings of cases. The body itself incorporates a combination of both bullet-style lists and paragraphs of text, selecting the most suitable one for a given sub-section. Summaries, by nature, tend to emphasize the notable points of an argument and would be better suited to a list-style layout. Case backgrounds, besides giving specific information, are also meant to establish a case's context and are written in paragraph form to maintain their "narrative" style. Besides contextual awareness this variety discreetly separates the text into manageable chunks, resulting in an immensely more readable page. UCS Similarities: Complex Search w/ advanced parsing, Mainly sidebar-driven, High-level context provided as initial view, Standard template format, Well-structured artifacts, Real-world examples 2.1.4

Discussion

You may have noticed that none of the selected case study libraries are of a usability engineering-centric, or even particularly technical (computer-wise) nature. This is mainly because the characteristics that we are attempting to gather should be those that are common among all successful case study libraries. If we were to restrict our observations to libraries that focus only on similar subject matters, we might very well be overlooking some essential feature of a successful library, perhaps one with a predisposition towards not appearing in a technicallyoriented case library. Secondly, it is generally assumed that observations between similar libraries have already been conducted to some degree (and likely in the original design stages of the UCS as well). By observing these common characteristics we wish for the ability to recreate particular features from scratch based on their demonstrated worth, rather than drawing on inspiration from a particular source.

3.

EXPERIMENTAL PROCEEDINGS

3.1

Purpose

Such case libraries are only useful when they are both easy to use, and helpful for those individuals who will be using them. The functional possibilities for access methods are many and varied. Certain preexisting notions of optimal usability and interface layout certainly exist, but it is often difficult to discern exactly what elements in a given implementation make one user interface inherently more usable over another. Additionally, a virtually limitless number of factors may contribute to the success of a case study library such as access demographics, data presentation, available tools, relevance of included material, quality of content, and so on. It is fairly self-evident

that even subtle changes to the user interface of a software program can have drastic effects on its subsequent usability, for better or for worse. Alterations in the structure, length, emphasis, and many other components of an individual use case can produce similar effects, and many parallels can be drawn between the two. Since we wish to explore the basic components of a use case library that can shape a user's experience, both are equally relevant to the goals of this study, and must be equally assessed to gain a comprehensive understanding of the system. 3.2

Basis

At this point, what would be the best method for gathering information that would provide useful insight into the effective design and implementation of a use case library? Despite the fact that such skills may only be mastered through practical experience, virtually all people tend to learn some strategies to improve their design approaches through experience and feedback [6]. Assuming for a moment that we know nothing about designing a successful case study resource, we can speculate that the relevant points of interest will most likely be reflected in the many interactions between a resource and its users. In this way, these points may be revealed through the careful observation of similar resources that are already successfully in use. A study of this nature would allow us to discern the most influential and effective aspects of these libraries and incorporate them into our own work. In essence, this study could be considered to be a momentary step back for a more complete system assessment (to "see the forest for the trees", so to speak) before proceeding ahead. It is our expectation that the information gathered from this study will eventually be used to support the development of an enhanced version of the original system that produces a more desirable result. 3.3

Procedure and Overview

Our experiments involved nine (9) participants, seven (7) male and two (2) female. They were not in any inclusion or exclusion criteria; however, the intent was to seek participants who were novice designers in computer science. This was to minimize the effects of participants with past specialization in interface design or usability engineering which might skew results. Additionally, since the goal of developing this software was to help this audience design better systems, it was likewise in our best interests to select our participants from this population. Each participant was randomly assigned a case study library, then participated in a three-part experiment. The first part involved a series of tasks written on separate sheets of paper, to be completed in order. These tasks, involving a specific use case library, can be found below in Table 1 and were designed for probing specific aspects of the user's interaction with the system.

Question Starting from a website's front page, attempt to navigate to the case study database's Homepage, titled respectively: Starting from a case study database's front page, attempt to locate a case study which: Attempt to find the answers to the following questions (within each case study):

Attempt

to

find

the

Pathology Case Studies "Department of Pathology Online Case Studies"

Trade Environment Database "The TED Case Studies - An Online Journal"

Employment Case Law "Employment database"

is categorized under "Molecular Diagnostics": involves a man with prior medical history of Behcet's disease.

is categorized as being published in January of 1995: involves the effects of tourism on the ecosystem of national parks.

is categorized under "Direct Discrimination": involves a photographer with 22 unsuccessful applications for promotion.

Which of the following symptoms is not exhibited by the patient in the initial diagnosis? [thin appearance; impaired vision; increased blinking; symmetrical reflexes] "Chemistry" data

According to the article, what island has stopped the hunt for seals?

According to the article, how many steps are there in identifying race discrimination?

What were two factors that

What was the officially cited

case

law

Attempt to find the answers to the following questions (within each case study):

"Chemistry" data What were two factors that What was the officially cited measurements in this case contributed to the decline of reason for the termination of include which of the rug trades in Iran in the late the applicant's membership? following? [phosphorus, 1980s? barium, fluorine, glucose, potassium] Identify the number of sections in this case that have appeared in other case studies explored thus far. From a case study database's Homepage, identify the closest means of providing user feedback. This could be an e-mail address or a scripted electronic form, for example. If none are available (or visible), navigate the site until one is found.

Table 1: Examples of Questions Questions could range from something as simple as locating the front page of a case study library from a website's main page to finding specific pieces of information within a specific case study. The emphasis should be made that the primary goal was not to test the performance of the user, nor was it to specifically assess the correctness or incorrectness of their answers, but to develop familiarity with a given case library. Once a participant completed all of their tasks, they were asked to fill out a short likert-style questionnaire requesting feedback on various aspects of the system. These questions, involving issues such as system usability and overall user satisfaction, were deemed appropriate at this time since the participant would now be able to make informed responses based on their recent experiences. Finally, an additional number of questions were administered verbally, expanding considerably upon those in part two. These questions informally solicited opinions such as the sort of activities one might use a given library for, or how certain characteristics of the case studies influenced the participant's impression of the library. We generally considered this last portion of the experiment to provide the most useful information since we could assess the participant's opinions of the system directly.

4.

RESULTS AND INITIAL ANALYSIS

Based on the averaged results of the electronic survey in step 2 of this study, it appears that the participants overwhelmingly selected the third use case library (Employment Case Law) as yielding the most satisfactory usability experience. Coincidentally, it was also the library with the shortest average case length. Conversely, the second use case library (Trade Environment Database) with the longest average use case lengths also garnered the lowest average value for participant satisfaction. These results indicate that the density of information plays a significant role in the usability of such a library. Users appear to be overly intimidated when the amount of text extends much farther than two full screens. In these situations, the tendency is to visually latch on to anything distinctive-looking in the page; often something as simple as a bolded line of text is adequate for distinguishing one section from another. Generally, users reacted poorly to disruptions in formatting consistency across pages, feeling that that the lack of consistent section headers and formatting requirements was overly disorienting and a fairly poor design decision overall. The fact that these sections were inconsistent also reflected poorly on the "glance-ability" of each page. That is, in the case of pages with a high density of presented information, some user felt that it was generally futile to try and spot the information through sequential visual scanning. Also, this affected the participants feelings of security in the information being presented. When the formatting was different users were not sure whether the information was from the same library which raised concerns of being too far off task. Interestingly enough, it has been observed that many participants tend to reserve the use of search functions as a last resort. Rather, they initially prefer to scan the presented information visually, and in sequential order from top to bottom until they decide that this method might be unfeasible. It may be that instead of trying to locate the specific answer for a question, the participant is trying to assemble a contextual background for the question that might aid them in locating the answer that much more quickly. Another common complaint indicated that the overall hierarchy of information for a given case study library was either so high-level as to be useless, or so dense as to be unusable. Users, apparently, are not adverse to different views of information - that is, the ability to "drill-down" a particular piece of content to simplify or expand the data to the desired level of complexity as necessary. At this point this highlights, perhaps, a way to apply Schneiderman's mantra of “overview first, zoom and filter, and then details on demand” [7].

Most participants revealed that more often than not, the mere inclusion of multimedia elements (such as pictures) seemed to have a marked effect on the user's reported usability of a case, regardless of whether or not the pictures actually appeared to have any sort of tangible effect on the structure or flow of the case study. Interestingly, when asked, participants said that the use of pictures allowed them to understand what the case was about. In a sense, it allowed them to gain a high level understanding of what the case was about. Similarly, it made the participants feel that the authors of the case study library took more time in crafting the case study. The participants then went on to correlate this reaction to the case being better than others. Other user reactions indicated, with surprising consistency, that they were somewhat skeptical of the legitimacy of a piece of information retrieved from such a library because of the apparent lack of citations (the Employment Case Law being the exception). The general sentiment was that there was only a limited trust of the library's information, and that one could only safely use it in conjunction with a third-party search to confirm its validity; that is, they wouldn't feel comfortable using it as a standalone resource, although some users decided to trust the given information based on the subject matter alone.

5.

CONCLUSION

This paper has presented a brief introduction to the appeal and usefulness of case study libraries in academic settings, as well as some motivations and strategies for our heuristics gathering activities. We have also reviewed some of the preliminary results from the experiments currently being collected for this study. The Case Law library appears to have garnered the most positive impressions from participants by far, thanks to its concise narrative, consistent layout and plain language. The "glance-ability" of this library was also improved by the subdivision of the main body of text into a series of easily accessible sections. The pathology library elicited the most neutral responses from experimental participants. It's use cases generally possessed the brevity shared with the Case Law library, but were hampered by an abundance of technical jargon, with a similar effect on participant performance. Finally, despite the extensive use of multimedia, the TED library cases proved to be less than popular to participants due to their incredible length and extensive pretext, marginal search capabilities for a database consisting of well over five hundred cases, and perpetually shifting page layouts, formats, colors, and virtually everything else from case to case.

6.

FUTURE WORKS

By analyzing this information, we wish to build a set of heuristics that will be indispensable in our future studies of educational resources, such as case study libraries, design and usability. Specific topics pending full analysis for future research may include the notion of "browsing vs. searching", the use of multiple data hierarchies to organize and streamline data access, the role of information density on a user's usability experience, and the role of trust as it applies to use case resources. Eventually, we wish to augment the Use Case System to a state where its functionality will be equivalent to the results found in the original study, and will serve as a useful learning tool for students in their study of the principles of design. From there, we wish to continue building on the successful features of the UCS, continuing to address issues that the original study may have overlooked.

7.

REFERENCES

[1]

Ardito, C., M. D. Marsico, et al. (2004). Usability of E-learning Tools. Proceedings of the Working Conference on Advanced visual interfaces. Gallipoli, Italy, ACM Press: 80-84.

[2]

Berry, B., L. Hobby, et al. (2006). Exploring Benefits of Visualization for Case Studies. World Conference on Educational Multimedia, Hypermedia and Telecommunications 2006. P. Kommers and G. Richards. Orlando, FL, USA, AACE: 485-492.

[3]

Carroll, J. M. and M. B. Rosson (2005). "A case library for teaching usability engineering: Design rationale, development, and classroom experience." Journal of Educational Resources in Computing 5(1): 3.

[4]

Gick, M. L. and K. J. Holyyoak (1980). "Analogical problem solving." Cognitive Psychology 12: 306-355.

[5]

Harrison, S., M. Back, et al. (2006). "It's Just a Method!": A Pedagogical Experiment in Interdisciplinary Design. Proceedings of the 6th ACM conference on Designing Interactive Systems. University Park, PA, USA, ACM Press: 261-270.

[6]

Lewis, T. L., M. B. Rosson, et al. (2004). What Do The Experts Say?: Teaching Introductory Design from an Expert's Perspective. Proceedings of the 35th SIGCSE Technical Symposium on Computer Science Education. Norfolk, Virginia, USA, ACM Press: 296-300.

[7]

Schneiderman, B. (1998). Designing the User Interface: Strategies for Effective Human-Computer Interaction. Reading, MA, Addison Wesley.

[8]

"Usability Case Study Library (UCS)." from http://ucs.ist.psu.edu/.

[9]

Wright, T., J. Noble, et al. (2005). Using a System of Tutorials and Groups to Increase Feedback and Teach User Interface Design. Proceedings of the 7th Australasian Conference on Computing Education. Newcastle, New South Wales, Australia. 42: 187-192.