Cost Effective - Quick Usability Testing

Talk structure: 3 techniques for better interface design

Cost-effective methods for rapid user research and usability testing

Rashmi Sinha www.rashmisinha.com www.uzanto.com

www.rashmisinha.com

BayCHI UE BOF, July 02

Understanding your users • Finding out what issues are relevant

to them • Exploring their mental models

• Card Sorting – Not used to its full potential – We need an easier way to do and record data for card sorts • Persona Creation – The process for creating personas needs to be demystified. Period. • Usability Tests – Its time to stop thinking of this monolithic entity called “usability tests”. There are many types of usability tests with different levels of complexity. – Tests for competitive benchmarking or for iterative design process can borrow from experimental design techniques.

www.rashmisinha.com

BayCHI UE BOF, July 02

Understanding the categorical structures of the mind • Card sorting is one of a family of techniques to

tap into the user’s categorical structures. Animal

• Methods

(has skin, can move, eats)

Birds

– – – –

Interview Observation Ethnographic Methods Card Sorting & other methods of exploring categorization – Persona Creation

Fish

(wings, flies, feathers)

(swims, gills, fins)

Canary

Ostrich

Salmon

Shark

(sings, yellow)

(tall, does not fly)

(edible, pink)

(dangerous, gray)

• How semantic memory is organized

Focus on methods for information rich sites Semantic and categorical refer to similar things in present cont ext www.rashmisinha.com

BayCHI UE BOF, July 02

www.rashmisinha.com

BayCHI UE BOF, July 02

1

Why you need to understand categorical structures?

Example of individual differences

• Semantic organization builds up user

Sushi

expectancies regarding how things are organized.

Fish

– Design in accordance to these expectancies for better information architecture – Violate these expectancies and you leave the user lost and confused

Shark Aquarium

Diving

Grilled Salmon

Fish

• Do people’s categorical structures differ

Shark

from each other?

– Yes, semantic networks are determined by one’s experience, learning, culture etc. – but there is enough consistency across people for effective design

• Length of arrows implies semantic distance • Semantic memory is a network. Points are connected in

multiple direct and indirect ways.

• Degree of closeness between items might vary between

people, but we recognize the idiosyncratic association others make

www.rashmisinha.com

www.rashmisinha.com

BayCHI UE BOF, July 02

Semantic Distance and Proximity / Similarity / Distance Matrix

Semantic distance as basis of categorization • At root of all categorization techniques is

• How similar are the two (on scale of 1-5)

– salmon & shark:

1

2

3

4

5

Not similar

– canary & ostrich:

1

Not similar

Very similar

2

3

4

Salmon Shark 5 3 4 5 2 5 5

www.rashmisinha.com

5 Very similar

Self correlation Salmon Shark Canary Ostrich Trout Tiger Robin

Canary Ostrich Trout

BayCHI UE BOF, July 02

Tiger

question: “How far is A from B?”

• Proximity / similarity matrix can help generate

the most complex semantic networks (by using cluster analysis and other statistical techniques).

Robin Animal

5 4 3 2 3 2

(has skin, can move, eats)

5 2 4 4 1

5 4 4 2

BayCHI UE BOF, July 02

5 5 4

5 5

5

Birds

Fish

(wings, flies, feathers)

(swims, gills, fins)

Canary

Ostrich

Salmon

Shark

(sings, yellow)

(tall, does not fly)

(edible, pink)

(dangerous, gray)

www.rashmisinha.com

BayCHI UE BOF, July 02

2

Methods to tap into categorical structures • Open Card Sorting for generating information

architecture

Card Sorting: Understanding user’s mental models • Why: Helps you build site structure -good for deciding how to group information

• Closed Card Sorting for verifying information

architecture

• Generating semantic associations • Identifying dimensions used in categorization

• How: Write down each topic on an index card – Ask users to sort cards into piles – Have them name each category later

• Data obtained from card -sorting is indirect

enough that it avoids pitfalls of direct questioning

www.rashmisinha.com

BayCHI UE BOF, July 02

Kind of questions card sorting helps answer • Do the users want to see the information grouped

by: subject, process, business group, or type of information?

• What are the most important items to put on the

main menu?

• How many menu items should there be, and how

deep should it go?

• How similar or different are the needs of the

users throughout the organization?

www.rashmisinha.com

BayCHI UE BOF, July 02

www.rashmisinha.com

BayCHI UE BOF, July 02

Open and closed card sorting • Open Card Sorting: users given bunch of items to

sort those into categories. They are not given name or number of categories. – Useful for generating new or drastically redefining information architecture

• Closed Card Sorting:users given items to place

into pre -defined categories.

– Useful for verifying pre -existing information architecture – Useful for verifying category labels

www.rashmisinha.com

BayCHI UE BOF, July 02

3

Conducting card-sorting exercises • Rather involved method, requires a lot of setup.

Sorts themselves are fun to conduct.

• Main problem is in getting data into any form

where one can easily look at it, or share information with design team.

• Software for conducting card sorts. – IBM EZ Sort – NIST WebCAT

• Alternative method: Using online survey software

to conduct card sorts

www.rashmisinha.com

BayCHI UE BOF, July 02

Conducting card sorting exercises

Card sorting for designing an online travel guide • Example: Designing an online

travel guide to help users plan their trips to various locations.

• Purpose of card sort: – Lots and lots of items. How to structure the information.

• Items include – lodging, entertainment, local information, When to Go, Travel by Car/Air/Bus, Music Events, Hiking, Day Trips, Skiing, Diving, Golf, Emergency Info.

www.rashmisinha.com

BayCHI UE BOF, July 02

Card sorting using online survey software

• Check to make sure that

Each item (card) occupies a row

– – – –

each term is clear and unambiguous all the items that need to categorized were included set up items to be presented in random order create comprehensive instructions so that all participants have the same understanding of the process – Participants should be left alone during the sorting to avoid placing them under time pressure, but tester should remain available for answering questions

Each potential (unnamed category) is represented by a column User indicates category membership by checking box

• Conducting card sorting with online survey

software

– Most online survey software will do – Make sure it allowed you to present questions in matrix format, and allows downloading of each user’s data separately. www.rashmisinha.com

BayCHI UE BOF, July 02

www.rashmisinha.com

BayCHI UE BOF, July 02

4

Adding labels to categories

Looking at card-sorting data

At end, ask users to assign names / labels to each category they created Some users like to do this along the way (noting labels on piece of paper)

Cat-1 Cat-2 Cat-3 Cat-4 Cat-5 Look at data from one user at a time Name Category Entertainment Outdoor Lodging Practical Culture Info Before you go Before you go Fast Food At a glance you can Duties/Customs look down a Duties & Customs particular column, DIving DIving and see what items Nightclubs Nightclubs belong in it. ATMS's ATMS's Emergency info Emergency info You can also draw Etiquette Etiquette hierarchical Day Trips Day Trips structures showing created Museums & user Galleries Museums & GalleriesMuseums etc. categories Films Films Films & Movies Gay/Lesbian Hangouts Highlights Highlights Statistical analysis Highlights Architectural Sites Architecturalcan Sites create these

pictures for you across users.

www.rashmisinha.com

www.rashmisinha.com

BayCHI UE BOF, July 02

Closed Card Sorts

BayCHI UE BOF, July 02

Closed card sorts: when & how Category labels have been provided

• Category names were provided. • Users simply asked to assign items to categories. • Results are not informative about whether

number and structuring of categories was appropriate.

• Generally users are asked to give a “Yes / No”

response.

– One spin on this method is to ask users to rate “degree of certainty” in assigning items to categories. – This will help highlight problem categories / items. If users are very uncertain about assigning an item to a category, that should be probed further.

www.rashmisinha.com

BayCHI UE BOF, July 02

www.rashmisinha.com

BayCHI UE BOF, July 02

5

Rating “degree of certainty” in closed card sorts

Advantages of using online survey software for card sorts • Easy to carry out, no software to install and

figure out. Can be done using online survey software that are easily available.

• Can be done remotely • Trivial to get user-created categories into Excel

(or your favorite application) for qualitative / quantitative analysis.

• Can be used to conduct either open / closed card

sorts

• Can also be adapted to individual / group

sessions

www.rashmisinha.com

BayCHI UE BOF, July 02

Other issues about card sorting • Should each item be in one category • Should card sorts be done individually or with

groups

– group card sorts provides an energy and enthusiasm that makes light work of an otherwise tedious task – Can overshadow individual differences highlighting opinion of a few.

www.rashmisinha.com

BayCHI UE BOF, July 02

Other methods to tap into categorical structures • Open Card Sorting for generating information

architecture

• Closed Card Sorting for verifying information

architecture

• Generating Semantic Associations • Identifying dimensions used in categorization

• Combining card sorting with importance ratings is

a good idea.

www.rashmisinha.com

BayCHI UE BOF, July 02

www.rashmisinha.com

BayCHI UE BOF, July 02

6

Generating semantic associations • Helps understand what lies nearest to target items. Can be

conducted in conjunction with or independently of card sorting.

• Helpful in understanding problem categories and items • Semantic Association – What words do you associate below target words with Music: _____ _____ _____ ______ _____ _____ Diving: _____ _____ _____ ______ _____ _____

Generating semantic associations… • Category Member Association – What member items would you expect to find in below categories – Music: _____ _____ _____ ______ _____ _____ _____ _____ _____ ______ _____ _____ – Diving: _____ _____ _____ ______ _____ _____ _____ _____ _____ ______ _____ _____

• Category Association – What categories would you expect to find below words in Music: _____ _____ _____ ______ Diving: _____ _____ _____ ______ www.rashmisinha.com

BayCHI UE BOF, July 02

Identifying dimensions for categorizing

www.rashmisinha.com

BayCHI UE BOF, July 02

Talk structure: Personas…

• Present two random items and ask user what differentiates

• Card Sorting – Not used to its full potential – We need an easier way to do card sorts

• What differentiates pair below. List one or more ways that

• Persona Creation

them in your domain context. Repeat question till dimensions start repeating frequently.

they are different.

– Diving & Live Music Event: _______ ______ ______ – Shopping & Air Tickets: _______ ______ ______ – Motels & Restaurants: _______ ______ ______ • Good way to explore domain you are unfamiliar with • Some of the pairs will sound ridiculous!

www.rashmisinha.com

BayCHI UE BOF, July 02

– The process for creating personas needs to be demystified. Period. • Usability Tests – Its time to stop thinking of this monolithic entity called “usability tests”. There are different types of “usability tests”. – Tests for competitive benchmarking or for iterative design process can borrow from experimental design techniques.

www.rashmisinha.com

BayCHI UE BOF, July 02

7

Personas as “User Archetypes” • Personas is designing for one person. • Personas is a “User Archetype” who

represents needs and goals of many other users. – Who should that one person be?

• According to Cooper – Not the average user – Not a real user – Derived from background user research (interviews etc.)

Creating personas • Method – Conduct interviews with various stakeholders – Find patterns (inclusive cases) – Pick a nugget, interesting tidbit and build persona around it – Should represent large group in terms of goals, and personal characteristics • Problems with method: – Interviews are not economical way to find who representative users are – No tight coupling between user research & personas – Would two designers creating personas for same product create same personas? • Bottom line: Cooper’s persona creation method works well

when you have time on your hand, resources to interview many people, and skilled persona developers.

www.rashmisinha.com

BayCHI UE BOF, July 02

What is needed (from the horse’s mouth) “What is needed is a product definition tool that takes the guesswork out of the process—a tool that provides insight into what motivates people to use a product, so that well-grounded decisions can be made about features and how they are presented.” – Cooper.com

www.rashmisinha.com

www.rashmisinha.com

BayCHI UE BOF, July 02

Will market segmenting techniques serve any purpose? • Market segmentation is used to identify

clusters of people product can appeal to

– Mostly on basis of demographics – Sometimes on the basis of psychological variables (also called psychographic variables).

• Techniques can forecast marketplace

acceptance of products and services. Can also help convince executives to build product.

• Techniques not helpful in defining product

BayCHI UE BOF, July 02

www.rashmisinha.com

BayCHI UE BOF, July 02

8

Deconstructing marketing techniques • Questions focus on like / dislike of product

concept

– what do you think of vanilla coke or green Heinz ketchup?

• Statistical techniques used in market

segmentation cluster users according to demographic variables, not according to user needs from products.

• Concern themselves with product at a high

conceptual level. For interaction design, one needs to focus on specifics: How will product satisfy needs / goals.

www.rashmisinha.com

BayCHI UE BOF, July 02

Challenge: How to reliably identify user archetypes in cost-effective manner • Solution:

• Ground the personas in reality – Common critique of personas is that they are based on the designer’s imagination

• Focus on motives/needs of users • Be complementary to existing methods e.g.,

interview, observation

• Develop a concrete series of steps that any

designer can follow to develop personas

• Be reliable. When used by other designers, at

other times will lead to similar personas.

www.rashmisinha.com

BayCHI UE BOF, July 02

Designing personas for a Bay Area restaurant finder • Goals of site: Be the premier site for Bay Area

– Use survey techniques that are also used in market segmentation – Focus questions around user needs rather than what they simply like / dislike – Identify constellations of needs rather than clusters of users – Use this information as the kernel to build personas around

www.rashmisinha.com

What is needed is a method that will…

BayCHI UE BOF, July 02

residents & visitors to find restaurants of their choice.

• Design quick restaurant-finders (a la product

advisors) apart from conventional advanced search / browse options

• Conducted survey on features are used in finding

restaurants

– 45 respondents answered survey about importance of features in choosing restaurants – Food quality, décor, wine selection, cuisine, service

www.rashmisinha.com

BayCHI UE BOF, July 02

9

Steps in process

Design of survey

• Collect information via survey, interview,

observation

• Analyze survey data to identify underlying

factors, (types of needs)

• Identify some user archetypes • Verify these user archetypes with your qualitative

data (information from interview, observation etc.).

• Continue with your usual process of creating

personas

• Broad and shallow look • Try to be comprehensive • Identify features and get their importance rated

on same scale (5 or 7 point Likert scales)

– Example: How important are following restaurant features 1 2 3 4 5 – Service: Not similar

Very similar

• Don’t frame question in abstract way, instead tie

it to actual behavior

• Assure respondents about privacy, ask if they will

participate in follow-up interview

www.rashmisinha.com

BayCHI UE BOF, July 02

The survey: Importance ratings of 25 restaurant features

www.rashmisinha.com

BayCHI UE BOF, July 02

Main results of survey (based on 39 respondents) • Most respondents thinks quality of food, type of

cuisine, cleanliness is very important

• Few respondents think valet parking is important • Mixed feelings about features such buffet,

location, wine selection, waiting in lines etc.

• Most report going to restaurants with friends, or

as a couple. Fewer with family. Even fewer alone.

• Most report going 2 -3 times a week.

Next step: Identifying user archetypes based on patterns of needs… www.rashmisinha.com

BayCHI UE BOF, July 02

www.rashmisinha.com

BayCHI UE BOF, July 02

10

What would marketing techniques do? Food Quality

User ID 5240744 5240760 5241210 5241375 5241496 5241562 5243833 5243926 5243943 5245631 5253253 5261459 5263121 5264284 5264327 5264503 5264616 5264685 5264820 5264925 5265174

Décor 3

R e s t a u r a n t F e a t u r e s Wine ChildOutdoor Romanti Selectio Buffet friendly seating c n

User ID

Service

3

1

5

4

2

1

1

5 4 5

5 1 3

2 1 1

3 1 1

5 5 5

3 1 5

1 1 3

2 1 2

5 5 5

4 4 3

1 1 2

1 5 4

5 5 3

3 4 1

1 1 3

1 1 4

5 5 5

4 5 3

1 1 1

3 5 5

5 5 5

1 3 1

2 1 3

3 5 5

5 5

4 3

1 1

4 5

4 4

1 1

1 2

1 3

5 3 5

2 3 2

1 1 1

1 2 2

2 3 5

2 2 4

2 2 1

3 1 1

4 3 5

3 2 3

1 1 1

4 1 2

4 3 5

2 1 2

1 2 1

1 1 1

4 5 5

4 3 1

1 1 1

1 5 3

4 4 3

5 2 1

1 2 1

Design techniques need to identify cluster of needs

Cluster A

Cluster B

1 1 1

Food Quality

5240744 5240760 5241210 5241375 5241496 5241562 5243833 5243926 5243943 5245631 5253253 5261459 5263121 5264284 5264327 5264503 5264616 5264685 5264820 5264925 5265174

Décor 3

R e s t a u r a n t F e a t u r e s Wine ChildOutdoor Romanti Selectio Buffet friendly seating c n 1

5

4

2

1

1

5 1 3

2 1 1

3 1 1

5 5 5

3 1 5

1 1 3

2 1 2

5 5 5

4 4 3

1 1 2

1 5 4

5 5 3

3 4 1

1 1 3

1 1 4

5 5 5

4 5 3

1 1 1

3 5 5

5 5 5

1 3 1

2 1 3

3 5 5

5 5

4 3

1 1

4 5

4 4

1 1

1 2

1 3

5 3 5

2 3 2

1 1 1

1 2 2

2 3 5

2 2 4

2 2 1

3 1 1

4 3 5

3 2 3

1 1 1

4 1 2

4 3 5

2 1 2

1 2 1

1 1 1

4 5 5

4 3 1

1 1 1

1 5 3

4 4 3

5 2 1

1 2 1

1 1 1

Cluster A www.rashmisinha.com

BayCHI UE BOF, July 02

From cluster of needs to user archetypes • Features that are highly related to each other will

form part of the same constellation, or group of needs

• Factor analysis techniques can identify such

groupings.

• Each grouping can be said to represent a need

type.

• These are groupings of features not of users.

What need types emerge from restaurant survey?

www.rashmisinha.com

BayCHI UE BOF, July 02

Service

3

5 4 5

www.rashmisinha.com

Cluster B BayCHI UE BOF, July 02

User Archetype 1: Romantic, relaxed, dining experience… • Important Features – Good decor – Romantic setting – Relaxed service – Trendy – Caters to special Requirements (Low - fat, vegan etc.) – Type of crowd • Not Important Features – price, buffet-style – Many in 42- 51 age- group, spend more than $20, go out as a couple Food Quality, Type of Cuisine & Cleanliness does not appear in these archetypes because it was important to everyone.

www.rashmisinha.com

BayCHI UE BOF, July 02

11

User Archetype 2: Buffet style, with kids in tow… • Important Features – – – – – –

Buffet style dining Good for groups Child Friendly Relaxed Service Credit Card Portion Size

User Archetype 3: Meet for drinks, eat some food too… • Important Features – – – – – –

• Not Important Features – views, outdoor dining, people- watching, bar

Wine selection & bar Outdoor seating Easy access to public transport Live entertainment Type of crowd that visits place Good for groups

• Not Important Features – long Lines, finding Parking nearby

• Many in age group 32 -41, dine mostly with

family, once a week or lesser

www.rashmisinha.com

BayCHI UE BOF, July 02

After establishing some user archetypes

www.rashmisinha.com

BayCHI UE BOF, July 02

Advantages of technique

• Use user archetypes as “candidate personas”

• Inexpensive in terms of time and expertise

• Verify archetypes with results of interviews,

• A reliable method, two designers using same

• If survey is first step, then these can guide

• Retains creativity and fictional elements of

• Generate sample scenarios, make final decision

• Is complementary to current process of

observations

interviews (interview examples of each user archetype) on primary and secondary personas according to your design goals

method are likely to get similar results

creating personas, while grounding them in real data generating personas.

• Do not treat user archetypes generated through

survey as final word. Use them in conjunction with other information.

www.rashmisinha.com

BayCHI UE BOF, July 02

www.rashmisinha.com

BayCHI UE BOF, July 02

12

Talk structure: designing usability tests • Card Sorting

Usability tests for iterative design process & comparative benchmarking • Both scenarios require testing of more than one

– Not used to its full potential – We need an easier way to do card sorts

product

• Iterative Design requires multiple tests of same

software.

• Persona Creation – The process for creating personas needs to be demystified. Period.

– It also calls for creation of norms, baselines to compare subsequent versions against.

• Competitive Benchmarking calls for comparing

one product with other similar ones

• Usability Tests – Tests for competitive benchmarking or for iterative design can borrow from experimental design techniques. www.rashmisinha.com

BayCHI UE BOF, July 02

Not an experiment, has some features of it. Why? • So that one can make generalizable conclusions

• When comparing one product with another, test

becomes complex enough to benefit from experimental design.

www.rashmisinha.com

BayCHI UE BOF, July 02

6 steps to designing good usability tests

1: Identify the goal of the usability test; write it down

• Thinking about study design insures that potential

problems are identified beforehand, rather than later • Study designs are adaptable, 3-4 study designs can

be adapted for most testing situations • Helps make best use of limited time and resources

2: Identify factor you are most interested in (Independent Variable) 3: What changes do you expect in test results (Dependent Variabl es) 4: What random factors can influence results (Controlled Variables)

• Helps reduce number of users 5: Decide on a within-user / between -users study design. Take care of learning & fatigue effects 6: Make sure test design is practical (rapid and cost -effective), and gives you the results you need. Else redesign test. www.rashmisinha.com

BayCHI UE BOF, July 02

www.rashmisinha.com

BayCHI UE BOF, July 02

13

Example study: Examine usability of AskJeeves Digital Camera Advisor • Step 1: Identify goal: – To find effectiveness of Jeeves digital camera advisor • Method:

Compare effectiveness of Jeeves camera advisor to two popular camera advisors

Step 2: Identify Independent Variables • is the factor that undergoes a change • “type of camera advisor” is Independent Variable with three

levels

• MySimon, ActiveDecisions, & AskJeeves Independent Variable: Type of Camera Advisor Jeeves Camera Advisor

MySimon Camera Advisor

Active Decisions Camera Advisor

• In Iterative Design Process, “Version” is your Independent

Variable. Different versions (e.g., version 1, Version 1.5, Version 2) are your levels of Independent Variable

www.rashmisinha.com

BayCHI UE BOF, July 02

What if there is no Independent Variable? If you are interested in the usability of a particular software only… •

BayCHI UE BOF, July 02

Step 3: Identify dependent variables and decide how to measure each • These are the “things” you are measuring:

satisfaction, time to task, errors. The expectation is that they will change with changes in Independent Variable

Identify a relevant benchmark, even if benchmark is theoretical ideal (i.e., make one up) – the ideal system – older system that current software is replacing – competing software / process in the market

•

www.rashmisinha.com

Comparison with a relevant benchmark makes tests results more compelling and easier to interpret.

www.rashmisinha.com

BayCHI UE BOF, July 02

• Possible Dependent Variables: – Satisfaction: User comments, satisfaction questionnaire – Task success: time taken, number and type of errors

www.rashmisinha.com

BayCHI UE BOF, July 02

14

Step 4: What controlled variables could inadvertently influence results • Random factors that can effect results (need to control

them). E.g., network speed, testing conditions (light, monitor size), fatigue of subjects.

• These are like Independent Variables, except they are not

of current interest

• One needs to rule out effects of these variables by

Review types of variables: Independent, Dependent & Controlled • Independent Variable: Is the factor that

undergoes a change, the factor you are interested in. – Two versions of a website, three authentication systems

• Dependent Variable: Is the outcome of the test,

the thing that is effected by the independent variable

controlling them (equalizing their effect in different conditions).

• Controlled variables in current study – Testing Conditions: light, network speed, monitor size – Instructions given: whether few/ detailed / no instructions – User Variables: age, gender, previous web & computer experience

www.rashmisinha.com

– No of errors in finding item, time to get credit card authenticated by system

• Controlled Variables: factors that can cause

random changes in test results. Need to be controlled

– Speed of network connection, Previous web experience of user

www.rashmisinha.com

BayCHI UE BOF, July 02

Step 5: Within or between user design

BayCHI UE BOF, July 02

Within user study design

• Within user design: each user participates in all conditions. – Advantage: do not need to control for user differences – Disadvantage: Learning and Fatigue effects

9 users (A-I), within-users design

• Between-user design: each user participates in one condition. – Advantage: No learning and fatigue effects – Disadvantage: Need to control for between user differences

Users

Order 1

Order 2

Order 3

A

MySimon

Jeeves

Actv.Decs

B

Jeeves

Actv.Decs

MySimon

C

Actv.Decs

MySimon

Jeeves

D

MySimon

Jeeves

Actv.Decs

E

Jeeves

Actv.Decs

MySimon

Independent Var.: Type of Camera Advisor

F

Actv.Decs

MySimon

Jeeves

Jeeves Camera MySimon Camera Advisor Advisor Users A,B,C Users A,B,C

G

MySimon

Jeeves

Actv.Decs

H

Jeeves

Actv.Decs

MySimon

I

Actv.Decs

MySimon

Jeeves

• Practical decision – Depends on number of users you can afford, length of test etc. – Determines how easy it is to interpret test results, and how far one can generalize from test results 2 diff. designs

Within-Users design (3 users) Between-Users design (9 Users A,B,C users) www.rashmisinha.com

Users D,E,F

BayCHI UE BOF, July 02

Act. Dec. Camera Advisor Users: A,B,C Users: G, H, I

www.rashmisinha.com

•Each system occurs 3 times in Order 1,2, & 3 •Study design takes care of learning, fatigue effects •Design is very efficient, but there might be learning/fatigue effects

BayCHI UE BOF, July 02

15

Between users study design • 9 users (A-I), between-users design Users

Order 1

A

MySimon

B

Jeeves

C

Actv.Decs

D

MySimon

E

Jeeves

F

Actv.Decs

G

MySimon

H

Jeeves

I

Actv.Decs

www.rashmisinha.com

•Each system is used by 3 users, each user only uses 1 system •Study design does not pose any learning, fatigue effects •Tricky issue is to make sure users for different systems are comparable •Design is not very efficient, but easy to interpret results (if users are comparable)

BayCHI UE BOF, July 02

Mixed Study Design • 9 users (A-I), mixed between-users design Users

Order 1

Order 2

A

MySimon

Jeeves

B

Jeeves

Actv.Decs

C

Actv.Decs

MySimon

D

MySimon

Jeeves

E

Jeeves

Actv.Decs

F

Actv.Decs

MySimon

G

MySimon

Jeeves

H

Jeeves

Actv.Decs

I

Actv.Decs

MySimon

www.rashmisinha.com

•Each system is used by 6 users, each user only uses 2 system •Each system is used 3 times in Order 1 and 3 times in order 2 •Design compromises between demands of efficiency and fatigue/learning effects •Tricky issue is to make sure users for different systems are comparable

BayCHI UE BOF, July 02

For more information go to…

www.rashmisinha.com www.uzanto.com Or email me at [email protected]

www.rashmisinha.com

BayCHI UE BOF, July 02

16