Cost Effective - Quick Usability Testing
This document was uploaded by user and they confirmed that they have the permission to share it. If you are author or own the copyright of this book, please report to us by using this DMCA report form. Report DMCA
Overview
Download & View Cost Effective - Quick Usability Testing as PDF for free.
More details
- Words: 4,788
- Pages: 16
Talk structure: 3 techniques for better interface design
Cost-effective methods for rapid user research and usability testing
Rashmi Sinha www.rashmisinha.com www.uzanto.com
www.rashmisinha.com
BayCHI UE BOF, July 02
Understanding your users • Finding out what issues are relevant
to them • Exploring their mental models
• Card Sorting – Not used to its full potential – We need an easier way to do and record data for card sorts • Persona Creation – The process for creating personas needs to be demystified. Period. • Usability Tests – Its time to stop thinking of this monolithic entity called “usability tests”. There are many types of usability tests with different levels of complexity. – Tests for competitive benchmarking or for iterative design process can borrow from experimental design techniques.
www.rashmisinha.com
BayCHI UE BOF, July 02
Understanding the categorical structures of the mind • Card sorting is one of a family of techniques to
tap into the user’s categorical structures. Animal
• Methods
(has skin, can move, eats)
Birds
– – – –
Interview Observation Ethnographic Methods Card Sorting & other methods of exploring categorization – Persona Creation
Fish
(wings, flies, feathers)
(swims, gills, fins)
Canary
Ostrich
Salmon
Shark
(sings, yellow)
(tall, does not fly)
(edible, pink)
(dangerous, gray)
• How semantic memory is organized
Focus on methods for information rich sites Semantic and categorical refer to similar things in present cont ext www.rashmisinha.com
BayCHI UE BOF, July 02
www.rashmisinha.com
BayCHI UE BOF, July 02
1
Why you need to understand categorical structures?
Example of individual differences
• Semantic organization builds up user
Sushi
expectancies regarding how things are organized.
Fish
– Design in accordance to these expectancies for better information architecture – Violate these expectancies and you leave the user lost and confused
Shark Aquarium
Diving
Grilled Salmon
Fish
• Do people’s categorical structures differ
Shark
from each other?
– Yes, semantic networks are determined by one’s experience, learning, culture etc. – but there is enough consistency across people for effective design
• Length of arrows implies semantic distance • Semantic memory is a network. Points are connected in
multiple direct and indirect ways.
• Degree of closeness between items might vary between
people, but we recognize the idiosyncratic association others make
www.rashmisinha.com
www.rashmisinha.com
BayCHI UE BOF, July 02
Semantic Distance and Proximity / Similarity / Distance Matrix
Semantic distance as basis of categorization • At root of all categorization techniques is
• How similar are the two (on scale of 1-5)
– salmon & shark:
1
2
3
4
5
Not similar
– canary & ostrich:
1
Not similar
Very similar
2
3
4
Salmon Shark 5 3 4 5 2 5 5
www.rashmisinha.com
5 Very similar
Self correlation Salmon Shark Canary Ostrich Trout Tiger Robin
Canary Ostrich Trout
BayCHI UE BOF, July 02
Tiger
question: “How far is A from B?”
• Proximity / similarity matrix can help generate
the most complex semantic networks (by using cluster analysis and other statistical techniques).
Robin Animal
5 4 3 2 3 2
(has skin, can move, eats)
5 2 4 4 1
5 4 4 2
BayCHI UE BOF, July 02
5 5 4
5 5
5
Birds
Fish
(wings, flies, feathers)
(swims, gills, fins)
Canary
Ostrich
Salmon
Shark
(sings, yellow)
(tall, does not fly)
(edible, pink)
(dangerous, gray)
www.rashmisinha.com
BayCHI UE BOF, July 02
2
Methods to tap into categorical structures • Open Card Sorting for generating information
architecture
Card Sorting: Understanding user’s mental models • Why: Helps you build site structure -good for deciding how to group information
• Closed Card Sorting for verifying information
architecture
• Generating semantic associations • Identifying dimensions used in categorization
• How: Write down each topic on an index card – Ask users to sort cards into piles – Have them name each category later
• Data obtained from card -sorting is indirect
enough that it avoids pitfalls of direct questioning
www.rashmisinha.com
BayCHI UE BOF, July 02
Kind of questions card sorting helps answer • Do the users want to see the information grouped
by: subject, process, business group, or type of information?
• What are the most important items to put on the
main menu?
• How many menu items should there be, and how
deep should it go?
• How similar or different are the needs of the
users throughout the organization?
www.rashmisinha.com
BayCHI UE BOF, July 02
www.rashmisinha.com
BayCHI UE BOF, July 02
Open and closed card sorting • Open Card Sorting: users given bunch of items to
sort those into categories. They are not given name or number of categories. – Useful for generating new or drastically redefining information architecture
• Closed Card Sorting:users given items to place
into pre -defined categories.
– Useful for verifying pre -existing information architecture – Useful for verifying category labels
www.rashmisinha.com
BayCHI UE BOF, July 02
3
Conducting card-sorting exercises • Rather involved method, requires a lot of setup.
Sorts themselves are fun to conduct.
• Main problem is in getting data into any form
where one can easily look at it, or share information with design team.
• Software for conducting card sorts. – IBM EZ Sort – NIST WebCAT
• Alternative method: Using online survey software
to conduct card sorts
www.rashmisinha.com
BayCHI UE BOF, July 02
Conducting card sorting exercises
Card sorting for designing an online travel guide • Example: Designing an online
travel guide to help users plan their trips to various locations.
• Purpose of card sort: – Lots and lots of items. How to structure the information.
• Items include – lodging, entertainment, local information, When to Go, Travel by Car/Air/Bus, Music Events, Hiking, Day Trips, Skiing, Diving, Golf, Emergency Info.
www.rashmisinha.com
BayCHI UE BOF, July 02
Card sorting using online survey software
• Check to make sure that
Each item (card) occupies a row
– – – –
each term is clear and unambiguous all the items that need to categorized were included set up items to be presented in random order create comprehensive instructions so that all participants have the same understanding of the process – Participants should be left alone during the sorting to avoid placing them under time pressure, but tester should remain available for answering questions
Each potential (unnamed category) is represented by a column User indicates category membership by checking box
• Conducting card sorting with online survey
software
– Most online survey software will do – Make sure it allowed you to present questions in matrix format, and allows downloading of each user’s data separately. www.rashmisinha.com
BayCHI UE BOF, July 02
www.rashmisinha.com
BayCHI UE BOF, July 02
4
Adding labels to categories
Looking at card-sorting data
At end, ask users to assign names / labels to each category they created Some users like to do this along the way (noting labels on piece of paper)
Cat-1 Cat-2 Cat-3 Cat-4 Cat-5 Look at data from one user at a time Name Category Entertainment Outdoor Lodging Practical Culture Info Before you go Before you go Fast Food At a glance you can Duties/Customs look down a Duties & Customs particular column, DIving DIving and see what items Nightclubs Nightclubs belong in it. ATMS's ATMS's Emergency info Emergency info You can also draw Etiquette Etiquette hierarchical Day Trips Day Trips structures showing created Museums & user Galleries Museums & GalleriesMuseums etc. categories Films Films Films & Movies Gay/Lesbian Hangouts Highlights Highlights Statistical analysis Highlights Architectural Sites Architecturalcan Sites create these
pictures for you across users.
www.rashmisinha.com
www.rashmisinha.com
BayCHI UE BOF, July 02
Closed Card Sorts
BayCHI UE BOF, July 02
Closed card sorts: when & how Category labels have been provided
• Category names were provided. • Users simply asked to assign items to categories. • Results are not informative about whether
number and structuring of categories was appropriate.
• Generally users are asked to give a “Yes / No”
response.
– One spin on this method is to ask users to rate “degree of certainty” in assigning items to categories. – This will help highlight problem categories / items. If users are very uncertain about assigning an item to a category, that should be probed further.
www.rashmisinha.com
BayCHI UE BOF, July 02
www.rashmisinha.com
BayCHI UE BOF, July 02
5
Rating “degree of certainty” in closed card sorts
Advantages of using online survey software for card sorts • Easy to carry out, no software to install and
figure out. Can be done using online survey software that are easily available.
• Can be done remotely • Trivial to get user-created categories into Excel
(or your favorite application) for qualitative / quantitative analysis.
• Can be used to conduct either open / closed card
sorts
• Can also be adapted to individual / group
sessions
www.rashmisinha.com
BayCHI UE BOF, July 02
Other issues about card sorting • Should each item be in one category • Should card sorts be done individually or with
groups
– group card sorts provides an energy and enthusiasm that makes light work of an otherwise tedious task – Can overshadow individual differences highlighting opinion of a few.
www.rashmisinha.com
BayCHI UE BOF, July 02
Other methods to tap into categorical structures • Open Card Sorting for generating information
architecture
• Closed Card Sorting for verifying information
architecture
• Generating Semantic Associations • Identifying dimensions used in categorization
• Combining card sorting with importance ratings is
a good idea.
www.rashmisinha.com
BayCHI UE BOF, July 02
www.rashmisinha.com
BayCHI UE BOF, July 02
6
Generating semantic associations • Helps understand what lies nearest to target items. Can be
conducted in conjunction with or independently of card sorting.
• Helpful in understanding problem categories and items • Semantic Association – What words do you associate below target words with Music: _____ _____ _____ ______ _____ _____ Diving: _____ _____ _____ ______ _____ _____
Generating semantic associations… • Category Member Association – What member items would you expect to find in below categories – Music: _____ _____ _____ ______ _____ _____ _____ _____ _____ ______ _____ _____ – Diving: _____ _____ _____ ______ _____ _____ _____ _____ _____ ______ _____ _____
• Category Association – What categories would you expect to find below words in Music: _____ _____ _____ ______ Diving: _____ _____ _____ ______ www.rashmisinha.com
BayCHI UE BOF, July 02
Identifying dimensions for categorizing
www.rashmisinha.com
BayCHI UE BOF, July 02
Talk structure: Personas…
• Present two random items and ask user what differentiates
• Card Sorting – Not used to its full potential – We need an easier way to do card sorts
• What differentiates pair below. List one or more ways that
• Persona Creation
them in your domain context. Repeat question till dimensions start repeating frequently.
they are different.
– Diving & Live Music Event: _______ ______ ______ – Shopping & Air Tickets: _______ ______ ______ – Motels & Restaurants: _______ ______ ______ • Good way to explore domain you are unfamiliar with • Some of the pairs will sound ridiculous!
www.rashmisinha.com
BayCHI UE BOF, July 02
– The process for creating personas needs to be demystified. Period. • Usability Tests – Its time to stop thinking of this monolithic entity called “usability tests”. There are different types of “usability tests”. – Tests for competitive benchmarking or for iterative design process can borrow from experimental design techniques.
www.rashmisinha.com
BayCHI UE BOF, July 02
7
Personas as “User Archetypes” • Personas is designing for one person. • Personas is a “User Archetype” who
represents needs and goals of many other users. – Who should that one person be?
• According to Cooper – Not the average user – Not a real user – Derived from background user research (interviews etc.)
Creating personas • Method – Conduct interviews with various stakeholders – Find patterns (inclusive cases) – Pick a nugget, interesting tidbit and build persona around it – Should represent large group in terms of goals, and personal characteristics • Problems with method: – Interviews are not economical way to find who representative users are – No tight coupling between user research & personas – Would two designers creating personas for same product create same personas? • Bottom line: Cooper’s persona creation method works well
when you have time on your hand, resources to interview many people, and skilled persona developers.
www.rashmisinha.com
BayCHI UE BOF, July 02
What is needed (from the horse’s mouth) “What is needed is a product definition tool that takes the guesswork out of the process—a tool that provides insight into what motivates people to use a product, so that well-grounded decisions can be made about features and how they are presented.” – Cooper.com
www.rashmisinha.com
www.rashmisinha.com
BayCHI UE BOF, July 02
Will market segmenting techniques serve any purpose? • Market segmentation is used to identify
clusters of people product can appeal to
– Mostly on basis of demographics – Sometimes on the basis of psychological variables (also called psychographic variables).
• Techniques can forecast marketplace
acceptance of products and services. Can also help convince executives to build product.
• Techniques not helpful in defining product
BayCHI UE BOF, July 02
www.rashmisinha.com
BayCHI UE BOF, July 02
8
Deconstructing marketing techniques • Questions focus on like / dislike of product
concept
– what do you think of vanilla coke or green Heinz ketchup?
• Statistical techniques used in market
segmentation cluster users according to demographic variables, not according to user needs from products.
• Concern themselves with product at a high
conceptual level. For interaction design, one needs to focus on specifics: How will product satisfy needs / goals.
www.rashmisinha.com
BayCHI UE BOF, July 02
Challenge: How to reliably identify user archetypes in cost-effective manner • Solution:
• Ground the personas in reality – Common critique of personas is that they are based on the designer’s imagination
• Focus on motives/needs of users • Be complementary to existing methods e.g.,
interview, observation
• Develop a concrete series of steps that any
designer can follow to develop personas
• Be reliable. When used by other designers, at
other times will lead to similar personas.
www.rashmisinha.com
BayCHI UE BOF, July 02
Designing personas for a Bay Area restaurant finder • Goals of site: Be the premier site for Bay Area
– Use survey techniques that are also used in market segmentation – Focus questions around user needs rather than what they simply like / dislike – Identify constellations of needs rather than clusters of users – Use this information as the kernel to build personas around
www.rashmisinha.com
What is needed is a method that will…
BayCHI UE BOF, July 02
residents & visitors to find restaurants of their choice.
• Design quick restaurant-finders (a la product
advisors) apart from conventional advanced search / browse options
• Conducted survey on features are used in finding
restaurants
– 45 respondents answered survey about importance of features in choosing restaurants – Food quality, décor, wine selection, cuisine, service
www.rashmisinha.com
BayCHI UE BOF, July 02
9
Steps in process
Design of survey
• Collect information via survey, interview,
observation
• Analyze survey data to identify underlying
factors, (types of needs)
• Identify some user archetypes • Verify these user archetypes with your qualitative
data (information from interview, observation etc.).
• Continue with your usual process of creating
personas
• Broad and shallow look • Try to be comprehensive • Identify features and get their importance rated
on same scale (5 or 7 point Likert scales)
– Example: How important are following restaurant features 1 2 3 4 5 – Service: Not similar
Very similar
• Don’t frame question in abstract way, instead tie
it to actual behavior
• Assure respondents about privacy, ask if they will
participate in follow-up interview
www.rashmisinha.com
BayCHI UE BOF, July 02
The survey: Importance ratings of 25 restaurant features
www.rashmisinha.com
BayCHI UE BOF, July 02
Main results of survey (based on 39 respondents) • Most respondents thinks quality of food, type of
cuisine, cleanliness is very important
• Few respondents think valet parking is important • Mixed feelings about features such buffet,
location, wine selection, waiting in lines etc.
• Most report going to restaurants with friends, or
as a couple. Fewer with family. Even fewer alone.
• Most report going 2 -3 times a week.
Next step: Identifying user archetypes based on patterns of needs… www.rashmisinha.com
BayCHI UE BOF, July 02
www.rashmisinha.com
BayCHI UE BOF, July 02
10
What would marketing techniques do? Food Quality
User ID 5240744 5240760 5241210 5241375 5241496 5241562 5243833 5243926 5243943 5245631 5253253 5261459 5263121 5264284 5264327 5264503 5264616 5264685 5264820 5264925 5265174
Décor 3
R e s t a u r a n t F e a t u r e s Wine ChildOutdoor Romanti Selectio Buffet friendly seating c n
User ID
Service
3
1
5
4
2
1
1
5 4 5
5 1 3
2 1 1
3 1 1
5 5 5
3 1 5
1 1 3
2 1 2
5 5 5
4 4 3
1 1 2
1 5 4
5 5 3
3 4 1
1 1 3
1 1 4
5 5 5
4 5 3
1 1 1
3 5 5
5 5 5
1 3 1
2 1 3
3 5 5
5 5
4 3
1 1
4 5
4 4
1 1
1 2
1 3
5 3 5
2 3 2
1 1 1
1 2 2
2 3 5
2 2 4
2 2 1
3 1 1
4 3 5
3 2 3
1 1 1
4 1 2
4 3 5
2 1 2
1 2 1
1 1 1
4 5 5
4 3 1
1 1 1
1 5 3
4 4 3
5 2 1
1 2 1
Design techniques need to identify cluster of needs
Cluster A
Cluster B
1 1 1
Food Quality
5240744 5240760 5241210 5241375 5241496 5241562 5243833 5243926 5243943 5245631 5253253 5261459 5263121 5264284 5264327 5264503 5264616 5264685 5264820 5264925 5265174
Décor 3
R e s t a u r a n t F e a t u r e s Wine ChildOutdoor Romanti Selectio Buffet friendly seating c n 1
5
4
2
1
1
5 1 3
2 1 1
3 1 1
5 5 5
3 1 5
1 1 3
2 1 2
5 5 5
4 4 3
1 1 2
1 5 4
5 5 3
3 4 1
1 1 3
1 1 4
5 5 5
4 5 3
1 1 1
3 5 5
5 5 5
1 3 1
2 1 3
3 5 5
5 5
4 3
1 1
4 5
4 4
1 1
1 2
1 3
5 3 5
2 3 2
1 1 1
1 2 2
2 3 5
2 2 4
2 2 1
3 1 1
4 3 5
3 2 3
1 1 1
4 1 2
4 3 5
2 1 2
1 2 1
1 1 1
4 5 5
4 3 1
1 1 1
1 5 3
4 4 3
5 2 1
1 2 1
1 1 1
Cluster A www.rashmisinha.com
BayCHI UE BOF, July 02
From cluster of needs to user archetypes • Features that are highly related to each other will
form part of the same constellation, or group of needs
• Factor analysis techniques can identify such
groupings.
• Each grouping can be said to represent a need
type.
• These are groupings of features not of users.
What need types emerge from restaurant survey?
www.rashmisinha.com
BayCHI UE BOF, July 02
Service
3
5 4 5
www.rashmisinha.com
Cluster B BayCHI UE BOF, July 02
User Archetype 1: Romantic, relaxed, dining experience… • Important Features – Good decor – Romantic setting – Relaxed service – Trendy – Caters to special Requirements (Low - fat, vegan etc.) – Type of crowd • Not Important Features – price, buffet-style – Many in 42- 51 age- group, spend more than $20, go out as a couple Food Quality, Type of Cuisine & Cleanliness does not appear in these archetypes because it was important to everyone.
www.rashmisinha.com
BayCHI UE BOF, July 02
11
User Archetype 2: Buffet style, with kids in tow… • Important Features – – – – – –
Buffet style dining Good for groups Child Friendly Relaxed Service Credit Card Portion Size
User Archetype 3: Meet for drinks, eat some food too… • Important Features – – – – – –
• Not Important Features – views, outdoor dining, people- watching, bar
Wine selection & bar Outdoor seating Easy access to public transport Live entertainment Type of crowd that visits place Good for groups
• Not Important Features – long Lines, finding Parking nearby
• Many in age group 32 -41, dine mostly with
family, once a week or lesser
www.rashmisinha.com
BayCHI UE BOF, July 02
After establishing some user archetypes
www.rashmisinha.com
BayCHI UE BOF, July 02
Advantages of technique
• Use user archetypes as “candidate personas”
• Inexpensive in terms of time and expertise
• Verify archetypes with results of interviews,
• A reliable method, two designers using same
• If survey is first step, then these can guide
• Retains creativity and fictional elements of
• Generate sample scenarios, make final decision
• Is complementary to current process of
observations
interviews (interview examples of each user archetype) on primary and secondary personas according to your design goals
method are likely to get similar results
creating personas, while grounding them in real data generating personas.
• Do not treat user archetypes generated through
survey as final word. Use them in conjunction with other information.
www.rashmisinha.com
BayCHI UE BOF, July 02
www.rashmisinha.com
BayCHI UE BOF, July 02
12
Talk structure: designing usability tests • Card Sorting
Usability tests for iterative design process & comparative benchmarking • Both scenarios require testing of more than one
– Not used to its full potential – We need an easier way to do card sorts
product
• Iterative Design requires multiple tests of same
software.
• Persona Creation – The process for creating personas needs to be demystified. Period.
– It also calls for creation of norms, baselines to compare subsequent versions against.
• Competitive Benchmarking calls for comparing
one product with other similar ones
• Usability Tests – Tests for competitive benchmarking or for iterative design can borrow from experimental design techniques. www.rashmisinha.com
BayCHI UE BOF, July 02
Not an experiment, has some features of it. Why? • So that one can make generalizable conclusions
• When comparing one product with another, test
becomes complex enough to benefit from experimental design.
www.rashmisinha.com
BayCHI UE BOF, July 02
6 steps to designing good usability tests
1: Identify the goal of the usability test; write it down
• Thinking about study design insures that potential
problems are identified beforehand, rather than later • Study designs are adaptable, 3-4 study designs can
be adapted for most testing situations • Helps make best use of limited time and resources
2: Identify factor you are most interested in (Independent Variable) 3: What changes do you expect in test results (Dependent Variabl es) 4: What random factors can influence results (Controlled Variables)
• Helps reduce number of users 5: Decide on a within-user / between -users study design. Take care of learning & fatigue effects 6: Make sure test design is practical (rapid and cost -effective), and gives you the results you need. Else redesign test. www.rashmisinha.com
BayCHI UE BOF, July 02
www.rashmisinha.com
BayCHI UE BOF, July 02
13
Example study: Examine usability of AskJeeves Digital Camera Advisor • Step 1: Identify goal: – To find effectiveness of Jeeves digital camera advisor • Method:
Compare effectiveness of Jeeves camera advisor to two popular camera advisors
Step 2: Identify Independent Variables • is the factor that undergoes a change • “type of camera advisor” is Independent Variable with three
levels
• MySimon, ActiveDecisions, & AskJeeves Independent Variable: Type of Camera Advisor Jeeves Camera Advisor
MySimon Camera Advisor
Active Decisions Camera Advisor
• In Iterative Design Process, “Version” is your Independent
Variable. Different versions (e.g., version 1, Version 1.5, Version 2) are your levels of Independent Variable
www.rashmisinha.com
BayCHI UE BOF, July 02
What if there is no Independent Variable? If you are interested in the usability of a particular software only… •
BayCHI UE BOF, July 02
Step 3: Identify dependent variables and decide how to measure each • These are the “things” you are measuring:
satisfaction, time to task, errors. The expectation is that they will change with changes in Independent Variable
Identify a relevant benchmark, even if benchmark is theoretical ideal (i.e., make one up) – the ideal system – older system that current software is replacing – competing software / process in the market
•
www.rashmisinha.com
Comparison with a relevant benchmark makes tests results more compelling and easier to interpret.
www.rashmisinha.com
BayCHI UE BOF, July 02
• Possible Dependent Variables: – Satisfaction: User comments, satisfaction questionnaire – Task success: time taken, number and type of errors
www.rashmisinha.com
BayCHI UE BOF, July 02
14
Step 4: What controlled variables could inadvertently influence results • Random factors that can effect results (need to control
them). E.g., network speed, testing conditions (light, monitor size), fatigue of subjects.
• These are like Independent Variables, except they are not
of current interest
• One needs to rule out effects of these variables by
Review types of variables: Independent, Dependent & Controlled • Independent Variable: Is the factor that
undergoes a change, the factor you are interested in. – Two versions of a website, three authentication systems
• Dependent Variable: Is the outcome of the test,
the thing that is effected by the independent variable
controlling them (equalizing their effect in different conditions).
• Controlled variables in current study – Testing Conditions: light, network speed, monitor size – Instructions given: whether few/ detailed / no instructions – User Variables: age, gender, previous web & computer experience
www.rashmisinha.com
– No of errors in finding item, time to get credit card authenticated by system
• Controlled Variables: factors that can cause
random changes in test results. Need to be controlled
– Speed of network connection, Previous web experience of user
www.rashmisinha.com
BayCHI UE BOF, July 02
Step 5: Within or between user design
BayCHI UE BOF, July 02
Within user study design
• Within user design: each user participates in all conditions. – Advantage: do not need to control for user differences – Disadvantage: Learning and Fatigue effects
9 users (A-I), within-users design
• Between-user design: each user participates in one condition. – Advantage: No learning and fatigue effects – Disadvantage: Need to control for between user differences
Users
Order 1
Order 2
Order 3
A
MySimon
Jeeves
Actv.Decs
B
Jeeves
Actv.Decs
MySimon
C
Actv.Decs
MySimon
Jeeves
D
MySimon
Jeeves
Actv.Decs
E
Jeeves
Actv.Decs
MySimon
Independent Var.: Type of Camera Advisor
F
Actv.Decs
MySimon
Jeeves
Jeeves Camera MySimon Camera Advisor Advisor Users A,B,C Users A,B,C
G
MySimon
Jeeves
Actv.Decs
H
Jeeves
Actv.Decs
MySimon
I
Actv.Decs
MySimon
Jeeves
• Practical decision – Depends on number of users you can afford, length of test etc. – Determines how easy it is to interpret test results, and how far one can generalize from test results 2 diff. designs
Within-Users design (3 users) Between-Users design (9 Users A,B,C users) www.rashmisinha.com
Users D,E,F
BayCHI UE BOF, July 02
Act. Dec. Camera Advisor Users: A,B,C Users: G, H, I
www.rashmisinha.com
•Each system occurs 3 times in Order 1,2, & 3 •Study design takes care of learning, fatigue effects •Design is very efficient, but there might be learning/fatigue effects
BayCHI UE BOF, July 02
15
Between users study design • 9 users (A-I), between-users design Users
Order 1
A
MySimon
B
Jeeves
C
Actv.Decs
D
MySimon
E
Jeeves
F
Actv.Decs
G
MySimon
H
Jeeves
I
Actv.Decs
www.rashmisinha.com
•Each system is used by 3 users, each user only uses 1 system •Study design does not pose any learning, fatigue effects •Tricky issue is to make sure users for different systems are comparable •Design is not very efficient, but easy to interpret results (if users are comparable)
BayCHI UE BOF, July 02
Mixed Study Design • 9 users (A-I), mixed between-users design Users
Order 1
Order 2
A
MySimon
Jeeves
B
Jeeves
Actv.Decs
C
Actv.Decs
MySimon
D
MySimon
Jeeves
E
Jeeves
Actv.Decs
F
Actv.Decs
MySimon
G
MySimon
Jeeves
H
Jeeves
Actv.Decs
I
Actv.Decs
MySimon
www.rashmisinha.com
•Each system is used by 6 users, each user only uses 2 system •Each system is used 3 times in Order 1 and 3 times in order 2 •Design compromises between demands of efficiency and fatigue/learning effects •Tricky issue is to make sure users for different systems are comparable
BayCHI UE BOF, July 02
For more information go to…
www.rashmisinha.com www.uzanto.com Or email me at [email protected]
www.rashmisinha.com
BayCHI UE BOF, July 02
16
Cost-effective methods for rapid user research and usability testing
Rashmi Sinha www.rashmisinha.com www.uzanto.com
www.rashmisinha.com
BayCHI UE BOF, July 02
Understanding your users • Finding out what issues are relevant
to them • Exploring their mental models
• Card Sorting – Not used to its full potential – We need an easier way to do and record data for card sorts • Persona Creation – The process for creating personas needs to be demystified. Period. • Usability Tests – Its time to stop thinking of this monolithic entity called “usability tests”. There are many types of usability tests with different levels of complexity. – Tests for competitive benchmarking or for iterative design process can borrow from experimental design techniques.
www.rashmisinha.com
BayCHI UE BOF, July 02
Understanding the categorical structures of the mind • Card sorting is one of a family of techniques to
tap into the user’s categorical structures. Animal
• Methods
(has skin, can move, eats)
Birds
– – – –
Interview Observation Ethnographic Methods Card Sorting & other methods of exploring categorization – Persona Creation
Fish
(wings, flies, feathers)
(swims, gills, fins)
Canary
Ostrich
Salmon
Shark
(sings, yellow)
(tall, does not fly)
(edible, pink)
(dangerous, gray)
• How semantic memory is organized
Focus on methods for information rich sites Semantic and categorical refer to similar things in present cont ext www.rashmisinha.com
BayCHI UE BOF, July 02
www.rashmisinha.com
BayCHI UE BOF, July 02
1
Why you need to understand categorical structures?
Example of individual differences
• Semantic organization builds up user
Sushi
expectancies regarding how things are organized.
Fish
– Design in accordance to these expectancies for better information architecture – Violate these expectancies and you leave the user lost and confused
Shark Aquarium
Diving
Grilled Salmon
Fish
• Do people’s categorical structures differ
Shark
from each other?
– Yes, semantic networks are determined by one’s experience, learning, culture etc. – but there is enough consistency across people for effective design
• Length of arrows implies semantic distance • Semantic memory is a network. Points are connected in
multiple direct and indirect ways.
• Degree of closeness between items might vary between
people, but we recognize the idiosyncratic association others make
www.rashmisinha.com
www.rashmisinha.com
BayCHI UE BOF, July 02
Semantic Distance and Proximity / Similarity / Distance Matrix
Semantic distance as basis of categorization • At root of all categorization techniques is
• How similar are the two (on scale of 1-5)
– salmon & shark:
1
2
3
4
5
Not similar
– canary & ostrich:
1
Not similar
Very similar
2
3
4
Salmon Shark 5 3 4 5 2 5 5
www.rashmisinha.com
5 Very similar
Self correlation Salmon Shark Canary Ostrich Trout Tiger Robin
Canary Ostrich Trout
BayCHI UE BOF, July 02
Tiger
question: “How far is A from B?”
• Proximity / similarity matrix can help generate
the most complex semantic networks (by using cluster analysis and other statistical techniques).
Robin Animal
5 4 3 2 3 2
(has skin, can move, eats)
5 2 4 4 1
5 4 4 2
BayCHI UE BOF, July 02
5 5 4
5 5
5
Birds
Fish
(wings, flies, feathers)
(swims, gills, fins)
Canary
Ostrich
Salmon
Shark
(sings, yellow)
(tall, does not fly)
(edible, pink)
(dangerous, gray)
www.rashmisinha.com
BayCHI UE BOF, July 02
2
Methods to tap into categorical structures • Open Card Sorting for generating information
architecture
Card Sorting: Understanding user’s mental models • Why: Helps you build site structure -good for deciding how to group information
• Closed Card Sorting for verifying information
architecture
• Generating semantic associations • Identifying dimensions used in categorization
• How: Write down each topic on an index card – Ask users to sort cards into piles – Have them name each category later
• Data obtained from card -sorting is indirect
enough that it avoids pitfalls of direct questioning
www.rashmisinha.com
BayCHI UE BOF, July 02
Kind of questions card sorting helps answer • Do the users want to see the information grouped
by: subject, process, business group, or type of information?
• What are the most important items to put on the
main menu?
• How many menu items should there be, and how
deep should it go?
• How similar or different are the needs of the
users throughout the organization?
www.rashmisinha.com
BayCHI UE BOF, July 02
www.rashmisinha.com
BayCHI UE BOF, July 02
Open and closed card sorting • Open Card Sorting: users given bunch of items to
sort those into categories. They are not given name or number of categories. – Useful for generating new or drastically redefining information architecture
• Closed Card Sorting:users given items to place
into pre -defined categories.
– Useful for verifying pre -existing information architecture – Useful for verifying category labels
www.rashmisinha.com
BayCHI UE BOF, July 02
3
Conducting card-sorting exercises • Rather involved method, requires a lot of setup.
Sorts themselves are fun to conduct.
• Main problem is in getting data into any form
where one can easily look at it, or share information with design team.
• Software for conducting card sorts. – IBM EZ Sort – NIST WebCAT
• Alternative method: Using online survey software
to conduct card sorts
www.rashmisinha.com
BayCHI UE BOF, July 02
Conducting card sorting exercises
Card sorting for designing an online travel guide • Example: Designing an online
travel guide to help users plan their trips to various locations.
• Purpose of card sort: – Lots and lots of items. How to structure the information.
• Items include – lodging, entertainment, local information, When to Go, Travel by Car/Air/Bus, Music Events, Hiking, Day Trips, Skiing, Diving, Golf, Emergency Info.
www.rashmisinha.com
BayCHI UE BOF, July 02
Card sorting using online survey software
• Check to make sure that
Each item (card) occupies a row
– – – –
each term is clear and unambiguous all the items that need to categorized were included set up items to be presented in random order create comprehensive instructions so that all participants have the same understanding of the process – Participants should be left alone during the sorting to avoid placing them under time pressure, but tester should remain available for answering questions
Each potential (unnamed category) is represented by a column User indicates category membership by checking box
• Conducting card sorting with online survey
software
– Most online survey software will do – Make sure it allowed you to present questions in matrix format, and allows downloading of each user’s data separately. www.rashmisinha.com
BayCHI UE BOF, July 02
www.rashmisinha.com
BayCHI UE BOF, July 02
4
Adding labels to categories
Looking at card-sorting data
At end, ask users to assign names / labels to each category they created Some users like to do this along the way (noting labels on piece of paper)
Cat-1 Cat-2 Cat-3 Cat-4 Cat-5 Look at data from one user at a time Name Category Entertainment Outdoor Lodging Practical Culture Info Before you go Before you go Fast Food At a glance you can Duties/Customs look down a Duties & Customs particular column, DIving DIving and see what items Nightclubs Nightclubs belong in it. ATMS's ATMS's Emergency info Emergency info You can also draw Etiquette Etiquette hierarchical Day Trips Day Trips structures showing created Museums & user Galleries Museums & GalleriesMuseums etc. categories Films Films Films & Movies Gay/Lesbian Hangouts Highlights Highlights Statistical analysis Highlights Architectural Sites Architecturalcan Sites create these
pictures for you across users.
www.rashmisinha.com
www.rashmisinha.com
BayCHI UE BOF, July 02
Closed Card Sorts
BayCHI UE BOF, July 02
Closed card sorts: when & how Category labels have been provided
• Category names were provided. • Users simply asked to assign items to categories. • Results are not informative about whether
number and structuring of categories was appropriate.
• Generally users are asked to give a “Yes / No”
response.
– One spin on this method is to ask users to rate “degree of certainty” in assigning items to categories. – This will help highlight problem categories / items. If users are very uncertain about assigning an item to a category, that should be probed further.
www.rashmisinha.com
BayCHI UE BOF, July 02
www.rashmisinha.com
BayCHI UE BOF, July 02
5
Rating “degree of certainty” in closed card sorts
Advantages of using online survey software for card sorts • Easy to carry out, no software to install and
figure out. Can be done using online survey software that are easily available.
• Can be done remotely • Trivial to get user-created categories into Excel
(or your favorite application) for qualitative / quantitative analysis.
• Can be used to conduct either open / closed card
sorts
• Can also be adapted to individual / group
sessions
www.rashmisinha.com
BayCHI UE BOF, July 02
Other issues about card sorting • Should each item be in one category • Should card sorts be done individually or with
groups
– group card sorts provides an energy and enthusiasm that makes light work of an otherwise tedious task – Can overshadow individual differences highlighting opinion of a few.
www.rashmisinha.com
BayCHI UE BOF, July 02
Other methods to tap into categorical structures • Open Card Sorting for generating information
architecture
• Closed Card Sorting for verifying information
architecture
• Generating Semantic Associations • Identifying dimensions used in categorization
• Combining card sorting with importance ratings is
a good idea.
www.rashmisinha.com
BayCHI UE BOF, July 02
www.rashmisinha.com
BayCHI UE BOF, July 02
6
Generating semantic associations • Helps understand what lies nearest to target items. Can be
conducted in conjunction with or independently of card sorting.
• Helpful in understanding problem categories and items • Semantic Association – What words do you associate below target words with Music: _____ _____ _____ ______ _____ _____ Diving: _____ _____ _____ ______ _____ _____
Generating semantic associations… • Category Member Association – What member items would you expect to find in below categories – Music: _____ _____ _____ ______ _____ _____ _____ _____ _____ ______ _____ _____ – Diving: _____ _____ _____ ______ _____ _____ _____ _____ _____ ______ _____ _____
• Category Association – What categories would you expect to find below words in Music: _____ _____ _____ ______ Diving: _____ _____ _____ ______ www.rashmisinha.com
BayCHI UE BOF, July 02
Identifying dimensions for categorizing
www.rashmisinha.com
BayCHI UE BOF, July 02
Talk structure: Personas…
• Present two random items and ask user what differentiates
• Card Sorting – Not used to its full potential – We need an easier way to do card sorts
• What differentiates pair below. List one or more ways that
• Persona Creation
them in your domain context. Repeat question till dimensions start repeating frequently.
they are different.
– Diving & Live Music Event: _______ ______ ______ – Shopping & Air Tickets: _______ ______ ______ – Motels & Restaurants: _______ ______ ______ • Good way to explore domain you are unfamiliar with • Some of the pairs will sound ridiculous!
www.rashmisinha.com
BayCHI UE BOF, July 02
– The process for creating personas needs to be demystified. Period. • Usability Tests – Its time to stop thinking of this monolithic entity called “usability tests”. There are different types of “usability tests”. – Tests for competitive benchmarking or for iterative design process can borrow from experimental design techniques.
www.rashmisinha.com
BayCHI UE BOF, July 02
7
Personas as “User Archetypes” • Personas is designing for one person. • Personas is a “User Archetype” who
represents needs and goals of many other users. – Who should that one person be?
• According to Cooper – Not the average user – Not a real user – Derived from background user research (interviews etc.)
Creating personas • Method – Conduct interviews with various stakeholders – Find patterns (inclusive cases) – Pick a nugget, interesting tidbit and build persona around it – Should represent large group in terms of goals, and personal characteristics • Problems with method: – Interviews are not economical way to find who representative users are – No tight coupling between user research & personas – Would two designers creating personas for same product create same personas? • Bottom line: Cooper’s persona creation method works well
when you have time on your hand, resources to interview many people, and skilled persona developers.
www.rashmisinha.com
BayCHI UE BOF, July 02
What is needed (from the horse’s mouth) “What is needed is a product definition tool that takes the guesswork out of the process—a tool that provides insight into what motivates people to use a product, so that well-grounded decisions can be made about features and how they are presented.” – Cooper.com
www.rashmisinha.com
www.rashmisinha.com
BayCHI UE BOF, July 02
Will market segmenting techniques serve any purpose? • Market segmentation is used to identify
clusters of people product can appeal to
– Mostly on basis of demographics – Sometimes on the basis of psychological variables (also called psychographic variables).
• Techniques can forecast marketplace
acceptance of products and services. Can also help convince executives to build product.
• Techniques not helpful in defining product
BayCHI UE BOF, July 02
www.rashmisinha.com
BayCHI UE BOF, July 02
8
Deconstructing marketing techniques • Questions focus on like / dislike of product
concept
– what do you think of vanilla coke or green Heinz ketchup?
• Statistical techniques used in market
segmentation cluster users according to demographic variables, not according to user needs from products.
• Concern themselves with product at a high
conceptual level. For interaction design, one needs to focus on specifics: How will product satisfy needs / goals.
www.rashmisinha.com
BayCHI UE BOF, July 02
Challenge: How to reliably identify user archetypes in cost-effective manner • Solution:
• Ground the personas in reality – Common critique of personas is that they are based on the designer’s imagination
• Focus on motives/needs of users • Be complementary to existing methods e.g.,
interview, observation
• Develop a concrete series of steps that any
designer can follow to develop personas
• Be reliable. When used by other designers, at
other times will lead to similar personas.
www.rashmisinha.com
BayCHI UE BOF, July 02
Designing personas for a Bay Area restaurant finder • Goals of site: Be the premier site for Bay Area
– Use survey techniques that are also used in market segmentation – Focus questions around user needs rather than what they simply like / dislike – Identify constellations of needs rather than clusters of users – Use this information as the kernel to build personas around
www.rashmisinha.com
What is needed is a method that will…
BayCHI UE BOF, July 02
residents & visitors to find restaurants of their choice.
• Design quick restaurant-finders (a la product
advisors) apart from conventional advanced search / browse options
• Conducted survey on features are used in finding
restaurants
– 45 respondents answered survey about importance of features in choosing restaurants – Food quality, décor, wine selection, cuisine, service
www.rashmisinha.com
BayCHI UE BOF, July 02
9
Steps in process
Design of survey
• Collect information via survey, interview,
observation
• Analyze survey data to identify underlying
factors, (types of needs)
• Identify some user archetypes • Verify these user archetypes with your qualitative
data (information from interview, observation etc.).
• Continue with your usual process of creating
personas
• Broad and shallow look • Try to be comprehensive • Identify features and get their importance rated
on same scale (5 or 7 point Likert scales)
– Example: How important are following restaurant features 1 2 3 4 5 – Service: Not similar
Very similar
• Don’t frame question in abstract way, instead tie
it to actual behavior
• Assure respondents about privacy, ask if they will
participate in follow-up interview
www.rashmisinha.com
BayCHI UE BOF, July 02
The survey: Importance ratings of 25 restaurant features
www.rashmisinha.com
BayCHI UE BOF, July 02
Main results of survey (based on 39 respondents) • Most respondents thinks quality of food, type of
cuisine, cleanliness is very important
• Few respondents think valet parking is important • Mixed feelings about features such buffet,
location, wine selection, waiting in lines etc.
• Most report going to restaurants with friends, or
as a couple. Fewer with family. Even fewer alone.
• Most report going 2 -3 times a week.
Next step: Identifying user archetypes based on patterns of needs… www.rashmisinha.com
BayCHI UE BOF, July 02
www.rashmisinha.com
BayCHI UE BOF, July 02
10
What would marketing techniques do? Food Quality
User ID 5240744 5240760 5241210 5241375 5241496 5241562 5243833 5243926 5243943 5245631 5253253 5261459 5263121 5264284 5264327 5264503 5264616 5264685 5264820 5264925 5265174
Décor 3
R e s t a u r a n t F e a t u r e s Wine ChildOutdoor Romanti Selectio Buffet friendly seating c n
User ID
Service
3
1
5
4
2
1
1
5 4 5
5 1 3
2 1 1
3 1 1
5 5 5
3 1 5
1 1 3
2 1 2
5 5 5
4 4 3
1 1 2
1 5 4
5 5 3
3 4 1
1 1 3
1 1 4
5 5 5
4 5 3
1 1 1
3 5 5
5 5 5
1 3 1
2 1 3
3 5 5
5 5
4 3
1 1
4 5
4 4
1 1
1 2
1 3
5 3 5
2 3 2
1 1 1
1 2 2
2 3 5
2 2 4
2 2 1
3 1 1
4 3 5
3 2 3
1 1 1
4 1 2
4 3 5
2 1 2
1 2 1
1 1 1
4 5 5
4 3 1
1 1 1
1 5 3
4 4 3
5 2 1
1 2 1
Design techniques need to identify cluster of needs
Cluster A
Cluster B
1 1 1
Food Quality
5240744 5240760 5241210 5241375 5241496 5241562 5243833 5243926 5243943 5245631 5253253 5261459 5263121 5264284 5264327 5264503 5264616 5264685 5264820 5264925 5265174
Décor 3
R e s t a u r a n t F e a t u r e s Wine ChildOutdoor Romanti Selectio Buffet friendly seating c n 1
5
4
2
1
1
5 1 3
2 1 1
3 1 1
5 5 5
3 1 5
1 1 3
2 1 2
5 5 5
4 4 3
1 1 2
1 5 4
5 5 3
3 4 1
1 1 3
1 1 4
5 5 5
4 5 3
1 1 1
3 5 5
5 5 5
1 3 1
2 1 3
3 5 5
5 5
4 3
1 1
4 5
4 4
1 1
1 2
1 3
5 3 5
2 3 2
1 1 1
1 2 2
2 3 5
2 2 4
2 2 1
3 1 1
4 3 5
3 2 3
1 1 1
4 1 2
4 3 5
2 1 2
1 2 1
1 1 1
4 5 5
4 3 1
1 1 1
1 5 3
4 4 3
5 2 1
1 2 1
1 1 1
Cluster A www.rashmisinha.com
BayCHI UE BOF, July 02
From cluster of needs to user archetypes • Features that are highly related to each other will
form part of the same constellation, or group of needs
• Factor analysis techniques can identify such
groupings.
• Each grouping can be said to represent a need
type.
• These are groupings of features not of users.
What need types emerge from restaurant survey?
www.rashmisinha.com
BayCHI UE BOF, July 02
Service
3
5 4 5
www.rashmisinha.com
Cluster B BayCHI UE BOF, July 02
User Archetype 1: Romantic, relaxed, dining experience… • Important Features – Good decor – Romantic setting – Relaxed service – Trendy – Caters to special Requirements (Low - fat, vegan etc.) – Type of crowd • Not Important Features – price, buffet-style – Many in 42- 51 age- group, spend more than $20, go out as a couple Food Quality, Type of Cuisine & Cleanliness does not appear in these archetypes because it was important to everyone.
www.rashmisinha.com
BayCHI UE BOF, July 02
11
User Archetype 2: Buffet style, with kids in tow… • Important Features – – – – – –
Buffet style dining Good for groups Child Friendly Relaxed Service Credit Card Portion Size
User Archetype 3: Meet for drinks, eat some food too… • Important Features – – – – – –
• Not Important Features – views, outdoor dining, people- watching, bar
Wine selection & bar Outdoor seating Easy access to public transport Live entertainment Type of crowd that visits place Good for groups
• Not Important Features – long Lines, finding Parking nearby
• Many in age group 32 -41, dine mostly with
family, once a week or lesser
www.rashmisinha.com
BayCHI UE BOF, July 02
After establishing some user archetypes
www.rashmisinha.com
BayCHI UE BOF, July 02
Advantages of technique
• Use user archetypes as “candidate personas”
• Inexpensive in terms of time and expertise
• Verify archetypes with results of interviews,
• A reliable method, two designers using same
• If survey is first step, then these can guide
• Retains creativity and fictional elements of
• Generate sample scenarios, make final decision
• Is complementary to current process of
observations
interviews (interview examples of each user archetype) on primary and secondary personas according to your design goals
method are likely to get similar results
creating personas, while grounding them in real data generating personas.
• Do not treat user archetypes generated through
survey as final word. Use them in conjunction with other information.
www.rashmisinha.com
BayCHI UE BOF, July 02
www.rashmisinha.com
BayCHI UE BOF, July 02
12
Talk structure: designing usability tests • Card Sorting
Usability tests for iterative design process & comparative benchmarking • Both scenarios require testing of more than one
– Not used to its full potential – We need an easier way to do card sorts
product
• Iterative Design requires multiple tests of same
software.
• Persona Creation – The process for creating personas needs to be demystified. Period.
– It also calls for creation of norms, baselines to compare subsequent versions against.
• Competitive Benchmarking calls for comparing
one product with other similar ones
• Usability Tests – Tests for competitive benchmarking or for iterative design can borrow from experimental design techniques. www.rashmisinha.com
BayCHI UE BOF, July 02
Not an experiment, has some features of it. Why? • So that one can make generalizable conclusions
• When comparing one product with another, test
becomes complex enough to benefit from experimental design.
www.rashmisinha.com
BayCHI UE BOF, July 02
6 steps to designing good usability tests
1: Identify the goal of the usability test; write it down
• Thinking about study design insures that potential
problems are identified beforehand, rather than later • Study designs are adaptable, 3-4 study designs can
be adapted for most testing situations • Helps make best use of limited time and resources
2: Identify factor you are most interested in (Independent Variable) 3: What changes do you expect in test results (Dependent Variabl es) 4: What random factors can influence results (Controlled Variables)
• Helps reduce number of users 5: Decide on a within-user / between -users study design. Take care of learning & fatigue effects 6: Make sure test design is practical (rapid and cost -effective), and gives you the results you need. Else redesign test. www.rashmisinha.com
BayCHI UE BOF, July 02
www.rashmisinha.com
BayCHI UE BOF, July 02
13
Example study: Examine usability of AskJeeves Digital Camera Advisor • Step 1: Identify goal: – To find effectiveness of Jeeves digital camera advisor • Method:
Compare effectiveness of Jeeves camera advisor to two popular camera advisors
Step 2: Identify Independent Variables • is the factor that undergoes a change • “type of camera advisor” is Independent Variable with three
levels
• MySimon, ActiveDecisions, & AskJeeves Independent Variable: Type of Camera Advisor Jeeves Camera Advisor
MySimon Camera Advisor
Active Decisions Camera Advisor
• In Iterative Design Process, “Version” is your Independent
Variable. Different versions (e.g., version 1, Version 1.5, Version 2) are your levels of Independent Variable
www.rashmisinha.com
BayCHI UE BOF, July 02
What if there is no Independent Variable? If you are interested in the usability of a particular software only… •
BayCHI UE BOF, July 02
Step 3: Identify dependent variables and decide how to measure each • These are the “things” you are measuring:
satisfaction, time to task, errors. The expectation is that they will change with changes in Independent Variable
Identify a relevant benchmark, even if benchmark is theoretical ideal (i.e., make one up) – the ideal system – older system that current software is replacing – competing software / process in the market
•
www.rashmisinha.com
Comparison with a relevant benchmark makes tests results more compelling and easier to interpret.
www.rashmisinha.com
BayCHI UE BOF, July 02
• Possible Dependent Variables: – Satisfaction: User comments, satisfaction questionnaire – Task success: time taken, number and type of errors
www.rashmisinha.com
BayCHI UE BOF, July 02
14
Step 4: What controlled variables could inadvertently influence results • Random factors that can effect results (need to control
them). E.g., network speed, testing conditions (light, monitor size), fatigue of subjects.
• These are like Independent Variables, except they are not
of current interest
• One needs to rule out effects of these variables by
Review types of variables: Independent, Dependent & Controlled • Independent Variable: Is the factor that
undergoes a change, the factor you are interested in. – Two versions of a website, three authentication systems
• Dependent Variable: Is the outcome of the test,
the thing that is effected by the independent variable
controlling them (equalizing their effect in different conditions).
• Controlled variables in current study – Testing Conditions: light, network speed, monitor size – Instructions given: whether few/ detailed / no instructions – User Variables: age, gender, previous web & computer experience
www.rashmisinha.com
– No of errors in finding item, time to get credit card authenticated by system
• Controlled Variables: factors that can cause
random changes in test results. Need to be controlled
– Speed of network connection, Previous web experience of user
www.rashmisinha.com
BayCHI UE BOF, July 02
Step 5: Within or between user design
BayCHI UE BOF, July 02
Within user study design
• Within user design: each user participates in all conditions. – Advantage: do not need to control for user differences – Disadvantage: Learning and Fatigue effects
9 users (A-I), within-users design
• Between-user design: each user participates in one condition. – Advantage: No learning and fatigue effects – Disadvantage: Need to control for between user differences
Users
Order 1
Order 2
Order 3
A
MySimon
Jeeves
Actv.Decs
B
Jeeves
Actv.Decs
MySimon
C
Actv.Decs
MySimon
Jeeves
D
MySimon
Jeeves
Actv.Decs
E
Jeeves
Actv.Decs
MySimon
Independent Var.: Type of Camera Advisor
F
Actv.Decs
MySimon
Jeeves
Jeeves Camera MySimon Camera Advisor Advisor Users A,B,C Users A,B,C
G
MySimon
Jeeves
Actv.Decs
H
Jeeves
Actv.Decs
MySimon
I
Actv.Decs
MySimon
Jeeves
• Practical decision – Depends on number of users you can afford, length of test etc. – Determines how easy it is to interpret test results, and how far one can generalize from test results 2 diff. designs
Within-Users design (3 users) Between-Users design (9 Users A,B,C users) www.rashmisinha.com
Users D,E,F
BayCHI UE BOF, July 02
Act. Dec. Camera Advisor Users: A,B,C Users: G, H, I
www.rashmisinha.com
•Each system occurs 3 times in Order 1,2, & 3 •Study design takes care of learning, fatigue effects •Design is very efficient, but there might be learning/fatigue effects
BayCHI UE BOF, July 02
15
Between users study design • 9 users (A-I), between-users design Users
Order 1
A
MySimon
B
Jeeves
C
Actv.Decs
D
MySimon
E
Jeeves
F
Actv.Decs
G
MySimon
H
Jeeves
I
Actv.Decs
www.rashmisinha.com
•Each system is used by 3 users, each user only uses 1 system •Study design does not pose any learning, fatigue effects •Tricky issue is to make sure users for different systems are comparable •Design is not very efficient, but easy to interpret results (if users are comparable)
BayCHI UE BOF, July 02
Mixed Study Design • 9 users (A-I), mixed between-users design Users
Order 1
Order 2
A
MySimon
Jeeves
B
Jeeves
Actv.Decs
C
Actv.Decs
MySimon
D
MySimon
Jeeves
E
Jeeves
Actv.Decs
F
Actv.Decs
MySimon
G
MySimon
Jeeves
H
Jeeves
Actv.Decs
I
Actv.Decs
MySimon
www.rashmisinha.com
•Each system is used by 6 users, each user only uses 2 system •Each system is used 3 times in Order 1 and 3 times in order 2 •Design compromises between demands of efficiency and fatigue/learning effects •Tricky issue is to make sure users for different systems are comparable
BayCHI UE BOF, July 02
For more information go to…
www.rashmisinha.com www.uzanto.com Or email me at [email protected]
www.rashmisinha.com
BayCHI UE BOF, July 02
16
Related Documents
Cost Effective - Quick Usability Testing
November 2019 11
Usability Testing
December 2019 12
Usability Testing
December 2019 15
Usability Testing Task Table
November 2019 11