Heuristic Evaluation

Assignment 2 - Heuristic Analysis 106CR - Designing for Usability March 2009 Thomas Davies 2179955

Table of Contents Introduction

4

Abstract

4

Information

4

Assignment

4

Tap Tap Revenge

4

iPhone

4

Implementation

5

Conducting the inspection

5

Inspection Method

5

Design Recommendations

5

Equipment

5

iPhone Game Heuristics

5

Research

5

Severity Rating

6

User Interface

6

Mobility Issues

7

Gameplay

7

Multiplayer

7

Issues User Interface

8 8

UI6 - Use standard user interface controls - Severity: Mod

8

UI10 - Audio should be relevant and support the game - Severity: Low

8

UI13 - Users should be kept informed about the system - Severity: Mod

8

UI14 - Do not force the user to remember previous actions - Severity: Mod

9

Mobility Issues

10

MI2 - Effectively deals with interruptions - Severity: High

10

MI4 - Games work with the surroundings - Severity: High

11 Heuristic Analysis

2

Gameplay GP1 - Game provides a tutorial - Severity: Low Multiplayer MP2 - The game allows users to play against each other - Severity: None

Recommendations User Interface

11 11 12 12

13 13

UI6 - Use standard user interface controls

13

UI13 - Users should be kept informed about the system

13

Mobility Issues

15

MI2 - Effectively deals with interruptions

15

MI4 - Games work with the surroundings

16

Gameplay GP1 - Game provides a tutorial

Reflection Usefulness of Heuristic Analysis

16 16

17 17

How successful was the method used?

17

Data Validity

17

Appendix iPhone Game Heuristics

References

18 18

21

Heuristic Analysis

3

Introduction Abstract For this assignment, I was given the task to complete a heuristic analysis on a mobile phone game to show how the analysis should be implemented and conducted. I selected ‘Tap Tap Revenge’ on the iPhone to see how developers are combatting many usability issues on this emerging mobile platform. Following the heuristic analysis I discovered that the application failed mostly on basic mobility-specific issues such as handling call and text message interruptions.

Information Assignment To discover certain usability issues the user will have to traverse through the application’s menu system. In this assignment, the order required in the menu system will be shown in brackets, for example: (One Player > Easy Tracks) - Select ‘One Player’ from the initial menu pane, and then select ‘Easy Tracks’. Tap Tap Revenge Developed by Tapulous, ‘Tap Tap Revenge’ is a music game similar to the popular console game, Guitar Hero. The game has been a success on the App Store with a reported 5 million downloads of the free game in 2008 (Arrington 2009). The popularity of the free game has allowed Tapulous to capitalise on specific markets; for example, releasing a paid Nine Inch Nails version of the game. iPhone ‘Tap Tap Revenge’ runs on the iPhone platform and though it has revolutionised the mobile phone industry, there are many usability issues that are a direct result of the operating system’s implementation. Therefore these issues can only be minimised and not resolved by 3rd party developers. One issue is that only one 3rd party application can be open, and utilising resources at once. For example, if the user selects a link in an application then it will quit, to allow Safari - the web browser - to open. This issue could be resolved by utilising the Web View in the SDK that allows websites to be viewed in that application. Related to this subject, no 3rd party applications are allowed to run in the background, so updates from the cloud, for example can not be received while the application is not active. Moreover, interruptions such as phone calls and text messages are handled by the operating system. Though text messages can be ignored and the user can continue playing the game; accepting phone calls quits the previously active application. To combat this, the application could automatically save, yet the issue of interruptions is not controlled by 3rd party developers.

Figure 1 iPhone physical controls

Heuristic Analysis

4

Implementation Conducting the inspection Inspection Method Nielsen recommends a two pass method of analysis, with the first pass ‘intended to get a feel for the flow of the interaction and the general scope of the system.’ while the second pass ‘allows the evaluator to focus on specific interface elements.’. Even though Nielsen believes conducting several passes through the application for each separate heuristic would be ‘tedious’, I feel for such a small project this method would work very successfully. Therefore, I will be looking for violations of small groups of heuristics at a time; for example, looking for consistency issues through one pass, then feedback in another. Moreover, Nielsen stated that ‘it would seem unnatural to evaluators to “overlook” usability issues that were not related to the one issue they were supposed to inspect for in a given pass.’ (Nielsen, Mack 1994: 29). To combat this problem, I will be noting down usability issues irrelevant to the one being inspected at that time, so I can investigate them further later. Each issue in the application will be given a severity rating based on the heuristic it contravenes. Design Recommendations During the final stage, I will provide suggestions for the designers in the form of written explanations or annotated mockups. Providing possible solutions to design flaws is vital as designers have ‘no information to help design the exact changes to the interface.’ (Nielsen, Mack 1994: 32) as heuristic analysis only highlights problems with systems and does not produce solutions. Equipment Known as a discount usability method, heuristic analysis requires no specific recording equipment, therefore only the items being tested are necessary. For the inspection, I used two devices that were both running the latest software. Device

Usage

Operating System

Tap Tap Revenge Version

iPhone 3G

For completing all the heuristics

2.2.1

1.6

2.2.1

1.6

iPod touch 1G During the multiplayer analysis where two devices were required

iPhone Game Heuristics Research The basis of these heuristics rely on Nokia’s ‘Mobile Game Playability Heuristics’ (2006), yet this document wasn’t the sole source. Though many of the mobility issues are apparent on all mobile devices, the uniqueness of the iPhone made many heuristics found in Nokia’s paper irrelevant. Therefore, during the process of creating the heuristics I consulted several other documents including Apple’s ‘iPhone Human Interface Guidelines’ (2008) as they contained many principles specifically related to distinctive iPhone features such as the multi-touch display and how it can only run one application at a time. These topics have been addressed in the ‘User Interface’ and ‘Mobility Issues’ heuristic sections. Moreover, Federoff’s thesis complimented these two documents as she ‘had visited a leading game development company to seek out the heuristics involved, as well as the usability evaluation processes in place’ (2002: 2). Federoff’s Heuristic Analysis

5

experience in the gaming industry therefore, allowed for this report to utilise some of the heuristics she believes need to be used in that sector. Severity Rating While conducting a heuristic analysis, a severity rating system is vital for the overall success of the project. Without a rating system, the designers would be unable to decide what usability issues require ‘the most resources to fix’ (Nielsen, Mack 1994: 47) such as time and financial costs. For this task, I chose a fourpoint rating system: Rating

Description

None

It is not a usability issue

Low

Cosmetic or minor usability issue that can easily be resolved by the user

Moderate

A significant usability issue that should be fixed before release as it could affect a large proportion of users

High

Usability issue makes the system unusable or will affect the majority of users and must be fixed immediately

Table 1 Severity Rating System During the analysis stage, these ratings will be judged on the three factors suggested by Nielsen; the frequency of the issue, the overall impact of the problem, and the persistence of the usability flaw (Nielsen, Mack 1994: 47). User Interface ID

Heuristic

Consistency & Interaction UI1

Navigation is consistent and logical. Well organised and easy to use workflow. Eg. Most frequent data at the top, and less frequent near the bottom.

UI2

Consistent typeface, style and size.

UI3

Simple and consistent colour palette.

UI4

Minimal text entry. Use lists and pickers instead.

UI5

Do not use zooming and panning when unnecessary.

UI6

Use standard user interface controls so interaction is predictable and familiar.

UI7

With knowledge of other iPhone applications, this behaves as expected.

UI8

Provide finger sized targets. Eg. At least a 44 point square.

Feedback UI9

Feedback should be provided immediately after user input. Eg. Tapping a button will highlight it to show it has accepted the input.

UI10

Audio should be relevant and support the game. However, can not be the primary source of feedback.

UI11

Animation should be subtle and provide meaningful feedback. It should enhance the user’s experience. Shouldn’t take over the application with extra flash.

Help UI12

The game should be obvious and require little or no help.

System State UI13

Users should be kept informed about the system. Eg. Show progress in a download.

UI14

Do not force the user to remember previous actions. Unless it is part of the gameplay.

Heuristic Analysis

6

Mobility Issues ID

Heuristic

Mobility MI1

Games should load within 5 seconds.

MI2

Effectively deals with interruptions. Should be able to pause and save current game. Receive a text or call should pause the game. Responding to text or answering call should save game.

MI3

Web content in applications should appear native. Properly optimised for the iPhone, including standard controls. Utilises the Web View.

MI4

Games work with the surroundings. Application should respect the device’s state. Eg. Ring/Silent switch.

Gameplay ID

Heuristic

Gameplay GP1

Provide a tutorial that teaches the user basic functions of the game and UI.

GP2

Users can view their progress and compare scores, either their own or from other users.

GP3

The game should be replayable so users can beat their previously best scores.

GP4

The user is in control of the game. Eg. Their interaction with the game controls everything.

GP5

‘A good game should be easy to learn and hard to master.’ Nolan Bushnell

Multiplayer ID

Heuristic

Multiplayer MP1

Networking should be hidden from the user, unless connection is lost.

MP2

The game allows users to play against each other.

MP3

Finding and maintaining friendship is possible. Eg. Bookmarking a friends account in the game.

MP4

Games can be played over a local network when playing with friends in a close vicinity.

MP5

Misuse of the multiplayer game can be reported. Eg. Bad language and offensive messages.

Heuristic Analysis

7

Issues User Interface UI6 - Use standard user interface controls - Severity: Mod Though Apple states in the ‘iPhone Human Interface Guidelines’ (HIG) that in immersive applications ‘it is often appropriate’ to use nons[tandard controls as it ‘strengthens the user’s sense of entering the world of the application’ (Apple Inc. 2008: 23) the user should still be able to understand how controls work. The application contravenes this heuristic in the ‘Options’ pane (Options), with the settings of ‘Arrows’ and ‘Location Features’. The two options look different to other options as the pre-set choices are already displayed as highlighted in Figure 2.

Figure 2 ‘Options’ pane

UI10 - Audio should be relevant and support the game - Severity: Low Though audio feedback should not be the sole feedback mechanism, it should reinforce what the user sees on screen. The only issue here is that the same sound is used throughout the application, whether the user is going forward or backwards in the hierarchical menu system. Using the same sound makes it difficult for the user to match the audio with the animation seen on screen. UI13 - Users should be kept informed about the system - Severity: Mod There are several instances in ‘Tap Tap Revenge’ where the user would be unaware of what is happening; usually when the application is connecting to the internet to download data or new songs. To view the leaderboard, the user has to complete a song and then select the leaderboard option on the ‘Post-Game Statistics’ pane (Post-Game Statistics > See The Leaderboard). This will take the user to a blank holding screen shown in Figure 3, where the user stays until data is downloaded from the internet for the global leaderboard. The blank screen communicates nothing to the user, and may even suggest the application has crashed. 1

1

This screen does not contain any form of progress indicators to inform the user of how long it will take.

Figure 3 Loading the leaderboard Heuristic Analysis

8

The issue of the system state is more prevalent when downloading more songs to play with (Download More Tracks). When downloading a single track the user is shown the first screenshot in Figure 4, while downloading multiple tracks displays the second screenshot. 1

2

Figure 4 Downloading tracks in ‘Tap Tap Revenge’ Though the second screenshot is more helpful as it informs the user of what they are downloading, both fail to communicate ways to carry on playing the game while the downloads continue in the background. Selecting either the ‘Back’ or ‘Home’ button will take the user to the respective screens, yet there is no way to go back to these download panes, suggesting that the downloads stop after the user leaves the screens. 1

2

1

Activity Indictator to show progress of an unknown duration

2

Progress View to show progress of a known duration

Figure 5 Two types of progress indicators Moreover, using two different progress indicators to represent the same download can confuse users as the controls mean different things as shown in Figure 5. As well as these issues, the application allows the user to download songs over different network types; GPRS, EDGE, 3G, and WIFI. While the last two are very fast, GPRS and EDGE can be slow and will leave the user waiting for a very long time. UI14 - Do not force the user to remember previous actions - Severity: Mod Downloading songs via the ‘Download More Tracks’ section will remove it from the available songs list, ensuring the user does not re-download a song and waste valuable storage space. However, downloading the promoted song (New Music) from the top button on the initial menu pane does not remove the option to re-download it at a later date.

Heuristic Analysis

9

1

1

The download option is still active after the download meaning the user will be able to re-download the song, wasting storage space.

Figure 6 Downloading the promoted song

Mobility Issues MI2 - Effectively deals with interruptions - Severity: High If a call comes in and is rejected, the caller screen is replaced by the ‘Post-Game Statistics’ pane, effectively finishing the game before it is over. The game does not pause or save, nor return the user to its previous state. Moreover, attempting to initiate another game will result in it immediately going to the ‘PostGame Statistics’ pane even though the new game hasn’t even started. Requires a restart of the application to work properly again.

Figure 7 Answering a call during gameplay Heuristic Analysis

10

Answering a phone call during gameplay will close the application - due to Apple’s one application at a time restriction - and then reloads ‘Tap Tap Revenge’ after the call has ended. However, the application doesn’t return the user to its previous state, rather creates a new instance of the application. A text message doesn’t pause the game, but strangely carries on in the background, behind the text alert dialog. Tapping the ‘Close’ button will remove the alert, allowing the user to continue but by then the game could have finished. MI4 - Games work with the surroundings - Severity: High Although the game respects the audio state that the user selects, changing the audio in gameplay mode (Switching the ‘Ring/Silent’ switch to the opposite state) ends the game and displays the Post-Game Statistics pane. The issue with this is that if the local environment changes, the user is unable to alter the audio immediately without ending the game. The user can press the volume buttons on the side, but that takes four seconds to complete, and will act differently depending on which state they are in, as shown in Table 2. Silent State

Ring State

Volume at its highest

Silent

Loud

Volume at its lowest

Silent

Quiet

Table 2 Volume Controls Therefore, the application violates guidelines set out in the HIG stating ‘that users can always use the volume buttons to silence any sound, regardless of the position of the Ring/Silent switch.’ (Apple Inc. 2008: 49) as is the case in Apple’s ‘iPod’ application.

Gameplay GP1 - Game provides a tutorial - Severity: Low ‘Tap Tap Revenge’ provides a quality tutorial; teaching the user how to play the game. This is achieved by providing visual cues and a voiceover to emphasise the controls.

Figure 8 Visual cues showing the ‘Tap’ and ‘Shake’ controls Being difficult to discover however, is the only issue, requiring the user to traverse through two layers of the menu system (One Player > Play Tutorial) as well as being placed at the end of the list. As highlighted by Figure 9, the more frequently used data should be at the top (Apple Inc. 2008), and while Tapulous’ current arrangement follows these guidelines, the discoverability of the tutorial is harmed. As tutorials will be used more during initial runs of the application, making it more accessible during these instances will improve user involvement as they will be aware of the game controls.

Heuristic Analysis

11

More frequently used or read

Less frequently used or read

Figure 9 The ‘One Player’ pane

Multiplayer MP2 - The game allows users to play against each other - Severity: None The application has two modes in which the user can play against others. One takes advantage of the multi-touch display (Two Player), which therefore equates to a multi-user display allowing users to play against each other simultaneously. The other allows users to compete against others across the world in the online mode (Play Online).

The 'Two Player' Mode

The 'Play Online' Mode

Figure 10 The different multiplayer modes

Heuristic Analysis

12

Recommendations User Interface UI6 - Use standard user interface controls 1

1

Non-standard UI controls. User may be unsure how to interact with this setting

2

Standard Switch control

Used to offer the user two completely opposite - but known - choices. Example: On or Off

Uses the Switch control because iPhone users know how to interact with them. Customised to fit the aesthetic style of the application

2

Previous Version

6

Recommended Version

UI13 - Users should be kept informed about the system 1

The screen displayed while waiting for the application to load the global leaderboard. The pane is useless as it communicates nothing to the user

Standard Activity Indicator

2 1

Used to show progress of unknown duration and when connecting to the network. Used for only a few seconds.

Uses the Activity Indicator because the application is only downloading text, so shouldn't take more than a few seconds Allows the user to stop downloading and instantly view the personal leaderboard - results that are already stored on the iPhone Previous Version

3

2

3

6

Recommended Version

Heuristic Analysis

13

Heuristic Analysis

14

7

2

Recommended Version

Previous Version

1

6

Status icon shows that the GPRS network is in use

9

8

Transparent message to inform the user that the download is complete and ready

Warns the user that the download will take a long time. States 'slow speed' instead of 'GPRS' because user may not be aware of what GPRS means

7

Ensures the user is fully aware of their actions if they select this. No ambiguity

5

4

Makes the user aware of what they are downloading Allows the user to still play the game, while the text underneath assures them the download will continue

3

Progress View to show progress of a known duration

2

Uses a Progress View because the game now gives an estimated duration

Activity Indictator to show progress of an unknown duration

1

8

3

6

4

Recommended Version

9

Recommended Version

5

Heuristic Analysis 6

The text message alert that is handled by the iPhone operating system

4

3

Smaller text gives the user some real life context, reminding them of when they were playing the game

'Game Paused' text blends in with the gameplay UI to emphasise its relation to the game

2

Example: Answering a call or replying to a text

Replaces the advert at the top of the initial menu pane if the game had been interrupted previously. Makes is easy for the user to continue a previous game, while still being able to select other options

Text Message Recommended Version (2)

5

Call Interruption Recommended Version

2

1

1

6

4

6

5

Selecting 'Close' from the alert will then display this dialog box. Could be too many dialog boxes for the user to interact with though for just one process

The text message alert that is handled by the iPhone operating system

Text Message Recommended Version (1)

3

Mobility Issues

MI2 - Effectively deals with interruptions

15

MI4 - Games work with the surroundings The application should ensure that when the user alters the audio via the ‘RIng/Silent’ switch the game will continue rather than end. The surrounding environment can change, but the user may still want to continue playing the game, therefore the application shouldn’t allow such a bug to remain.

Gameplay GP1 - Game provides a tutorial 1

1

Access to the tutorial on the initial menu screen. Helps new users learn how to play the game. Will only be at the top of the menu until the user plays it or has used the game more than 5 times If the user had a game interrupted, then the that will be displayed over the 'Play Tutorial' option, as it has a higher priority

2

3

User is informed that the tutorial is complete Says that the tutorial will be in the 'One Player' section and not on the initial menu screen. Still available just in case they need some more assistance Recommended Version

2 3

6

Recommended Version

Heuristic Analysis

16

Reflection Usefulness of Heuristic Analysis How successful was the method used? The use of heuristic analysis was very successful as it highlighted several major design flaws in ‘Tap Tap Revenge’. Though this method will never find all the usability issues in an application and should be used in conjunction with user testing, the quick turnaround and minor financial cost of heuristic analysis means that it is a viable and important mode of usability evaluation. If the system is domain dependent, the expert evaluators may be unable to discover domain specific issues, even if they have been provided with industry knowledge beforehand. This was proven to be a problem during Nielsen’s tests, where users testing the system were able to find a flaw that the expert evaluators were not (Nielsen, Mack 1994: 46). The ISO 9241-11 definition of usability focuses on three indicators; effectiveness, efficiency and satisfaction of the interaction. While according to a lecture delivered as part of the 106CR module, learnability ensuring the user habituates to a given task - should be listed with those indicators as well (Every 2009). Heuristic analysis however, is only able to provide good results for effectiveness and learnability, as these can be measured without user input. Yet with efficiency and satisfaction, user involvement is critical as expert evaluators will not be the intended users of the application and therefore will be unable to judge these criteria. Data Validity As only one evaluator was used for this heuristic analysis and the fact that I am an undergraduate student means the validity of the results could easily be questioned. Unqualified evaluators pose an issue as they may not have the skill set required to follow and understand the criteria that the system is being judged on, while for user testing, the user can never be wrong, so being unqualified in that evaluation method is irrelevant. Moreover, Nielsen casts doubt on using a single evaluator suggesting that the severity ratings from them could be ‘too unreliable to be trusted.’ (Nielsen, Mack 1994: 55), and this could adversely affect the system as designers could waste time and money on features that only one evaluator believes are flawed. Therefore, as Nielsen discovered that ‘single evaluators found only 35 percent of usability problems in the interfaces.’ he recommended using between 5 and 7 evaluators as they found roughly 75% of all usability issues (Nielsen, Mack 1994: 33).

Heuristic Analysis

17

Appendix iPhone Game Heuristics

iPhone Game Heuristics User Interface ID

Heuristic

Notes

Severity

Consistency & Interaction UI1

Navigation is consistent and logical. Well organised and easy to use workflow. Eg. Most frequent data at the top, and

None Low Mod High

less frequent near the bottom. UI2

Consistent typeface, style and size.

UI3

Simple and consistent colour palette.

UI4

Minimal text entry. Use lists and pickers

None Low Mod High

None Low Mod High instead.

None Low Mod High UI5

Do not use zooming and panning when unnecessary.

None Low Mod High UI6

Use standard user interface controls so interaction is predictable and familiar.

None Low Mod High UI7

With knowledge of other iPhone applications, this behaves as expected.

None Low Mod High UI8

Provide finger sized targets. Eg. At least a 44 point square.

None Low Mod High Feedback UI9

Feedback should be provided immediately after user input. Eg. Tapping a button will highlight it to

None Low Mod High

show it has accepted the input. UI10 Audio should be relevant and support the game. However, can not be the primary source of feedback.

None Low Mod High

UI11 Animation should be subtle and provide meaningful feedback. It should enhance the user’s experience. Shouldn’t take

None Low Mod High

over the application with extra flash.

Heuristic Analysis

18

ID

Heuristic

Notes

Severity

Help UI12 The game should be obvious and require little or no help.

None Low Mod High System State UI13 Users should be kept informed about the system.

None Low Mod High

Eg. Show progress in a download. UI14 Do not force the user to remember previous actions. Unless it is part of the

None Low Mod High

gameplay.

Mobility Issues ID

Heuristic

Notes

Severity

Mobility MI1

Games should load within 5 seconds.

None Low Mod High MI2

Effectively deals with interruptions. Should be able to pause and save

None Low Mod High

current game. Receive a text or call should pause the game. Responding to text or answering call should save game. MI3

Web content in applications should appear native. Properly optimised for the

None Low Mod High

iPhone, including standard controls. Utilises the Web View. MI4

Games work with the surroundings. Application should respect the device’s

None Low Mod High

state. Eg. Silent/Ring switcher.

Gameplay ID

Heuristic

Notes

Severity

Gameplay GP1 Provide a tutorial that teaches the user basic functions of the game and UI.

None Low Mod High GP2 Users can view their progress and compare scores, either their own or from other users.

None Low Mod High

GP3 The game should be replayable so users can beat their previously best scores.

None Low Mod High

Heuristic Analysis

19

ID

Heuristic

Notes

Severity

GP4 The user is in control of the game. Eg. Their interaction with the game

None Low Mod High

controls everything. GP5 A good game should be easy to learn and hard to master.’ Nolan Bushnell

None Low Mod High

Multiplayer ID

Heuristic

Notes

Severity

Multiplayer MP1 Networking should be hidden from the user, unless connection is lost.

None Low Mod High MP2 The game allows users to play against each other.

None Low Mod High MP3 Finding and maintaining friendship is possible. Eg. Bookmarking a friends account in the game.

None Low Mod High

MP4 Games can be played over a local network when playing with friends in a close vicinity.

None Low Mod High

MP5 Misuse of the multiplayer game can be reported. Eg. Bad language and offensive messages.

None Low Mod High

Heuristic Analysis

20

References ISO 9241-11: Ergonomic requirements for office work with visual display terminals (VDTs) - Part 11: Guidance on usability. 1998. Apple Inc., 2008. User Interface Design for iPhone Applications. [Online Video]. Cupertino, California: Apple Inc. Apple Inc., 2008. iPhone Human Interface Guidelines. 3rd ed. Cupertino, California: Apple Inc. Arrington, M., 2009. Tapulous Email. [Blog] Available from: http://www.techcrunch.com/ 2009/01/09/leaked-investor-email-from-tapuloussay-breakeven-december-more-funding-newproducts/ [accessed 15 February 2009 ]. Every, P., 2009. Heuristic Inspection. [Lecture] Federoff, M., 2002. Heuristics and Usability Guidelines for the creation and evaluation of fun in video games. Indiana University. Nielsen, J., Mack, R., 1994. Usability Inspection Methods. 1st ed. New York: John Wiley & Sons, Inc. Nokia, 2006. Mobile Game Playability Heuristics. 1st ed. Nokia User Focus, 2009. ISO 9241 Part 11: Guidance on usability. [Webpage] Available from: http://www.userfocus.co.uk/ resources/iso9241/part11.html [accessed 2 March 2009].

Heuristic Analysis

21