Evaluation

Evaluation
 Electronic
Attendance
System
 Joseph
Ng
Chow
 





[email protected]


Victoria
Mui















[email protected]


Brian
Shim
 








Veronica
Wong
























[email protected]


[email protected]


DAVE
DEARMAN
 CSC318
–
THE
DESIGN
OF
INTERACTIVE
COMPUTATIONAL
MEDIA
 NOVEMBER
26,
2008
 


Table of Contents EVALUATION TECHNIQUES 

3 

COGNITIVE WALKTHROUGH  USABILITY TESTS  HEURISTIC EVALUATION  PARTICIPANTS

3  4  6  7

RATIONALE 

7 

CHOICE OF PARTICIPANTS  CHOICE OF TECHNIQUES/TASKS  CHOICE OF MATERIALS 

7  7  9 

EVALUATION DATA AND RESULTS 

9 

COGNITIVE WALKTHROUGH RESULTS  QUALITATIVE DATA  USABILITY TEST RESULTS  QUALITATIVE DATA  QUANTITATIVE DATA  HEURISTIC EVALUATIONS RESULTS  QUALITATIVE DATA  QUANTITATIVE DATA  ERROR-STEP TEST 

9  9  10  10  11  11  11  12  12 

DATA ANALYSIS AND DISCUSSION 

15 

COGNITIVE WALKTHROUGH  USABILITY TEST  HEURISTIC EVALUATION  ERROR-STEP TEST

15  16  19  22


DESIGN IMPLICATIONS AND IMPROVEMENTS 

23 

CRITIQUE OF EVALUATION PLAN 

25 

KNOWING WHAT WE KNOW NOW, WHAT WOULD WE HAVE DONE DIFFERENTLY?  WHAT WOULD WE HAVE DONE IF WE HAD MORE RESOURCES? 

25  26 

Page

2

Evaluation Techniques The objectives of our evaluation are to learn: •

Whether or not the design requirements were met, and

•

How well the prototype met the expectations of the design requirements.

To answer these questions, we used usability tests, cognitive walkthroughs, and heuristic evaluation techniques. We performed 2 separate sets of tests on our participants; usability tests for the end users, and cognitive walkthroughs and heuristic evaluations for the HCI experts. To ensure that the results of one test would not influence the results of the other, our second group of participants either did a cognitive walkthrough or a heuristic evaluation. Although tests were divided, we felt that observing the number of errors and testing the result was both important and generic enough that it needed to be applied to both user groups. This will be discussed later in the Evaluation of Results section.

Cognitive Walkthrough Input Interaction Task: The high-level interaction task was “taking the attendance”. Since this was what the system was designed to do, it was the primary task that potential users performed. Action Sequence: 0. Take the attendance 1. Log in with employee number and PIN 1.1. Use the stylus to activate the employee number input field 1.2. Input given employee number 1.2.1 Tap the numbers on the number pad with the stylus 1.3. Tap the numbers on the number pad with the style for PIN 1.4. Hit enter with the stylus 2. Mark each student in the class, present, absent, or late 2.1. Call out the student’s name 2.2. Tap the bubble next to each student’s name that corresponds to their attendance status 2.3. Repeat 2.1-2.2. for each student on the class list

Page

3

2.4. Click on the “Next” button to go to the next page of students, if necessary 3. Submit the attendance. 3.1. Click on the “Submit Attendance” button 4. Correct the attendance 4.1. Click on the “Modify” button next to the student whose attendance needs to be corrected 4.2. Tap on the bubble that corresponds to their attendance status 4.4. Click on the “Update Attendance” button 5. Logout Identify the users: As we mentioned above, this test was performed on HCI experts. We assumed that they are familiar with HCI principles, but have no knowledge of the domain. To educate them, we gave an overview of our project—a summary that is similar to our project description from G4, and a description of the existing attendance system. Prototype: We used our physical prototype in conjunction with our functional screens that we developed for G3.

Walkthrough While one of us addressed each step of the task sequence and played the Wizard of Oz, the remainder of our team made observations as the participant walked through the system. To debrief, each participant was asked to answer the four questions used to develop a believability story based on the interface, their knowledge of HCI and their understanding of the users:

1. Is this a task that you would want to perform? 2. From the interface, can you see how the system can allow you to perform the task? 3. From the interface, do you know if the system would produce the correct result? 4. Does the system provide informative feedback of the system’s response to your request?

Usability Tests We designed our usability tests with respect to our usability requirements. Recall our requirements were:

Page

4

1. Product design and interface must be intuitive so users can use the system with little or no special training. To see whether our interface was easy to learn, rather than walking them through our system as in the cognitive walkthrough, we had the testers explore the physical prototypes for 3 minutes. If the interface was intuitive enough, they would have little-to-no problems completing the tasks we asked them to perform. 2. Allows users to access features and menus through minimal system interactions; these include: screen touches, mouse clicks, button presses… etc. The set of activities that testers were asked to perform were tasks that teachers would normally carry out with their attendance. We compiled this list from the observations we conducted in earlier research. For each task, we recorded the time and the number of clicks that the participants had to make in order to complete it. The tasks were: Daily Attendance -

Login

-

Taking the attendance

-

Change student attendance records

-

Logout

Emergency Attendance -

Accept global announcement; acknowledge lockdown

-

Taking the attendance

-

Adopting a missing student

-

Contacting the office for EMS

3. Visual feedback of system activities. To test the visual feedback of system activities, we performed another set of tests. As the participants performed a task, we would interrupt them with another assignment. Once they addressed the interruption, they were required to go back the original task. Were they able to re-orientate themselves? Did they use the “back” button? Did they end up at screen where they did not intended?

Page

5

The participants were randomly placed into 2 equal sized groups. One group took attendance with a regular class list with no default status for students; the other group took attendance with the same list, but with each student’s status defaulted to “Present”. The distraction involved stopping the participant part way through the class list, and asking them to work on a Sudoku puzzle. After a few minutes, the participant was urged to continue with the attendance. We measured how long it took for them to reach the same point in the class list where they left off before the distraction. The results in both groups were analyzed to determine the effect of the EAS’s layout.

Heuristic Evaluation Input Before the evaluation, using the storyboard we developed in G4, the participants were given an overview of the project, a description of the existing system, and the high-level intentions of our proposed solution. We left out the rationale behind the design of our interface so that it would not influence their judgment. If there was component that they could not rationalize for themselves, it may be a bug. Our physical prototype and functional screens were used for the evaluations.

Evaluations Participants were given the prototype, and a rubric based on five main criterions/heuristics with which they used to evaluate the interface. From the Nielsen’s list, the heuristics that we deemed to be the most important are: 1. Match between the system and real world; e.g. was it a natural way to take the attendance? 2. Visibility of system status; e.g. can the user tell which class the attendance is for? 3. Error prevention; e.g. how difficult it is to forget to mark a student present? 4. Consistency and standards; e.g. changes between screens aren’t jarring

5. Aesthetics and minimalist design; e.g. is anything on the interface unnecessarily? Unhelpful? A more detailed rubric can be found in the appendix.

Page

6

Debriefing and information gathering; Severity To debrief, the experimenter and tester went over the evaluation together to rank the severity of the bugs that were found. Discussions about possible solutions were also part of the debriefing.

Participants In our study, we used randomly selected participants between the ages of 20 and 45. The research subjects in the cognitive walkthrough comprised of 4 male HCI experts. Similarly, the heuristic evaluation was applied to 4 other male HCI experts. As well, 4 teachers (3 male, 1 female) were selected to participate in the usability tests; however, all participants had their number of errors recorded and analyzed.

Rationale Choice of Participants It was important to include participants from our end-users because they would be the ones who would be using the product when completed. Furthermore, having used the existing system, their insights to our proposed solution in comparison were invaluable. Due to the difficulty in gathering participants from our end-user group, we supplemented the study with HCI experts. They lacked the domain background of our end-users, but that was not critical in the more technical tests. We could have also used their perspectives as users who were being exposed to the attendance system for the first time

Choice of Techniques/Tasks Given the limited resources that we had, we chose to use the discounted techniques of a cognitive walkthrough and a heuristic evaluation. We had neither the time nor the participants to do some of the more expensive evaluations; however, due to the spoon-fed nature of the cognitive walkthrough, it was necessary to do a more authentic evaluation to see how the prototype met some of the more technical requirements of usability.

Page

7

Cognitive Walkthrough: The task of taking the attendance is what the system was designed to accomplish; it is only natural that we asked our participants to do it. In our choice of participants and the task, we were able to see whether or not the functional requirements were met. In particular, whether it kept all the functionality of the old system, and whether it was an improvement to the old system (e.g. was it less prone to errors? Was it faster to complete the task?). Usability Test: As an extension of completing a task with a minimal amount of clicks, we wanted to test for the percentage of errors they made. We expected the average number of errors would be less than 5% when taking the attendance as we believed our interface was fairly intuitive, but not quite ready for production. To determine if this was a correct assumption, we applied a t-test to the results. Interruptions were made to test the visual and navigation aids in the system. We learned from our initial research that distractions while the teacher was in the middle of taking the attendance were common. Consequently, the teacher may forget who the last student that they recorded was. In the worst case, the teacher would incorrectly mark a student’s status as a result. Similarly, feedback from earlier studies had suggested that there may be advantages if a default “present” status was set for each student. From this, we designed a test to see whether or not the layout of the interface would impact the time it took to complete the attendance. Since the time to complete the attendance is proportional to the number of students it would not have been accurate to test the total time to complete the attendance. We chose instead to observe the time it took an individual to recover from distraction and return to where they were in the class list before interruption. This is the motivation behind why we wanted to see if the tester experienced any difficulty in recovering from the interruption. To analyze this, we chose a two sample t-test to apply to the data as it accurately shows the impact of such a factor. Heuristics Evaluation: The advantage of this technique is that the participants are equipped with the tools to objectively identify usability bugs; as the experimenters, we do not have to interpret their behaviour. We chose to perform the heuristic evaluation on our peers because they were already familiar with HCI principles and heuristics. We believed it was unrealistic to give our end users a crash course on material that we were taught over the course of 4 months.

Page

8

Choice of Materials We developed a paper prototype to demonstrate the functionality of the system. The advantage of paper-prototyping is its low-fidelity nature which does not discourage the participants from giving honest feedback. To facilitate that experiment, one of us sat with each tester as they explored the system to answer any questions, and to “wizard” system actions. Also, paper and pen are some things that everyone can relate to. In contrast, if we developed the prototype on the computer, we would give the impression that a lot of time was spent on it which discourages honest opinion. We also made use of our storyboard. It was especially helpful in communicating the overview of our entire system to our HCI peers because it illustrated the system from a third-person perspective.

Evaluation Data and Results Participant Information -

12 testers in total o

4 teachers; 8 HCI experts

-

We had 4 HCI experts perform the Cognitive Walkthrough

-

We had 4 teachers perform the Usability Tests

-

And we had the remaining 4 HCI experts perform the Heuristic Evaluations

-

All participants had their errors noted and counted

Cognitive Walkthrough Results Qualitative Data •

Participants expressed their approval of the timed log out aspect of the device.

•

Users found that the student pictures were quite useful and could help them learn the names of the students faster.

•

Users found that it was clumsy to navigate between pages using the "next" and "previous" page buttons.

•

Keyboard seemed out of place.

•

Too few students were displayed on a screen.

Page

9

•

When updating attendance records, some of the users disapproved of the "Modify" button's placement.

Usability Test Results Qualitative Data Daily Attendance Results: •

A user struggled for a minute trying to decide on which hand they should hold the device with.

•

Some participants had a different method of taking attendance. While some preferred to mark the present students down, others asked if they could only mark the absent down.

•

Participants made use of the back button frequently.

•

Users requested that the panel should automatically retrieve the time when marking down late students.

•

Many users also disliked how they could not see a record of what time the attendance was submitted or updated.

•

Too few students seemed to be displayed on a screen since some inquired about class size.

•

Users found that the student thumbnail photos were useful and would help them learn the names of the students faster.

•

When updating attendance records, some users tried to click on the student’s current status to change it rather than use the “Modify” button.

Emergency Situation Results: •

Teachers were initially confused as to why their emergency class list was missing some students.

•

Users struggled with using the keyboard while still holding the panel. Most users needed to rest it on a surface. Those same users had difficulty switching between typing and using the stylus. One such participant actually started pushing buttons on the keyboard with the stylus.

•

Teachers expressed their opinion that many younger students do not know their own student number, which is required in order to adopt the student in the case of an emergency.

Page 10

•

Teachers were confused as to where the login screen went during the emergency situation. They expressed their concern for the possible misuse of the system.

Quantitative Data Distraction Test Due to the small sample size of the distraction test, it was more appropriate for our group to do a nonparametric bootstrap sample with replacement on the data for each group. We took samples of 500 for each. Using the central limit theorem, we attempted to have the data more normalized in order to apply a Welch Two Sample t-test with a confidence level of 0.95. We let H0 : µ1 = µ2 and HA : µ1 ≠ µ2; that is, the true difference in means is 0 versus not equal to 0 respectively. The results can be seen as follows: Distraction Test: Group 1 (Default “Present”) Participant 1 2

Recovery Time (seconds) 16 13

Distraction Test: Group 2 (No Default) Participant 1 2

Recovery Time (seconds) 2 3

R OUTPUT: Welch Two Sample t-test data: a and b t = 170.2069, df = 608.591, p-value < 2.2e-16 Alternative hypothesis: true difference in means is not equal to 0 95 percent confidence interval: 11.90108 12.17892 Sample estimates: mean of x; mean of y 14.554

2.514

Heuristic Evaluations Results Qualitative Data •

No title on the login screen and similarly no title on the main attendance taking screen to identify purpose.

Page 11

•

"Modify" buttons at the summary screen seem out of place.

•

Attendance summary looks nothing like the previous attendance page.

•

Status modification selector activated by "Modify" button on Attendance Summary page fails to indicate the student's current status. The student's status is therefore inconsistent with the attendance taking screen prior to the summary page.

•

Keyboard seemed out of place.

•

Attendance summary page does not match the bubble sheet format that teachers are familiar with from the existing Trillium system.

•

Too few students displayed on a screen. Teachers are used to having more students per page.

•

Teachers liked that they cannot accidentally submit the attendance until they have gone through the entire list (i.e., submit button is underneath the last student on the entire class list).

•

Student thumbnails were too small.

•

Attendance does not indicate which student was marked last.

•

The only indicator distinguishing between the statuses of each student is the coloured dot.

•

The colour-coded bubbles visible when teachers take the attendance appeal to more senses of the user.

•

The static menu bar along the top that remains visible at all times was deemed useful.

•

Users liked that the attendance taking bubbles are similar to those of the current Trillium System.

•

Users liked the notification bar as it allows teachers to keep track of what just happened.

Quantitative Data See next section.

Error-Step Test We believe that the percentage of error is less than 5%, that is, H0 : µ0 < 0.05 in both regular attendance cases and in an emergency. We used t-tests with a confidence level of 0.95. Regular Attendance Steps and Error Observations (Regular) Participant 1

Steps 36

Errors 4

Errors/Step 0.11111111

Page 12

2

44

3

0.06818182

3 4

40 37

4 1

0.1 0.02702703

5

33

3

0.09090909

6 7

45 35

3 2

0.06666667 0.05714286

8

43

3

0.06976744

9 10

33 44

3 3

0.09090909 0.06818182

11

36

2

0.05555556

12

33

3

0.09090909

R Output: One Sample t-test Regular Attendance Case data: z t = 3.7018, df = 11, p-value = 0.001745 Alternative hypothesis: true mean is greater than 0.05 95 percent confidence interval: 0.06271535

Inf

sample estimates: mean of x 0.0746968 Summary Statistics for Regular Attendance Case Min. 1st Qu. Median Mean 3rd Qu. 0.02703

0.06429

0.06897

0.07470

0.09091

Max. 0.11110

Errors By Task (Normal Attendance) Task

Minimum Presses Average Presses Average # of Mistakes

Login

11

11.16666667

0.166666667

Take Attendance

21

22.25

1.583333333

Change Student Attendance Record

3

3.833333333

1.083333333

Logout

1

1

0

*Note: Some individuals forgot to update the attendance

Page 13

Emergency Attendance

Participant

Steps and Error Observations (Emergency) Steps Errors

Errors/Step

1

29

2

0.06896552

2

33

3

0.09090909

3

31

2

0.06451613

4

29

1

0.03448276

5

28

1

0.03571429

6

30

2

0.06666667

7

30

2

0.06666667

8

29

2

0.06896552

9

29

1

0.03448276

10

33

4

0.12121212

11

28

0.03571429

12

32

1 4

0.125

R Output: One Sample t-test Emergency Case data: e t = 1.9512, df = 11, p-value = 0.03848 Alternative hypothesis: true mean is greater than 0.05 95 percent confidence interval: 0.05141497 Inf sample estimates: mean of x 0.06777465

Page 14

Summary Statistics for Emergency Case Min. 1st Qu. Median Mean 0.03448 0.03571 0.06667 0.06777

3rd Qu. 0.07445

Max. 0.12500

Errors by Task (Emergency) Task

Minimum Presses Average Presses Average # of Mistakes

Accept Global Announcement

1

1.25

0.083333333

Taking attendance in emergency

16

16.91666667

1.166666667

Acknowledge a student has been adopted

1

1.166666667

0.083333333

Adopt a missing student

7

8.5

0.666666667

Contact main office for EMS

2

2.25

0.083333333

Data Analysis and Discussion Cognitive Walkthrough As presented in the previous section, in carrying out our cognitive walkthrough with HCI experts, many issues regarding our design were brought to our attention. We will be focusing on a few of these issues in more depth:

•

When stepping through the attendance-taking process with our participants, many of them brought to our attention that it was difficult and unnatural to flip between the class list through the use of the Previous and Next buttons. The main cause of this could be the fact that the Previous and Next navigation buttons are positioned with

Page 15

respect to the content of the page. In other words, they do not have a consistent designated position on each screen, which is not very user-friendly.

•

When asking participants to perform tasks that strictly required the use of the external keyboard, participants often hesitated and were unsure as to what they were to do with the stylus they had been using throughout the course of the evaluation. This led to our reconsideration of whether or not the external keyboard is necessary, and if an on-screen keyboard would be more suitable.

•

In terms of our attendance-taking interface, all 4 participants commented on the fact that there is too much whitespace in between the names of each student on the class list. They felt that this space could be used more appropriately by including a few more students on each page of the class list.

•

Many participants appreciated how EAS supports a timed expiry session in additional to the manual logout option. They mentioned that this is a good security feature that will be useful in school environments where potential users may often have to handle a number of issues at once. One participant also mentioned that having their interactive session expire and automatically log them out is a feature that they can find in many electronic systems they are familiar with. For example, they encounter these features regularly when they use school or library computers.

•

Two of our participants commented on how we have designed our interface to allow users to modify student attendance records. They mentioned that having the Modify button beside the current student status could potentially lead to confusion. In taking this feedback into consideration, we starting rethinking our attendance summary interface, and whether or not we could possibly remove the Modify button if it seems unnatural to the users.

Usability Test We also conducted usability tests with our target users, that is, elementary and secondary school teachers. As in the analysis for the cognitive walkthroughs performed, we will be analyzing a few aspects of the collected feedback more thoroughly:

Page 16

Daily Attendance Taking Usage:

•

As mentioned in the Evaluation Technique section, we provided our evaluation participants with 3 minutes so they could explore the EAP used for evaluation. During these 3 minutes, it was evident to our observers that the participant experienced some difficultly in deciding how they should hold the device. Specifically, the participant asked our team members whether or not the EAP accommodates for left-handed users.

•

When we asked our participants to take the attendance as they would normally do so, many participants completed the task assuming that unmarked students had the default status of being present. This is because the existing Trillium Attendance System treats unmarked students as marked present. However, we also had a few participants that commented on the fact that they liked how we let the teachers mark the students present as well because it is more intuitive for them to mark present as they call out the names of each student. This required us to look into our interface a little more thoroughly, and decide if there was something we could do in order to satisfy both usability preferences.

•

A participant suggested on how we could make our operations return more informative results. For example, teachers may often want to know how much later a student arrived on a particular day. So they suggested that we timestamp when a teacher updates a student’s status from absent to late, and have this time marked beside the student’s name. Along the same lines, they also suggested that we include the time at which the attendance record was submitted or updated on the notifications bar. This way, teachers and school staff will have more descriptive information in dealing with attendance related tasks and procedures.

•

Because our participants were teachers, they were able to point out that there were too few students on a single attendance-taking screen from a different perspective than the HCI experts that participated in the cognitive walkthrough. Our participants, as part of this usability test, felt that there should be more students on each screen of the class list because they are more accustomed to being able to view more students’ names at once.

•

All 4 participants responded positively to EAS’s thumbnail pictures of each student on the attendance-taking screen. In particular, they indicated that it would help them

Page 17

learn the names of the students at the beginning of the academic year, as well as facilitate substitute teachers’ ability to control the class. Emergency Situation Usage:

•

We also asked our usability test participants to accomplish high-level tasks during simulated emergency situations. When we asked teachers to take the attendance of the students as they would normally during the case of an emergency, we saw how they hesitated because they were confused as to why the emergency class list was missing some students. Although we had a brief description along the top of the screen explaining that this class list is only comprised of those students that were present that day, it still lead to confusion. This means that we will have to present this message in a more eye-catching and understandable way.

•

In designing the EAP, one of our central design requirements was to have the panel portable so teachers can use it in different ways and carry it around with them during emergency situations. However, it was evident to our observers that our evaluation participants preferred to place it on a flat surface rather than to hold onto it like a clipboard. Are there certain physical characteristics of the EAP that are making our users feel as through they need to put it on a flat surface? For example, is the EAP too big? Is it too heavy? Or is there a way we can design it so that users know that they can hold onto it like a clipboard?

•

Also in the case of an emergency, when users discovered that they had to use the external keyboard in order to interact with the system, we saw that their interaction became more difficult and clumsy. There were many instances where the participants were holding onto the panel and trying to figure out how they were to use the keyboard. Specifically, they were trying to find flat surfaces where they could use both the panel and the keyboard simultaneously. In one interesting instance, the user began using the physical tactile external keyboard with the stylus that he was using throughout the entire course of the evaluation process.

•

In the case of a lockdown, a student must remember their own student number in order for a teacher to adopt them into their class. However, as all 4 participants pointed out, from their experience, younger students are not likely to know the student numbers. This means that we will be required to come up with a way of identifying students in which young children can still inform the teachers as to who they are.

Page 18

•

Most teachers were worried about the way we lifted all security features in cases of emergencies. Users will no longer be required to log into the EAPs just to use them. However, there will be global announcements instructing school staff as to what procedures need to be carried out during emergencies. Hence teachers have identified their concern regarding the possibility of an EAP being misused by an intruder.

Analysis of Distraction Test Results With a p value of less than 2.2e-16 we have strong evidence to reject H0. Thus we can see that the layout of the EAS does affect the usability.

Heuristic Evaluation As mentioned in the evaluation criteria, participants were asked to assign a severity rating to each concern or issue that had with the prototype. In analyzing these results, we categorized the most common issues according to the five Neilson’s heuristic values that we have been focusing on. We also assigned each issue an overall severity rating by taking the median of the collection of participant ratings. Heuristic Visibility of System Status

Severity 1

Comment/Bug •

No title can be found on the login screen to indicate what the application is.

1

•

No title can be found on the main attendance-taking screen to identify the purpose of the current screen.

Aesthetic and Minimalistic

2

•

Design

The Modify button provided on the attendance summary screen is not very natural to users.

Consistency

4

•

Keyboard seems out of place.

2

•

The student thumbnail photos are small.

3

•

The attendance summary screen does not look like the previous attendance-taking screen.

3

•

Status modification selector activated by the Modify button on the attendance summary screen fails to indicate the student’s current status. The student’s

Page 19

status on the summary screen is therefore inconsistent with the same student’s status on the attendance-taking screen. Match Between System and

3

•

Real World

Attendance summary does not match the bubble sheet format that users are familiar with from their existing Trillium System.

4

•

There are too few students displayed on each page of the class list for attendance purposes. Teachers are used to having more students visible per page.

Error Prevention

2

•

The only indication to distinguish between the statuses of each student is a coloured dot.

These heuristic results can be analyzed as follows: Visibility of System Status: •

Although the notification bar on the side of the EAP provides users with a log of all the events that have happened since their login, participants have indicated that they were not aware of the system state at all times. Specifically, on the screen that users are required to take the attendance on, there is no title indicating the purpose of the screen. A similar visibility problem is the fact that there is no title on the login screen specifying that this device is an EAP. This may be problematic for substitute teachers who may not be familiar with EAS.

Aesthetic and Minimalistic Design: •

In terms of the aesthetics of EAS, participants have indicated that they liked how the status bubbles of the attendance sheets are colour-coded with green bubbles for present students, red for absent, and yellow of late students. They found this especially useful because it appeals to more cognitive senses. Although they really liked the idea of having thumbnail sized photos of each student beside their names, many participants indicated that they would like these thumbnails larger.

•

As for having a minimalistic design, almost all participants found it cumbersome to have an external keyboard during emergency situations. A few participants also pointed out that having a Modify button on the summary page is not very natural.

Page 20

Consistency: •

One of our central design requirements is to develop an interface that is similar to the Trillium Attendance System so users will find it easier to transition from the existing system to EAS. However, we did not successfully maintain the consistency between the attendance-taking screen and the attendance summary screen. We have designed the attendance-taking screen to resemble the current Trillium system bubble sheets, but the summary of student attendance records have been organized into a text-based format (i.e., present will appear beside the name of a present student, as opposed to having the present bubble beside the student’s name).

•

In order to update the status of a student, users are required to press the Modify button on the attendance summary screen in order to activate the status modification selector. This selector has 3 empty bubbles: 1 for present, 1 for absent, and 1 for late. However, most participants pointed out that this is not consistent with the attendancetaking screen prior to the summary page since it does not indicate the current status the student is marked as.

•

On the other hand, all participants indicated that having the menu bar along the top of the EAP remains constantly visible was very useful.

Match Between System and Real World: •

As mentioned earlier, participants found the attendance summary screen inconsistent with the attendance-taking screen. This is because the attendance summary screen is not in the bubble sheet format that users are familiar with from the existing Trillium system.

•

With the existing Trillium system, teachers can view more students on one page than on a single EAP screen. Therefore, this is a mismatch between the real world application and the system, since teachers are used to having more students visible at once.

•

On the other hand, many participants have indicated that they appreciate that in general, our attendance-taking screen somewhat resembles the Trillium bubble sheets with which they are all familiar.

Page 21

Error Prevention: •

A single participant responded that they noticed that the only indicator that distinguishes between the statuses of each student is the coloured dot in the attendance column corresponding to their status. They mentioned that it might be more informative if we could make these attendance statuses more eye-catching.

•

Many participants noticed that how we positioned the Submit attendance button below the last student on the class list. This prevents the teacher from submitting the attendance record before they have entirely gone through the class list.

Error-Step Test Regular Attendance: With a confidence level of 0.95 and a p-value = 0.001745 < 0.05 there was strong evidence against the null hypothesis; that is, our claim was incorrect when we believed the average percent of errors would be less than 5% when taking regular attendance. When errors were broken down by task, it was apparent that many participants had difficulty just doing the attendance. As well, modifications to the system also seem unclear.

Emergency With a confidence level of 0.95 and a p-value = 0.03848 < 0.05 there is there is strong evidence against the null hypothesis; that is, our claim was incorrect when we believed the average percent of errors would be less than 5% when taking attendance in an emergency. Errors in modifying attendance defeat the purpose of our initial goal of providing an efficient and quick way of taking the attendance. When errors were broken down by task, it was apparent that many participants again had difficulty completing the attendance. We saw that even adopting a student was difficult for them. This could lead to a very serious problem during such an emergency. From these results, it is clear that errors were much more apparent than we had thought. It seems that our system is not as intuitive as expected as the error value was much greater than our expected 5%. As a result, we need to focus more on the basic usability issues that we did not address correctly. Although the number of steps were above the minimum number required, it is clear that errors were the major cause of this. In many cases errors and

Page 22

additional steps were not in a 1:1 ratio. If many people are making errors on the main usability requirements, then it is clear that we need to rethink our interface. These will be discussed in the next sections.

Design Implications and Improvements After examining the implications, there are a few points in the design that need to be modified that can address these problems. Implications

The device interface should be flexible enough

Improvements

•

Add option that detects how the device is

so that users can use it regardless of hand

oriented. The display will be flipped

preference. This would be simple to

depending on with which hand you hold it

implement as all interaction is done within the

with.

touch display. The keyboard should to be easy to use and not

•

cumbersome.

The physical keyboard should be left out of the final implementation. The interface can be changed to have on screen keyboard as method of input using the stylus.

•

Silent messaging system should allow for handwritten text.

The display should to make good use of the on

•

Decrease font size and increase the

screen space. There must be a balance

amount of students displayed onscreen at

between readability and efficiency of use.

a time. •

Add a magnification option, similar to the way “Mac” computers implement it.

•

Add line separators in combination with the above methods to promote readability.

Page 23

•

Remove the “next” and “previous” page buttons and add smooth scrolling for efficiency.

The device should allow the user to distinguish

•

Add a color coding scheme to display the

whether a particular student's attendance has

currently selected student (I.e. the entire

been marked for that session yet.

line would change colour) •

Add a coloured background to the already completed students' status.

When adopting a student, the system should

•

In the adoption procedure, add multiple

be robust enough to distinguish a student's

ways to input student data. Rather than

identity without depending on only 1 source of

be restricted to just a student number,

information (i.e. student number.)

the user will also have the option of inputting a student name, address or any other identifying information. For example, if multiple students exist with that same name, a list of all the students with that matching name will appear with corresponding pictures to distinguish them.

The device should to cater to user preferences

•

When the user profiles are created, there

when taking attendance. (i.e. Some prefer to

should be an option to set status defaults.

just mark absent students, while some may

Students could be marked as

prefer to mark present students)

absent/present by default.

The device should to continue to ensure data integrity and prevent potential misuse

•

Require users to login even in emergency procedures.

especially in an emergency.

Page 24

Critique of Evaluation Plan The goal of this project was to provide teachers with a fast, efficient and safe way of taking their attendance on both a daily basis, and in the case of an emergency. Now that we have executed our evaluation plan, we must reflect upon both it and how it might have influenced our results. Given what we know now, there are many things that we could have done differently. Similarly, there are many things that we were aware of, but could not do due to a lack of resources.

Knowing what we know now, what would we have done differently? In the evaluation process, one of the biggest drawbacks was the use of a functional paperprototype. While we still believe that presenting a high-fidelity prototype might affect the honesty of our participants, paper-prototyping is not without its own shortcomings. In playing the wizard to facilitate the functionality, we had to intrude on the participant’s personal space. Our involvement also skews some of our results like the measure of time and the impressions of system actions. For example, we measured the time it took for users to complete tasks; while it gives us a means of comparing the speed of tasks relative to each other, we cannot use the numbers as the absolute time it takes to complete each task because it takes time for us to react as wizards. We used stickers instead of markers to simulate radio buttons because it’s gives the impression the EAS requires bubbling in a student’s status like the existing system. Instead, the EAS was designed to replace the bubbling process with a tapping gesture. This way, it eliminates the errors of inaccurate attendance records from the system misreading an incorrectly filled bubble. With a self sustaining prototype like a digital simulation, we minimize our presence in the evaluation process to create a more realistic usage environment.

Cognitive Walkthrough Even though the cognitive walkthrough is supposed to be done with HCI experts, we feel that it could have been done with our end-users. In fact, we might have had more beneficial results if we performed it with the teachers. In doing so, we would not need to develop a believability story to speculate the value of the task to the user; the users would be evaluating it for themselves. Teachers are also in a better position to compare the functionality of the existing system with our proposed solution because they are users of the current system.

Page 25

Usability Tests In retrospect, the Sudoku distraction test may have been inappropriate as our participants found it slightly awkward. Although teachers experience distractions, this was an unrealistic example.

Heuristic Evaluations The heuristic evaluations were our smoothest set of tests. We suspect it was because both the evaluators and the experimenters were familiar with the nature of heuristic evaluations and HCI principles.

What would we have done if we had more resources? While we were able to collect a set of useful feedback from the users, we could have acquired even more valuable feedback if it were not for certain restricting factors such as time, cost, and participant co-operation. First and foremost, a greater number of evaluation participants would yield a more representative sample population. We would then be able to perform tests on participants across all age, sexes and technological skill levels. We would be able to determine things like a default font-size for old and young users, and an ideal weight for both sexes. A larger sample population would have also meant that we would not have had to do bootstrapping in our analysis which would greatly increase the power of our statistical tests. With more resources, we could have afforded to do the more expensive usability techniques. For example, we would have liked to do a focus group composed of end-users, HCI experts and developers. The advantage of this technique is that we could discuss the different aspects of the prototype face-to-face. By bouncing ideas off each other, and voicing our understanding of the problem, we reduce the chance of misinterpretations any results. Any solutions would be a collaborative effort with input from the end-user. Of course, we were unable to do this because we could not schedule a time when it would be convenient for all the concerned parties. The HCI participants that evaluated our system were our peers due to the lack of connections to professional HCI experts. The problem with this is that most of them were aware of our project. For the same reason that we cannot objectively evaluate our own system because we

Page 26

are biased, our peer’s connection to us may influence the feedback that they give. For example, because of our affiliation, some of them saw our device even before the evaluation. Consequently, their proficiency of our system may be from previous exposure rather than current exploration. On, another note, taking one HCI course does not make any of us experts in the field. Given more time and participants, we would have liked to design more tests based on the feedback of an initial set of tests. This way, we could explore further some particular aspect of design. To illustrate, we were certain that a keyboard was valuable addition to the device, but feedback from our usability tests suggests otherwise. Keeping in mind that we only performed the test on 4 users, we are uncertain of the representative power of their evaluations. Ideally, we would have designed an experiment that tests the value of the keyboard based on what we have learned.

Page 27