...

Briefing

Thank you for volunteering to test our ski school scheduler prototype.  Today we will ask you to try to perform several tasks with the paper prototype of the program, and one of us will pretend to be the computer and change the paper prototype as needed.  The overall goal of the system is to help a small ski area manage their instructors.  They normally have about 150 instructors who work 2 to 3 shifts a week each.  Many of the instructors and admins have difficulty using computers.  Each instructor submits a schedule before the season starts, and it must be approved by an admin.  After that, an instructor can request changes, which can be approved or rejected.

Today we will ask you to perform tasks both as an average instructor of the ski area and as an admin.  We’ll clearly indicate which user you represent at any given time.  As an average instructor, you can only view your own schedule and must have schedule changes approved by an admin.  As an admin, you have access to all of the instructors’ schedules and can approve or reject changes.  You can assume that when we tell you which user you represent, you have already logged in with your email and a password to authenticate yourself.  

This particular ski area has three shifts, day, evening, and night, although the night shift is only taught on weekends and holidays.  Instructors are permitted to teach more than one shift in a given day.  They also teach five disciplines, adult ski, adult snowboard, children’s ski, chlidren’s snowboard, and race (ski only), some of which are only taught a certain times.  An instructor can only sign up for one discipline per shift.  Do you have any questions?

Wiki Markup\[pause for questions\]

You will greatly help us improve our interface if you think aloud as you go through the tasks we will give you.  We welcome any and all feedback, so please say anything that comes to mind!

Scenario Tasks

1. As instructor, create and submit your initial schedule before the season begins.  You wish to work as a children’s snowboard instructor every Sunday during the day, a racing instructor all Tuesday evenings except December 25, 2012 and January 1, 2013, and an adults’ ski instructor on two Friday nights, January 11, 2013 and February 1, 2013.
2. As instructor, request to remove yourself from the schedule for January 20, 2013.
3. As instructor, request to add yourself to the schedule as an adults’ snowboard instructor for the evening shift on January 21, 2013.
4. As admin, notice that the instructor Sam has requested several schedule changes.  Approve his schedule changes.
5. As admin, mark Sam as being excused from his shift on January 13, 2013.  Also mark him as having been absent on December 16, 2012.

Observations

First iteration user testing

...

User Test #4

Task 1

• Difficulties
  • After making a mistake, thought clicking again without changing the mode would undo the mistake
  • Had trouble differentiating between evening and night (did not see the legend)
  • Gave up and submitted request before technically finishing the task
  • Was expecting a way to select all for a given shift on a given day of week
• Successes
  • Expected the mouse cursor to change when it actually did
  • Had no trouble using the eraser once he found it
  • Knew to use scroll bar to move from month to month, even though was expecting to have left/right arrows next to month to move to previous/next month

Task 2

• Successes
  • Was clear that eraser must be used for this task

Task 3

• Successes
  • Did the correct actions on the first try
  • Changed back to regular mouse cursor before submitting (seems to make user more comfortable even though it’s not necessary)

Task 4

• Successes
  • Went for the radio buttons right away

Task 5

• Successes
  • A and E were clear, like in the first iteration

User Test #5

Task 1

• Difficulties
  • Frustrated by not being able to find a way to select all for a given shift on a given day of week
  • Did not see scroll bar initially (probably #paperprototypeproblems) but was expecting left/right arrows to go to previous/next month
  • Had trouble differentiating between evening and night shifts, referred to the three shifts as morning, afternoon/sundown, and evening while thinking aloud
• Successes
  • Learned the behavior of the cursor after some experimenting

Task 2

• Difficulties
  • Tried initially to both click and drag away and click and hold the shift he wanted to remove
  • Misread “Remove shift” as “Rename shift” and thought the eraser icon looked like a trash can (probably handwriting/drawing #paperprototypeproblems)
  • Continued to confuse evening and night shifts
• Successes
  • Knew to click “Submit Request”

Task 3

• Successes
  • Did exactly what we expected without any errors

Task 4

• Difficulties
  • Not immediately obvious where schedule changes were made (expected dates to be written out, e.g. January 21 instead of 1/21)
  • Thought A stood for accept, not absent
  • Confused by what “excused” meant (thought it meant pardoning someone for doing something wrong)
• Successes
  • Obvious how to clear the A’s and E’s from the schedule
  • Knew what to do with the radio buttons once he found them

Task 5

• Successes
  • Had learned how to use A and E by this time

User Test #6

Task 1

• Difficulties
  • Did not immediately notice scroll bar (possibly #paperprototypeproblems)
• Successes
  • Expected cursor to change exactly when it did
  • Thought it was very clear how input shifts on the calendar, even though having a way to select all for a given day of week would be helpful
  • Used scroll bar while the cursor was a non-standard mouse icon without hesitation

Task 2

• Successes
  • Used Remove Shift (eraser) cursor correctly and thought it was easy

Task 3

• Difficulties
  • A bit confused by the color scheme (not sure what it represents)
  • Did not immediately notice the warning message telling the user that they must submit before any changes are permanently recorded
• Successes
  • Did exactly what we expected without difficulties

Task 4

• Difficulties
  • Unsure if the changes the instructor has requested could be viewed on the calendar
• Successes
  • Found the radio buttons quickly and knew how to use them

Task 5

• Successes
  • Once again, A and E were very straightforward to use

Second iteration summary

Overall difficulties

• They had trouble figuring out how to move from month to month.  It’s possible that it’s a paper prototype issue, but two of the users explicitly expected to have left/right arrows to click on to move from month to month.
• Two of the three users had trouble differentiating between evening and night shifts; they did not see the legend that mapped the names of the shifts to the icons.
• They all wanted a way to select all for a given shift on a given day of the week.
• They wished the messages at the top of the interface indicating that Submit must be pressed (and also the admins’ radio buttons for approving changes) were more prominent.** May also be a paper prototype problem
• They were unclear as to whether or not the calendar changed dynamically as they selected the accept/reject radio buttons in Task 4.

Overall successes

• They had no trouble finding/understanding the E and A icons and the corresponding icons to undo excused and absent markings.
• They had no trouble differentiating between disciplines.
• They knew to press the Submit Request (for instructors) or Submit (for admins) button when they were done making changes.
• They expected the mouse cursor to change the way it does when clicking on icons on the left sidebar.
• They clearly understood what the Remove Shift (eraser) mode does.

Prototype Iteration

These are the changes made to the paper prototype between the two rounds of testing:

1. Changed “Discipline” to “Select Discipline” text left sidebar to make sure that the user knows they need to select
2. Added modes (crossed out A and crossed out E buttons) to allow the admin to remove Absent/Excused markings.
3. Changed “Clear” to “Remove Shift” to make it more obvious  that it doesn’t clear the entire schedule
4. Changed “Submit” to “Submit Request” on the instructor’s view so that the instructor knows that the schedule must be approved before changes are recorded
5. Add text “You must hit ‘Submit’ to save changes” at the top of the admin view so that they know that changes aren’t saved until they submit.
6. Add text “You must hit ‘Submit Request’ to save changes” at the top of the instructor view so that they know that changes aren’t saved until they submit.
7. Newly added text (in changes 5 and 6) and list of pending changes do not scroll so that user can see this information at all times
8. Changed approval for pending changes to be done with radio buttons with two options (accept or reject) for each pending change, instead of normal buttons, to indicate that nothing is submitted until the admin clicks the “Submit” button. The schedule still updates dynamically as the admin selects the radio buttons.
9. We chose not to add the functionality of selecting all for a given day of the week because we wanted to prioritize safety (i.e. not accidentally selecting all days) over the added efficiency because our user population is not very familiar with computers.  In addition, the only time a user is likely to select all of a given day of the week is when submitting their initial schedule, and this task is only performed once a year.