Prototype Photos



Briefing

Thank you for volunteering to test our ski school scheduler prototype.  Today we will ask you to try to perform several tasks with the paper prototype of the program, and one of us will pretend to be the computer and change the paper prototype as needed.  The overall goal of the system is to help a small ski area manage their instructors.  They normally have about 150 instructors who work 2 to 3 shifts a week each.  Many of the instructors and admins have difficulty using computers.  Each instructor submits a schedule before the season starts, and it must be approved by an admin.  After that, an instructor can request changes, which can be approved or rejected.

Today we will ask you to perform tasks both as an average instructor of the ski area and as an admin.  We’ll clearly indicate which user you represent at any given time.  As an average instructor, you can only view your own schedule and must have schedule changes approved by an admin.  As an admin, you have access to all of the instructors’ schedules and can approve or reject changes.  You can assume that when we tell you which user you represent, you have already logged in with your email and a password to authenticate yourself.  

This particular ski area has three shifts, day, evening, and night, although the night shift is only taught on weekends and holidays.  Instructors are permitted to teach more than one shift in a given day.  They also teach five disciplines, adult ski, adult snowboard, children’s ski, chlidren’s snowboard, and race (ski only), some of which are only taught a certain times.  An instructor can only sign up for one discipline per shift.  Do you have any questions?

[pause for questions]

You will greatly help us improve our interface if you think aloud as you go through the tasks we will give you.  We welcome any and all feedback, so please say anything that comes to mind!

Scenario Tasks

1. As instructor, create and submit your initial schedule before the season begins.  You wish to work as a children’s snowboard instructor every Sunday during the day, a racing instructor all Tuesday evenings except December 25, 2012 and January 1, 2013, and an adults’ ski instructor on two Friday nights, January 11, 2013 and February 1, 2013.
2. As instructor, request to remove yourself from the schedule for January 20, 2013.
3. As instructor, request to add yourself to the schedule as an adults’ snowboard instructor for the evening shift on January 21, 2013.
4. As admin, notice that the instructor Sam has requested several schedule changes.  Approve his schedule changes.
5. As admin, mark Sam as being excused from his shift on January 13, 2013.  Also mark him as having been absent on December 16, 2012.

Observations

First iteration user testing

User Test #1

Task 1

• Difficulties
  • Not sure whether to click a day or discipline first, esp. if it will filter the display
  • Not sure how to get back to the regular cursor
  • Not sure how to scroll months (prob. didn’t see the scrollbar because it was partially obscured at first #paperprototypeproblems)
  • Assumes clicking on a selected shift will toggle it off
• Successes
  • The mouse cursor changing indicates that you can use it for multiple actions

Task 2

• Difficulties
  • Hoping you could clear a shift by clicking on it
• Successes
  • Went straight to submit when done

Task 3

• Difficulties
  • Had a very unclear mental model – didn’t know if the calendar was something he was controlling, or requesting for changes

Task 4

• Difficulties
  • Wanted a way to view all instructors currently on a shift (ended up “opening a new tab” to see that view)
  • Not sure if he had to press Submit, since the approve buttons disappeared and they were buttons
• Successes
  • Found the approve buttons really quickly

Task 5

• Successes
  • Immediately went for the buttons and the shifts, and hit submit. (Noted that the E and A seemed appropriately clear.)

User Test #2

Task 1

• Difficulties
  • Was about to click on a shift icon to clear it, but then clicked the eraser (still, was confused for a bit)
  • Wanted a way to select the same shift every week
    • spent a while trying to find one
    • tried clicking on the sun icon in the month header
    • and tried clicking on the Sunday header
  • Expected that clicking on the box after 1/31 would mark 2/1
• Successes
  • Went straight for Jr. Snow icon
  • Used scrollbar to move view
  • When she needed to clear more mistakes, she went straight for the clear button.
  • Used scrollbar while cursor was a diff. icon without trouble.
  • Hit submit when done.

Task 2

• Difficulties
  • Clicked the icon for the shift to clear (toggling)
• Successes
  • Went straight for Submit

Task 3

• Successes
  • Did it right immediately

Task 4

• Difficulties
  • Probably only hit Submit because we looked expectant
• Successes
  • Hit both Accept buttons quickly

Task 5

• Successes
  • It all went very smoothly

User Test #3

Task 1

• Difficulties
  • Clicked Jr-board to clear the cursor so he could switch it
  • Clicked “Submit Request” too early // it had been changed
  • He missed a lot of shifts, but we think this wouldn’t have happened normally, because a real instructor would have known to look for each shift he/she wanted #paperprototypeproblems
  • Was confused about shifts that weren’t there (“Does that mean it’s already taken?”)
• Successes
  • Went straight for Jr-board icon

Task 2

• Difficulties
  • Tried to click on a shift to clear it (toggling!)
  • Afraid to hit clear, as if it might clear everything
• Successes
  • Went straight for Submit at the end

Task 3

• Successes
  • Had no problems

Task 4

• Difficulties
  • Unsure of where to click (probably didn’t see the panel at top, since it was a transparency #paperprototypeproblems)

Task 5

• Successes
  • No problems – A and E were again clear enough

First iteration summary

Overall difficulties

• They were not sure how to get the regular cursor back, or what they could do with the icons as the cursor.
• All three of our users assumed clicking on a shift already in place would toggle it (i.e. unselect it).
• They wanted a way to select a shift the same day/time every week.
• They weren't sure if changes on the calendar were requested or immediately changed.
• They weren't sure if they had to press Submit to approve changes as an admin.
• They had trouble scrolling through the months
  • likely a paper prototype issue

Overall successes

• They appreciated that the mouse cursor changed, indicating that they could use it for multiple actions.
• By task 2, they had learned to hit Submit when done in the instructor mode.
• Once they found the discipline icons, they seemed to have no trouble differentiating between disciplines.
• They had no trouble finding and using E and A icons.

Second iteration user testing

User Test #4

Task 1

• Difficulties
  • After making a mistake, thought clicking again without changing the mode would undo the mistake
  • Had trouble differentiating between evening and night (did not see the legend)
  • Gave up and submitted request before technically finishing the task
  • Was expecting a way to select all for a given shift on a given day of week
• Successes
  • Expected the mouse cursor to change when it actually did
  • Had no trouble using the eraser once he found it
  • Knew to use scroll bar to move from month to month, even though was expecting to have left/right arrows next to month to move to previous/next month

Task 2

• Successes
  • Was clear that eraser must be used for this task

Task 3

• Successes
  • Did the correct actions on the first try
  • Changed back to regular mouse cursor before submitting (seems to make user more comfortable even though it’s not necessary)

Task 4

• Successes
  • Went for the radio buttons right away

Task 5

• Successes
  • A and E were clear, like in the first iteration

User Test #5

Task 1

• Difficulties
  • Frustrated by not being able to find a way to select all for a given shift on a given day of week
  • Did not see scroll bar initially (probably #paperprototypeproblems) but was expecting left/right arrows to go to previous/next month
  • Had trouble differentiating between evening and night shifts, referred to the three shifts as morning, afternoon/sundown, and evening while thinking aloud
• Successes
  • Learned the behavior of the cursor after some experimenting

Task 2

• Difficulties
  • Tried initially to both click and drag away and click and hold the shift he wanted to remove
  • Misread “Remove shift” as “Rename shift” and thought the eraser icon looked like a trash can (probably handwriting/drawing #paperprototypeproblems)
  • Continued to confuse evening and night shifts
• Successes
  • Knew to click “Submit Request”

Task 3

• Successes
  • Did exactly what we expected without any errors

Task 4

• Difficulties
  • Not immediately obvious where schedule changes were made (expected dates to be written out, e.g. January 21 instead of 1/21)
  • Thought A stood for accept, not absent
  • Confused by what “excused” meant (thought it meant pardoning someone for doing something wrong)
• Successes
  • Obvious how to clear the A’s and E’s from the schedule
  • Knew what to do with the radio buttons once he found them

Task 5

• Successes
  • Had learned how to use A and E by this time

User Test #6

Task 1

• Difficulties
  • Did not immediately notice scroll bar (possibly #paperprototypeproblems)
• Successes
  • Expected cursor to change exactly when it did
  • Thought it was very clear how input shifts on the calendar, even though having a way to select all for a given day of week would be helpful
  • Used scroll bar while the cursor was a non-standard mouse icon without hesitation

Task 2

• Successes
  • Used Remove Shift (eraser) cursor correctly and thought it was easy

Task 3

• Difficulties
  • A bit confused by the color scheme (not sure what it represents)
  • Did not immediately notice the warning message telling the user that they must submit before any changes are permanently recorded
• Successes
  • Did exactly what we expected without difficulties

Task 4

• Difficulties
  • Unsure if the changes the instructor has requested could be viewed on the calendar
• Successes
  • Found the radio buttons quickly and knew how to use them

Task 5

• Successes
  • Once again, A and E were very straightforward to use

Second iteration summary

Overall difficulties

• They had trouble figuring out how to move from month to month.  It’s possible that it’s a paper prototype issue, but two of the users explicitly expected to have left/right arrows to click on to move from month to month.
• Two of the three users had trouble differentiating between evening and night shifts; they did not see the legend that mapped the names of the shifts to the icons.
• They all wanted a way to select all for a given shift on a given day of the week.
• They wished the messages at the top of the interface indicating that Submit must be pressed (and also the admins’ radio buttons for approving changes) were more prominent.** May also be a paper prototype problem
• They were unclear as to whether or not the calendar changed dynamically as they selected the accept/reject radio buttons in Task 4.

Overall successes

• They had no trouble finding/understanding the E and A icons and the corresponding icons to undo excused and absent markings.
• They had no trouble differentiating between disciplines.
• They knew to press the Submit Request (for instructors) or Submit (for admins) button when they were done making changes.
• They expected the mouse cursor to change the way it does when clicking on icons on the left sidebar.
• They clearly understood what the Remove Shift (eraser) mode does.

Prototype Iteration

These are the changes made to the paper prototype between the two rounds of testing:

1. Changed “Discipline” to “Select Discipline” text left sidebar to make sure that the user knows they need to select
2. Added modes (crossed out A and crossed out E buttons) to allow the admin to remove Absent/Excused markings.
3. Changed “Clear” to “Remove Shift” to make it more obvious  that it doesn’t clear the entire schedule
4. Changed “Submit” to “Submit Request” on the instructor’s view so that the instructor knows that the schedule must be approved before changes are recorded
5. Add text “You must hit ‘Submit’ to save changes” at the top of the admin view so that they know that changes aren’t saved until they submit.
6. Add text “You must hit ‘Submit Request’ to save changes” at the top of the instructor view so that they know that changes aren’t saved until they submit.
7. Newly added text (in changes 5 and 6) and list of pending changes do not scroll so that user can see this information at all times
8. Changed approval for pending changes to be done with radio buttons with two options (accept or reject) for each pending change, instead of normal buttons, to indicate that nothing is submitted until the admin clicks the “Submit” button. The schedule still updates dynamically as the admin selects the radio buttons.
9. We chose not to add the functionality of selecting all for a given day of the week because we wanted to prioritize safety (i.e. not accidentally selecting all days) over the added efficiency because our user population is not very familiar with computers.  In addition, the only time a user is likely to select all of a given day of the week is when submitting their initial schedule, and this task is only performed once a year.