Prototype Photos

(photos go here)

Briefing

Thank you for volunteering to test our ski school scheduler prototype.  Today we will ask you to try to perform several tasks with the paper prototype of the program, and one of us will pretend to be the computer and change the paper prototype as needed.  The overall goal of the system is to help a small ski area manage their instructors.  They normally have about 150 instructors who work 2 to 3 shifts a week each.  Many of the instructors and admins have difficulty using computers.  Each instructor submits a schedule before the season starts, and it must be approved by an admin.  After that, an instructor can request changes, which can be approved or rejected.

Today we will ask you to perform tasks both as an average instructor of the ski area and as an admin.  We’ll clearly indicate which user you represent at any given time.  As an average instructor, you can only view your own schedule and must have schedule changes approved by an admin.  As an admin, you have access to all of the instructors’ schedules and can approve or reject changes.  You can assume that when we tell you which user you represent, you have already logged in with your email and a password to authenticate yourself.  

This particular ski area has three shifts, day, evening, and night, although the night shift is only taught on weekends and holidays.  Instructors are permitted to teach more than one shift in a given day.  They also teach five disciplines, adult ski, adult snowboard, children’s ski, chlidren’s snowboard, and race (ski only), some of which are only taught a certain times.  An instructor can only sign up for one discipline per shift.  Do you have any questions?

[pause for questions]

You will greatly help us improve our interface if you think aloud as you go through the tasks we will give you.  We welcome any and all feedback, so please say anything that comes to mind!

Scenario Tasks

1. As instructor, create and submit your initial schedule before the season begins.  You wish to work as a children’s snowboard instructor every Sunday during the day, a racing instructor all Tuesday evenings except December 25, 2012 and January 1, 2013, and an adults’ ski instructor on two Friday nights, January 11, 2013 and February 1, 2013.
2. As instructor, request to remove yourself from the schedule for January 20, 2013.
3. As instructor, request to add yourself to the schedule as an adults’ snowboard instructor for the evening shift on January 21, 2013.
4. As admin, notice that the instructor Sam has requested several schedule changes.  Approve his schedule changes.
5. As admin, mark Sam as being excused from his shift on January 13, 2013.  Also mark him as having been absent on December 11, 2012.

Observations

First iteration user testing

User Test #1

Task 1

• Difficulties
  • Not sure whether to click a day or discipline first, esp. if it will filter the display
  • Not sure how to get back to the regular cursor
  • Not sure how to scroll months (prob. didn’t see the scrollbar because it was partially obscured at first #paperprototypeproblems)
  • Assumes clicking on a selected shift will toggle it off
• Successes
  • The mouse cursor changing indicates that you can use it for multiple actions

Task 2

• Difficulties
  • Hoping you could clear a shift by clicking on it
• Successes
  • Went straight to submit when done

Task 3

• Difficulties
  • Had a very unclear mental model – didn’t know if the calendar was something he was controlling, or requesting for changes

Task 4

• Difficulties
  • Wanted a way to view all instructors currently on a shift (ended up “opening a new tab” to see that view)
  • Not sure if he had to press Submit, since the approve buttons disappeared and they were buttons
• Successes
  • Found the approve buttons really quickly

Task 5

• Successes
  • Immediately went for the buttons and the shifts, and hit submit. (Noted that the E and A seemed appropriately clear.)

User Test #2

Task 1

• Difficulties
  • Was about to click on a shift icon to clear it, but then clicked the eraser (still, was confused for a bit)
  • Wanted a way to select the same shift every week
    • spent a while trying to find one
    • tried clicking on the sun icon in the month header
    • and tried clicking on the Sunday header
  • Expected that clicking on the box after 1/31 would mark 2/1
• Successes
  • Went straight for Jr. Snow icon
  • Used scrollbar to move view
  • When she needed to clear more mistakes, she went straight for the clear button.
  • Used scrollbar while cursor was a diff. icon without trouble.
  • Hit submit when done.

Task 2

• Difficulties
  • Clicked the icon for the shift to clear (toggling)
• Successes
  • Went straight for Submit

Task 3

• Successes
  • Did it right immediately

Task 4

• Difficulties
  • Probably only hit Submit because we looked expectant
• Successes
  • Hit both Accept buttons quickly

Task 5

• Successes
  • It all went very smoothly

User Test #3

Task 1

• Difficulties
  • Clicked Jr-board to clear the cursor so he could switch it
  • Clicked “Submit Request” too early // it had been changed
  • He missed a lot of shifts, but we think this wouldn’t have happened normally, because a real instructor would have known to look for each shift he/she wanted #paperprototypeproblems
  • Was confused about shifts that weren’t there (“Does that mean it’s already taken?”)
• Successes
  • Went straight for Jr-board icon

Task 2

• Difficulties
  • Tried to click on a shift to clear it (toggling!)
  • Afraid to hit clear, as if it might clear everything
• Successes
  • Went straight for Submit at the end

Task 3

• Successes
  • Had no problems

Task 4

• Difficulties
  • Unsure of where to click (probably didn’t see the panel at top, since it was a transparency #paperprototypeproblems)

Task 5

• Successes
  • No problems – A and E were again clear enough

First iteration summary

Overall difficulties

• They were not sure how to get the regular cursor back, or what they could do with the icons as the cursor.
• All three of our users assumed clicking on a shift already in place would toggle it (i.e. unselect it).
• They wanted a way to select a shift the same day/time every week.
• They weren't sure if changes on the calendar were requested or immediately changed.
• They weren't sure if they had to press Submit to approve changes as an admin.
• They had trouble scrolling through the months
  • likely a paper prototype issue

Overall successes

• They appreciated that the mouse cursor changed, indicating that they could use it for multiple actions.
• By task 2, they had learned to hit Submit when done in the instructor mode.
• Once they found the discipline icons, they seemed to have no trouble differentiating between disciplines.
• They had no trouble finding and using E and A icons.

Second iteration user testing

User Test #4

Task 1

Task 2

Task 3

Task 4

Task 5

User Test #5

Task 1

Task 2

Task 3

Task 4

Task 5

User Test #6

Task 1

Task 2

Task 3

Task 4

Task 5

Second iteration summary

Overall difficulties

Overall successes

Prototype Iteration

These are the changes made to the paper prototype between the two rounds of testing:

1. Changed “Discipline” to “Select Discipline” text left sidebar to make sure that the user knows they need to select
2. Added modes (crossed out A and crossed out E buttons) to allow the admin to remove Absent/Excused markings.
3. Changed “Clear” to “Remove Shift” to make it more obvious  that it doesn’t clear the entire schedule
4. Changed “Submit” to “Submit Request” on the instructor’s view so that the instructor knows that the schedule must be approved before changes are recorded
5. Add text “You must hit ‘Submit’ to save changes” at the top of the admin view so that they know that changes aren’t saved until they submit.
6. Add text “You must hit ‘Submit Request’ to save changes” at the top of the instructor view so that they know that changes aren’t saved until they submit.
7. Newly added text (in changes 5 and 6) and list of pending changes do not scroll so that user can see this information at all times
8. Changed approval for pending changes to be done with radio buttons with two options (accept or reject) for each pending change, instead of normal buttons, to indicate that nothing is submitted until the admin clicks the “Submit” button. The schedule still updates dynamically as the admin selects the radio buttons.
9. We chose not to add the functionality of selecting all for a given day of the week because we wanted to prioritize safety (i.e. not accidentally selecting all days) over the added efficiency because our user population is not very familiar with computers.  In addition, the only time a user is likely to select all of a given day of the week is when submitting their initial schedule, and this task is only performed once a year.