Prototype Photos
(photos go here)
|
|
Briefing
Thank you for volunteering to test our ski school scheduler prototype. Today we will ask you to try to perform several tasks with the paper prototype of the program, and one of us will pretend to be the computer and change the paper prototype as needed. The overall goal of the system is to help a small ski area manage their instructors. They normally have about 150 instructors who work 2 to 3 shifts a week each. Many of the instructors and admins have difficulty using computers. Each instructor submits a schedule before the season starts, and it must be approved by an admin. After that, an instructor can request changes, which can be approved or rejected.
Today we will ask you to perform tasks both as an average instructor of the ski area and as an admin. We’ll clearly indicate which user you represent at any given time. As an average instructor, you can only view your own schedule and must have schedule changes approved by an admin. As an admin, you have access to all of the instructors’ schedules and can approve or reject changes. You can assume that when we tell you which user you represent, you have already logged in with your email and a password to authenticate yourself.
This particular ski area has three shifts, day, evening, and night, although the night shift is only taught on weekends and holidays. Instructors are permitted to teach more than one shift in a given day. They also teach five disciplines, adult ski, adult snowboard, children’s ski, chlidren’s snowboard, and race (ski only), some of which are only taught a certain times. An instructor can only sign up for one discipline per shift. Do you have any questions?
[pause for questions]
You will greatly help us improve our interface if you think aloud as you go through the tasks we will give you. We welcome any and all feedback, so please say anything that comes to mind!
Scenario Tasks
- As instructor, create and submit your initial schedule before the season begins. You wish to work as a children’s snowboard instructor every Sunday during the day, a racing instructor all Tuesday evenings except December 25, 2012 and January 1, 2013, and an adults’ ski instructor on two Friday nights, January 11, 2013 and February 1, 2013.
- As instructor, request to remove yourself from the schedule for January 20, 2013.
- As instructor, request to add yourself to the schedule as an adults’ snowboard instructor for the evening shift on January 21, 2013.
- As admin, notice that the instructor Sam has requested several schedule changes. Approve his schedule changes.
- As admin, mark Sam as being excused from his shift on January 13, 2013. Also mark him as having been absent on December 11, 2012.
Observations
User Test #1
Task 1
- Difficulties
- Not sure whether to click a day or discipline first, esp. if it will filter the display
- Not sure how to get back to the regular cursor
- Not sure how to scroll months (prob. didn’t see the scrollbar because it was partially obscured at first #paperprototypeproblems)
- Assumes clicking on a selected shift will toggle it off
- Successes
- The mouse cursor changing indicates that you can use it for multiple actions
Task 2
- Difficulties
- Hoping you could clear a shift by clicking on it
- Successes
- Went straight to submit when done
Task 3
- Difficulties
- Had a very unclear mental model – didn’t know if the calendar was something he was controlling, or requesting for changes
Task 4
- Difficulties
- Wanted a way to view all instructors currently on a shift (ended up “opening a new tab” to see that view)
- Not sure if he had to press Submit, since the approve buttons disappeared and they were buttons
- Successes
- Found the approve buttons really quickly
Task 5
- Difficulties
- None
- Successes
- Immediately went for the buttons and the shifts, and hit submit. (Noted that the E and A seemed appropriately clear.)
User Test #2
Task 1
- Difficulties
- Was about to click on a shift icon to clear it, but then clicked the eraser (still, was confused for a bit)
- Wanted a way to select the same shift every week
- spent a while trying to find one
- tried clicking on the sun icon in the month header
- and tried clicking on the Sunday header
- Expected that clicking on the box after 1/31 would mark 2/1
- Successes
- Went straight for Jr. Snow icon
- Used scrollbar to move view
- When she needed to clear more mistakes, she went straight for the clear button.
- Used scrollbar while cursor was a diff. icon without trouble.
- Hit submit when done.
Task 2
- Difficulties
- Clicked the icon for the shift to clear (toggling)
- Successes
- Went straight for Submit
Task 3
- Successes
- Did it right immediately
Task 4
- Difficulties
- Probably only hit Submit because we looked expectant
- Successes
- Hit both Accept buttons quickly
Task 5
- Successes
- It all went very smoothly
User Test #3
Task 1
- Difficulties
- Clicked Jr-board to clear the cursor so he could switch it
- Clicked “Submit Request” to early // it had been changed
- He missed a lot of shifts, but we think this wouldn’t have happened normally, because a real instructor would have known to look for each shift he/she wanted #paperprototypeproblems
- Was confused about shifts that weren’t there (“Does that mean it’s already taken?”)
- Successes
- Went straight for Jr-board icon
Task 2
- Difficulties
- Tried to click on a shift to clear it (toggling!)
- Afraid to hit clear, as if it might clear everything
- Successes
- Went straight for Submit at the end
Task 3
- Successes
- Had no problems
Task 4
- Difficulties
- Unsure of where to click (probably didn’t see the panel at top, since it was a transparency #paperprototypeproblems)
Task 5
- Successes
- No problems – A and E were again clear enough
Prototype Iteration
These are the changes made to the paper prototype between the two rounds of testing:
- Changed “Discipline” to “Select Discipline” text left sidebar to make sure that the user knows they need to select
- Added modes (crossed out A and crossed out E buttons) to allow the admin to remove Absent/Excused markings.
- Changed “Clear” to “Remove Shift” to make it more obvious that it doesn’t clear the entire schedule
- Changed “Submit” to “Submit Request” on the instructor’s view so that the instructor knows that the schedule must be approved before changes are recorded
- Add text “You must hit ‘Submit’ to save changes” at the top of the admin view so that they know that changes aren’t saved until they submit.
- Add text “You must hit ‘Submit Request’ to save changes” at the top of the instructor view so that they know that changes aren’t saved until they submit.
- Newly added text (in changes 5 and 6) and list of pending changes do not scroll so that user can see this information at all times
- Changed approval for pending changes to be done with radio buttons with two options (accept or reject) for each pending change, instead of normal buttons, to indicate that nothing is submitted until the admin clicks the “Submit” button. The schedule still updates dynamically as the admin selects the radio buttons.
- We chose not to add the functionality of selecting all for a given day of the week because we wanted to prioritize safety (i.e. not accidentally selecting all days) over the added efficiency because our user population is not very familiar with computers. In addition, the only time a user is likely to select all of a given day of the week is when submitting their initial schedule, and this task is only performed once a year.