Design
Design Overview
Our final design drew heavily from our original Designs #1 and #2; it is based on modes, and the display resembles a calendar. When inputting or modifying a schedule, a user clicks a button on the sidebar to select the mode, which is either a discipline to schedule a shift, the eraser to clear a shift, or the arrow to reset the cursor and mode. If the user is an admin, there are also buttons to mark shifts as excused or absent, and to clear these markings.
| |
Once a mode is selected, the cursor changes to match the icon on the button, and the calendar updates so that only relevant shift buttons are enabled. For example, if the mode is a discipline, only shifts that are valid to schedule and are not already scheduled with that discipline are enabled.
Racing shifts only occur during the Saturday Day shifts.
Instructor Interface
Until an instructor is done submitting his initial schedule, there is a message indicating that the initial schedule has not yet been accepted. After hitting Submit Request, the schedule is marked as ready for approval by an admin.
After the schedule has been approved by an admin, any changes an instructor makes to his schedule are marked as pending. A pending shift has a gray border around it, and is accompanied by a message at the top of the screen detailing the pending shift.
The pending changes after Sam requests to add an adult ski day shift on 12/16/12, and delete his adult ski evening shift on 12/16/12.
After an admin approves or rejects the changes, the border and message disappear. If the change is rejected, the shift button image also returns to its previous state.
Admin Interface
After an instructor has submitted his schedule for approval, the admin can approve the entire schedule, along with any changes he has made, by clicking on the Accept button at the top of that instructor’s schedule.
If an instructor has any pending changes, the admin can approve them using the radio buttons above the calendar. As in the instructor interface, anything marked “Approve” will change and the pending border will disappear, any change marked “Reject” will cause the shift to return to its previous state, and anything marked “Clear selection” will keep the pending border and leave the decision unmade. If a pending change was to delete a shift, accepting that change does not empty that shift button on the calendar, but rather marks it as excused.
An admin can mark a shift as excused or absent using the corresponding buttons on the sidebar. These letters will appear on the shifts in the instructor interface as well.
Design Decisions
Originally, we chose a calendar-based interface with modes because we believed the modes would allow for efficient shift scheduling or modifying once a mode is selected, and that the calendar was an externally consistent interface which would improve learnability for users who are not as comfortable with computers.
During our paper prototype testing, we found that users were confused by some of our labels, so we made them more clear. We also added the “Clear Excused” and “Clear Absent” buttons, which hadn't been there before. For the pending changes from the admin view, we had initially had two buttons for each message, one for accept and the other for reject, but users found that this confused their mental model, so we replaced the buttons with radio buttons.
In our heuristic evaluation, the main issues we found were that users were not always sure of which mode they were in, did not know if their requests had been submitted, and thought that our icons were far too complicated. As a result, we added a black border around the current mode, added a confirmation dialog box after hitting Submit or Submit Request, and simplified the icons representing the three shifts.
During the final user testing, we found that the arrow button in the sidebar panel confused users, so we added the text “Reset cursor” to label that button. Also, when logged in as an instructor, our users were unsure of what E and A represented, so our next iteration would add a legend explaining that E is for excused shifts and A is for absent shifts. Finally, our users suggested we have a more visible way to tell that a shift is pending, for example adding a P (similar to the E and A) to indicate a pending shift, in addition to the border, and this would likely be added to our next iteration.
Implementation
The biggest design decision we made for our implementation was to use the Django framework. We chose to use a framework in general, because it would allow us to make our application much more robust with much less code. We chose to use a database, because it provides an easy and persistent way to store data. We specifically chose Django, because it provides an excellent admin interface, which we could use without having to actually write it and which makes it possible to edit the database without being a technical person. This was important for our project because it makes it so that the administrators at the ski school could actually directly edit the database if need be even though they do not have any coding abilities. Django also provides authentication methods, which made implementing login very easy.
Additionally, Django models enables us to easily follow an MVC pattern, since the models make it easy to separate the model from the controller. Furthermore, we choose to keep all of our CSS in a separate file to separate out the view from the controller. By following MVC, it was much easier for us to make changes to the view as we got feedback from HW2, because many of the changes, namely the cosmetic ones, only required editing the CSS, but did not affect any of the logic.
We also used Django templates. This allowed us to inherit the heading and navigation bar and also allowed us to use the same views for the admin and instructor interfaces simply by adding in if-statements. All of these features of templating allowed us to write significantly less code than we otherwise would have had to write, which allowed us to implement more features and reduce the number of bugs that could be introduced.
We used javascript to implement much of the logic. We chose to draw much of the calendar with javascript, rather than doing it directly in the html. This makes our application unusable for users without javascript enabled browsers, which could be a usability problem, but since pretty much all modern browsers have javascript enabled, we don’t expect it to actually be a problem. We chose to implement the calendar in the javascript because doing so made it very easy to keep a copy of the working schedule in memory on the client side without editing the database. This made it very easy for users to cancel all changes in a session simply by going to another page (we do warn users before they do this), but also limited the ability of the application to recover any unsaved changes if the browser crashed.
The implementation decisions that we made allowed us to use many of the Django features, thus enabling us to focus on the most interesting user interface portions of our application design, while still providing a fully functional application.
Evaluation
User Testing
How it was conducted
We conducted our user tests via Skype and asked the users to share their screen with us, so that we could see where they were clicking. We ran our website on one of our computers for the duration of the test and gave the users a URL to go to.
For our first two user tests, we had the users perform tasks both as an admin and as an instructor. However, users in reality would only either be an admin or an instructor. Our third user test involved the user only performing the instructor’s tasks in order to see what usability problems might arise from not having knowledge of the admin interface.
Our users
Since one of our group members (Julia) is an instructor at the small midwestern ski area for which we made our user interface, we were able to contact other instructors and have them be our users for this round of testing. Therefore our users for testing are actual users in the user population. They are also very representative of the user population in that their proficiency with computers vary.
Briefing
We did not do a demo as part of our briefing.
For User Tests #1 and #2
Thank you for volunteering to test our ski school scheduler.
Tasks
Today we will ask you to perform tasks both as a normal instructor of the ski area and as an admin. We’ll give you separate login names and passwords for each type of user and instruct you which logins to use at any given time. The program requires admins to approve entire schedules at the beginning of the year. After that, if an instructor wants to make a change to their schedule, each individual change must be approved by an admin. Normal instructors are only permitted to view/make change requests their own schedules. Admins can view and change all instructors’ schedules.
Do you have any questions?
[pause for questions]
You will greatly help us improve our interface if you think aloud as you go through the tasks we will give you. We welcome any and all feedback, so please say anything that comes to mind!
For User Test #3
Thank you for volunteering to test our ski school scheduler.
Today we will ask you to perform tasks as a normal instructor of the ski area. We’ll give you the login name and password for the instructor. The program requires admins to approve entire schedules at the beginning of the year. After that, if an instructor wants to make a change to their schedule, each individual change must be approved by an admin. Normal instructors are only permitted to view/make change requests their own schedules. Admins can view and change all instructors’ schedules.
Do you have any questions?
[pause for questions]
You will greatly help us improve our interface if you think aloud as you go through the tasks we will give you. We welcome any and all feedback, so please say anything that comes to mind!
Tasks
Task 1: Log
...
in as the instructor Sam (username and password sseaborn):
...
- Create your initial schedule before the season begins.
- Submit your schedule for approval.
- Log out when you’re done.
...
Task 2: Log in as the admin Darryl (username and password dsmith):
...
- Approve Sam’s schedule.
- Log out when you’re done.
...
Task 3: Log in as Sam
...
- Request to remove yourself from a shift.
- Request to add yourself to a different shift.
- Oops! You requested to add the wrong shift. Delete that shift and add a different one.
- Log out when you’re done.
...
Task 4: Log in as Darryl:
...
- Approve one of Sam’s pending changes.
- Reject one of Sam’s pending changes.
- Mark Sam as being excused from one of his shifts.
- Mark Sam as having been absent from one of his shifts.
- Log out when you’re done.
Our third user
Usability Problems
(For the third user test, we asked the users to perform the instructor's tasks 1 and 3, and one of us did the admin's tasks 2 and 4.)
Usability Problems
- CATASTROPHIC problem: Instructors have no idea what the red E and the blue A mean unless they have seen the admin interface.
- Fortunately, we figured this out because one of our users were forgetful and kept thinking that A stood for "approved" rather than "absent" even though this user had seen the admin interface.
- Possible solution: Add a legend to the normal instructor interface indicating what these letters mean.
- MAJOR problem: Users are unclear what the button with the white arrow (which reverts their cursor to a normal mouse cursor) does.
- Our first user testers spent a significant amount of time trying to figure out what this button did.
- Possible solution: Label the button with the text "Revert cursor". We actually implemented this change before the third user test.
- MAJOR problem: Drawing a gray box around shifts that are pending add or pending delete does not convey the pending status to the user.
- Our users did not even really notice the gray box; instead, they noticed the text that changes to indicate which changes are pending approval.
- Possible solution: Instead of or in addition to the gray box, overlay a large letter P on top of the pending shift, similar to how we indicate excused and absent with the letters E and A. This possible solution was suggested by one of our users.
- MINOR problem: The majority of the users during user testing expected instructions.
- We explained to them that the interface is ideally learnable without needing to read instructions.
- Possible solution: Put instructions on a separate page that is accessible via the navigation bar. This way, users who feel more comfortable with reading instructions before trying the scheduling program can do so, but nobody will be obligated to read instructions.
- MINOR problem: It is easy to miss the "You must submit..." message at the top, since users' attentions tend to be drawn to the calendar.
- We categorized this as a minor problem, since none of our users had any trouble remembering to press submit.
- Possible solution: Make the message more contrasting, perhaps with a different font color. Also position the message so that it is slightly lower on the page and closer to the calendar.
- MINOR problem: On the admin interface, the users did not immediately notice the radio buttons used for accepting/rejecting requests.
- Admins would probably remember where the radio buttons are after using the interface a few times, so this is a minor learnability problem.
- Possible solution: Make the background of the table that holds the radio buttons a contrasting color (maybe the same light color as the background to the calendar), so that it draws attention to itself.
- MINOR problem: The wording of the message on the popup box that appears if the user tries to navigate away from the calendar does not make it extremely clear that the user must press submit to save changes.
- Again, this is a minor problem because our users always remembered to press submit before leaving the page.
- Possible solution: Change wording of the message on the popup box to "Your changes will be lost if you do not press submit. Are you sure you want to leave this page?" or some other message that makes it explicit that users must press submit to save changes.
- COSMETIC problem: The color scheme (yellow/blue/black to represent day/evening/night) the buttons have when users schedule shifts is not extremely intuitive until the user has many shifts scheduled.
- Only one of our users mentioned that she did not know what the colors mean, and an instructor can probably still figure out his schedule without the colors being there.
- Possible solution: Incorporate these colors into the day/evening/night legend.
Reflection
Over the course of the iterative design process, we learned how important planning is. While had we not been forced to follow an iterative design process, we likely would not have done nearly as much planning near the beginning of our interface, we found that implementing our final version was much easier because of all of the planning we had put in early on. Furthermore, we learned the importance of user testing because there were things that we thought would be very intuitive, but that users sometimes couldn’t figure out. Since we found out many of these problems early on in our paper prototype, we were able to fix them much more quickly than had we not found them until after our final implementation. Finally, we learned the importance of testing on users that are representative of our user population. We found that our paper-prototype testers (who were all MIT students) often wanted features that would make the application more efficient to use; however, when we tested on our actual user population (much less technically savvy), their primary concern was just being able to figure out the interface. We did a good job of recognizing this and using our intuition to guide how we actually responded to user feedback; however, it would definitely be more ideal to use actual user feedback rather than having to rely on our intuition for how our testers differed from actual users.
If we were to repeat the process, it would be ideal to do the paper prototype testing on users who were more representative of our actual population. We also should have done more testing where users only played the role of either admin or instructor, rather than having users switch back and forth between different types of users. We discovered in our final testing, that we didn’t have a legend for some of our symbols in the instructor interface but that the symbols were explained in button labels in the admin interface. The only reason that we discovered this was because one of our testers forgot what she had seen in the admin interface when acting as the instructor. Had she not forgotten this, we might have not noticed a very important learnability problem. To prevent this problem in the future, we would be better off having each tester only test as only one type of user so that things that they learn in part of the test would not unduly influence how they act in other parts of the test.
Overall, the iterative design process that we followed was very helpful to designing an interface that our testers loved at the end of the process, while also enabling us to implement the user interface in a timely manner.
...