Evaluation

User Testing

Users were selected from our network. Because QuickPick is currently focused for students in 6-3, we selected MIT students who were 6-3. Two of them knew they wanted to major in CS when they arrived to MIT. One of them was course 2 first, and then switched to course 6. This is representative of the user population of QuickPick in its current state. When we expand to other majors, our users testing population should be expanded and diversified to reflect this change.  There was no demo, but rather we gave them the tasks (included below) and access to the live site on one of our computers. 

Briefing

You’ve been selected to test QuickPick, a four year course planner for MIT 6-3 Students. There are three tasks for your to complete. When doing so, please voice aloud your thoughts as you try and complete the tasks. If you have any questions, please ask them, but note that we may not be able to answer them. Thanks! 

Task 1: Schedule GIRs

Remember back to freshmen year, all fresh faced and excited. You’re trying to figure out what classes you want to take for the next four years, but first! You have to schedule your GIRs. Look through the options for each of the GIRs, and schedule your freshman year at MIT. 

If you don’t know which GIRs you want to take, select the following schedule: 

• Fall: 8.01, 18.01, 5.111
• Spring: 8.02, 18.02, 7.013

Task 2: Schedule All Four Years

Fast forward one year. You’re about to enter sophomore year! Schedule your next three years at MIT. For your AUS/LAB requirements, look through the different options and select ones you’re actually interested in / have taken. If you don’t know which classes to take, select the following schedule: 

• Sophomore Fall: 6.01, 6.042, 18.03
• Sophomore Spring: 6.02, 6.005, 6.006
• Junior Fall: 6.004, 6.034, 6.172
• Junior Spring: 6.033, 6.046, 6.813
• Senior Fall: 6.828, 6.UAT
• Senior Spring: 6.UAP 

Task 3: Search 

Now that you’ve scheduled your requirements, you’ve got a few more spaces to take some fun classes on your own. Search for a few classes you’d like to take and add them to the schedule. 

If you don’t know what classes to search for, add the following to your schedule: 

• 21L.004 Reading Poetry
• Professor Winston’s AI seminar, Human Intelligence Enterprise
• MAS.110 - Computational Media Design by Professor Bove.  

Usability Problems

Catastrophic: 7.01X does not work

• Fix should be relatively straight forward, there were some parsing issues in our JSON file for bio classes. It was fixed but the fix was not pushed. 

Severe: 0 out of 0

• As mentioned in GR5, our original design idea for the pagination of classes > 9 would be that it would animate, like the iPhone home screen. This was not implemented. Users did not know what 0 out of 1 meant, but because it looked like (and is) a button, they clicked it and found the effect. ⅔ users did this without verbalizing the question. 
• Fix: We were unable to figure out how to do this animation, so fixing is difficult. 

Severe: 18.0X Confusion

• 18.0X was a confusing designation for ⅔ of our users. They eventually got it, but as one of them is MEnging (and has taken 18.03, 18.05, and 18.06), there was some confusion as to what it was supposed to indicate, and for the MEnging user, what they were supposed to do if they have taken both 18.03 and 18.06). We classified this as severe, because the fix will require some design thinking and it caused significant confusion considering how small of an aspect it is. Relabeling the course as 18.03 OR 18.06 will require us to restructure the blocks (or make the font considerably smaller), 18.0X was confusing for ⅔ tested users. “Math”  is too generic of a label. 

Severe: Names of classes

• For classes where the selector screen comes up, the name of the class is not listed. For the course requirements, this is less important. However, for the AUS/LABs, and especially for the HASS classes, just the number designation is not useful (and as pointed out by our TA, any use of just the numbers and not the names would be difficult for the graduate student users). For the HASS classes, unless you knew already which class you wanted to take (and, furthermore, there’s no affordance for scrolling in the  selector screen), its difficult to find a class to find without clicking on a bunch of the options. 
• Fix: Either some sort of classification, whether by major or by topic. Or a search feature? Search would not be too difficult, as we already have it implemented, but if we could tie the classes that people searched for to the requirements (more on this below), this could help.  
• 

Major: Search/Requirements Fulfilling 

• One user felt strongly that they should be able to search for classes using the search bar and by selecting a class, have that fill the requirement that it could fill. For example, if the user searched for 6.170, it should replace the 6.LAB block. The user preferred the search interface to the scroll / visual search interface, especially for the HASS classes. 
• Fix: It would be doable to fix this, but a bit of a time sink. We’d just have to search the “Possible Classes to Satisfy a Requirement” list for the class the user searched for in the autocomplete.  

Minor: New HASS requirements

• QuickPick does not currently handle the new HASS requirements. One of the testers was a sophomore and they pointed this out. 
• Fix: We’d have to allocate new blocks (divs) for the HASS requirements, and then reclassify the classes in the classes JSON file. Because the classifications were parsed automatically from the registrar’s site, this would only require a bit of time to fix.  

Minor: Duplicate classes? 

• You can’t search for/add duplicate classes, however, there’s no affordance to let the user know the reason for the failure (a user tried to add 6.813 after already adding it through the AUS requirement. The user tried to search for 813 again before realizing the cause of failure). 

Cosmetic: Color blindness

• GIRs are read, course requirements are green. While, fucntionally speaking, the colors make no difference, picking colors not linked for colorblindness in 5-7% of the male population might be better.