...
In addition, we added a tutorial for the user when they enter the website. There is also a help link on all the pages, so the user can go back and view the tutorial.
User Testing
- User 1 (Election employee): Please note this test had to be done on Internet Explorer, which did not support all functionality. This is because the office had no access to wi-fi and we had to use her machine.** She was surprised that our program had glitches on Internet Explorer, since she mentioned all offices are set up with Internet Explorer.** Wanted to have the layout from the fix mistake menu to be the primary layout, she found the pie-chart confusing*** She said it would be faster to see things laid out in the same order as the ballot.
- Most other comments were specific to IE glitches and are hence not mentioned here because they are concerns we addressed in our actual project.
- User 2 (Town Clerk in Concord, MA): ** Didn’t seem to understand that the main screen tutorial followed into a “fix mistake menu” – she was going through the tutorials but didn’t enter the fix mistake menu. It might be helpful if we made it clearer that the tutorial continues on that page.** Despite going through the tutorial which states that Democratic Candidates are on the top, Republican on the bottom and others on the left, she was confused why some candidates ended up on the left.
- “Write-In, where do I do that?” She wanted a space to enter who the write-in candidate was.
Evaluation
Evaluation of the Issues discovered during the User Tests:
| Issue | Severity | Possible Solutions || Didn’t notice that the Fix Mistake Menu had a tutorial | Minor | Despite not using the tutorial, she was still able to navigate the menu and complete her tasks. This is just a matter of changing the wording at the end of our first tutorial to make things clearer. || User wanted to try the tutorial but accidentally clicked “Skip” instead of “Continue” | Cosmetic | We think this is because the “Skip” button has a brighter color (red). This should be an issue of just changing the button colors to be more intuitive. || Expected to see a ballot-like UI, which listed candidates in order. Commented that this felt slower. (Current approach is based on a pie-menu) | Minor | This is a new UI, but we put a lot of thought into how to make this most efficient. After a bit of practice, we think this would be faster. This implies our learnability could be higher but the tradeoff is for long-term efficiency. || The order of candidates on the page seemed random | Minor | Again, we considered efficiency and placed the candidates based on party and frequency with which that party is selected. Once again this is probably a trade-off we consciously make between learnability and efficiency.
Perhaps we could change the button colors to be slightly tinted with the party colors to make it clearer how we chose this arrangement. || No place to write in the name of a write-in candidate | Minor | We were told in our first interview that usually the write-in ballots are put to the side and not recorded, so we are handling the needs of many of our users.
We can improve this by adding a modal asking for the name any time a write-in is selected but also including a “skip” option for users that don’t want to enter the name. This way we can serve both needs. || User didn’t see the “cancel” button to return from the fix mistake menu | Minor | She just went back to the previous element so it was not too inefficient.
We can make that button larger or featured more prominently to make it more obvious. || “This would be so much better than the hand counts” | Good | She thought this system as a whole was a success and an improvement on the existing methods. || Restarting from a mistake somewhere on this ballot or the previous ballot rather than allowing a fix of just one mistake was a good decision for elderly users | Good | She thought it was very helpful that the elderly users won’t need to jump around pages but can just back track a little bit to be sure of accuracy. || Wanted a “fix one mistake” option for ballots more than 2 ago. Currently we assume that means the audit is off and requires a reset. | Minor | Mistakes that long ago were not a major concern of hers, and she said this makes sense too. |
- User 3 (Worker at the office of the Town Clerk in Concord, MA):** Accidentally skipped tutorial when she wanted to go through it
- Wasn’t quite sure how to fix a mistake – originally was going for “reset ballot” but figured out to click on a candidate name before actually selecting anything. So she figured it out, it just took a moment.
- Correctly walked through fix mistake menu without a tutorial** Though didn’t realize you press “ok” to reset.** Didn’t see the “cancel” button to go back
- To fix two ballots ago, didn’t think of restart audit. Didn’t see restart audit.** Would prefer just changing one, thought that was more likely than the off-by-one errors we were concerned about.** Restarting from somewhere on this ballot on the previous one is fine, not too much
- Actually very helpful for the elderly users.
- Button placement seemed random other than blank and write-in “If the system ever went down, this would be much nicer than doing the hand counts”
Reflection
We originally had considered two user classes: the auditor and the warden. In our subsequent GRs, we focused on the auditor's needs, but we haven't addressed the warden. The warden acts as an administrator and would have the ability to create/assign/view results. Without the warden account, our website just focuses on the input side of the election, not really the output, and isn't really meaningful in the context of an election. We've discussed the possibility of maybe just showing the results after the audit, but then our website is only relevant for the 2012 presidential election (the warden would have the ability to create new elections). In our work before GR4, we created a nearly complete computer prototype of the warden and then decided to focus on the auditor and throw that out. In re-doing this process, we should have identified the need to narrow our project down to just the auditor earlier and stuck to that.
...