Briefing:
Thank you for participating in our user interface test. Your participation will allow us to find problems with our interface and will help us to build a more user-friendly interface.
In the following mock scenario, you will be a diligent and studious 11th grade student studying a variant of pidgin English language called Singlish. Your Singlish I class uses the textbook “Singlish for Dummies”, and your class has studied up to and is currently studying chapter 7 of this book. Being the hard-worker that you are, you have discovered “Reading Practice”, a web application that helps members to practice reading in their language of study by providing a database of sentences in that language to read. The paper prototype you are about to be shown is a possible design for the web application’s user interface. You will be using it to practice Singlish.
In order to help us understand the user’s experience, we ask that you think aloud and ask about any uncertainties that you may have. Remember, this test is entirely voluntary, and you may stop at anytime.
Scenario Tasks
1. Find sentences that use only Singlish vocabulary from all chapters up to chapter 7 of the “Singlish for Dummies” textbook, aiming for a review of all vocabulary.
2. Read the 1st displayed sentence and find out its meaning.
3. Find out the definition of “gamen”.
4. Read more sentences that use the Singlish word “gamen”.
5. You believe you’ve mastered the word “gamen”, so return back to reviewing sentences containing all vocabulary (not just sentences containing “gamen”).
6. Find a sentence that uses the word "goondu", and read it.
7. You are offended by the word “goondu” and don’t want it to ever appear in a sentence again.
8. Contribute the Singlish sentence “He is atas”, which means “He is arrogant” in English.
Observations during User Testing
Iteration 1:
One of the main issues with the first paper prototype was that users did not understand the purpose of selecting a textbook. Many people expected to be reading sentences from the textbook. It was not apparent to the users that they were only ‘importing’ vocabulary into their collection of learned vocabulary.
Another issue involved the wording used for the interface. Users felt that the wording was difficult to understand. For example, we used “not known” (instead of “ignore”, which was suggested by a user) to describe vocabulary that the user does not want to see. Two users thought “My Vocab” meant only ‘already-learned-vocabulary’
A third issue was that all users were unsatisfied with the efficiency of the interface. For example, a complete change of environment was necessary for the user to change the words being studied because the user would have to change to the ‘My Vocab’ tab to manage his/her vocabulary settings. This constant switch caused users to feel like they were losing the vocabulary they had just studied. The user would then have to manage his or her vocabulary and then switch tabs to begin practicing again.
All three test users wanted to be able to focus on one specific word and see many sentences for that one word. Only one user knew how to change the settings in the ‘My Vocab’ tab to accomplish this. After we told them an approach to accomplish this, all users thought that the proposed approach (which required going to ‘My Vocab’ and changing the word lists) was unreasonable because it required too many permanent changes in the word lists and would be difficult to revert once finished studying the word.
Iteration 2:
User 5
During briefing, the user was offended by the "Singlish for Dummies" title of the textbook she was studying from - "I'm not a dummy!".
When seeing the instruction list that was displayed by default ("before you can start practicing reading sentences, you must select a language, and the vocab that should be displayed in sentences"), she initially tried clicking on the first instruction itself (even though it wasn't underlined, or ). she commented that the drop-down list for selecting the language was far from the instructions that asked her to select it. However, she didn't have any issues with noticing the drop-down list, or selecting the language.
To locate the textbook and add vocab for chapters 1-7, the user expanded the "vocab in textbooks" expandable menu (in contrast to the previous user, who had typed in the textbook name), and then expanded chapter 1 to see what it contained. Upon seeing that it contained vocab words, she collapsed chapter 1, and individually checked chapters 1 to 7.
After the user finished checking chapter 7, she was initially unsure what to do. She hesitantly (saying she was "just exploring") tried clicking on "focus on specific word" (before having specified any words to focus studying on), but it was greyed out and did nothing. Only after re-reading the instructions displayed in the center did she realize that she should have clicked on "review all vocab", and did it.
The user had no issues with selecting the
In the right hand side,
Iterations made on the Prototype
This is what the user sees upon logging in. The sidebars are initially hidden, and the user is given instructions to select a language from the drop-down menu.
After selecting a language, the sidebar appears. Here, the vocabulary that the user wants to be displayed in sentences can be selected from an accordian menu with checkboxes. This shows the selection of textbooks that appears if the user expands the "Vocabulary from Textbooks" accordian menu. Note that in our first iteration, this functionality was split across 2 interfaces - first, the textbox selection which appeared only upon 1st login (which imported all vocabulary from that textbook, but could not be accessed again), and also separately in the "My Vocabulary" tab. Now, because in this second iteration we have a single interface for importing vocabulary, which is always present in the sidebar, it is more discoverable because it is always visible and the user uses it upon the 1st login.
Upon expanding the entry for a book in the accordian menu, the list of chapters appears. Checkboxes appear next to each chapter for controlling whether the vocabulary items in that chapter should be displayed in sentences. The chapters can themselves be expanded to show individual words, if the user wants more fine-grained control. Because on each level of the hierarchy, the checkbox is available, this interface is more efficient for adding new chapters or new textbooks - one simply checks the checkbox next to the chapter to allow all vocab in that chapter to appear in sentences (whereas in the first iteration interface, one needs to filter the "all vocab" section by textbook and chapter, then use the "select all" option, and move them to the "learning" list to accomplish this task). Because we have only a single list now (as opposed to 3 in the original), this also increases the simplicity of the interface.
The user next clicks on "review all vocab", and this causes the initial list of instructions to be replaced with a sentence. The second iteration interface has a radio box indicating "review all vocab" or "focus on a specific word" - this is to increase visibility of application state, because in the first iteration interface, one would see this only by going to the "My vocab" tab, and observing whether the "studying" list is empty (which is equivalent in this to reviewing all vocab in the 2nd iteration interface), or nonempty (equivalent to "focus on a specific word" in this 2nd interface). Because users found it confusing on what indicating multiple words to be studied would do, then we made it such that "focus on a specific word" can be used for only 1 word at a time, while providing "history" list of previously selected words so they can be referred back to.
Users encountered few issues with the sentence viewer itself during testing, hence it is essentially unchanged compared to the first iteration. Clicking on "Show English Translation" shows the English translation in the accordian menu.
Clicking on a word shows its definition, and an option to toggle whether it will be displayed, and whether it should be made the study focus.
Making a word the study focus (as shown here, "gamen" was just made the study focus) will change the radio button in the sidebar to "focus on a specific word", add the previous word (if any, in this case there was no word) into the top of the history section, and make the new study focus appear next to the "current word" label.
If the user wants to make an arbitrary word the study focus, in this case "goondu", he types it into the "search" section in the sidebar (he can alternatively expand out "All Vocabulary" and find it, though this is likely more time-consuming). This is more discoverable compared to the initial iteration, as in the second iteration, this is always visible in the sidebar, whereas in the first iteration, one had to first go to the "My Vocab" before one could even see the vocab list or search button at all (which caused some users to fail this task with the first iteration).