Design
Describe the final design of your interface. Illustrate with screenshots. Point out important design decisions and discuss the design alternatives that you considered. Particularly, discuss design decisions that were motivated by the three evaluations you did (paper prototyping, heuristic evaluation, and user testing).
Implementation
Describe the internals of your implementation, but keep the discussion on a high level. Discuss important design decisions you made in the implementation. Also discuss how implementation problems may have affected the usability of your interface.
Evaluation
Describe how you conducted your user test. Describe how you found your users and how representative they are of your target user population (but don't identify your users by name). Describe how the users were briefed and what tasks they performed; if you did a demo for them as part of your briefing, justify that decision. List the usability problems you found, and discuss how you might solve them.
We found our users by asking people who we knew were not experts in the language but had at least academic exposure. These people are part of our target population because they had a desire to review the vocabulary they already knew.
Briefing
Thank you for participating in our user interface test. Your participation will allow us to find problems with our interface and will help us to build a more user-friendly interface.
In the following user testing, you will be reviewing Mandarin Chinese. The review uses the textbook "Learning Chinese: A Foundation Course for Mandarin Chinese", and you want to practice reading sentences that use the words found in this book. To do so, you're using Reading Practice, a web application that helps members practice reading in their language of study by providing a database of sentences in that language to read.
In order to help us understand the user’s experience, we ask that you think aloud and ask about any uncertainties that you may have. Remember, this test is entirely voluntary, and you may stop at anytime.
Scenario Tasks
1. Create an account and log in.
2. Enable all vocabulary from Chapter 1 of your textbook, "Learning Chinese: A Foundation Course for Mandarin Chinese" to be displayed in sentences.
3. Find a sentence containing 好 (hao4), read it, and find out its English translation
4. This sentence contains 忙. Find out its pronunciation and definition, and find another sentence containing it. (In the event that the displayed sentence didn't contain this word, one of the words present in the currently displayed sentence was used for this task).
5. Prevent 我 (wo3) from appearing in subsequently fetched sentences
6. Make it such that the 2 sentences currently being displayed in the "Currently Reading" tab are no longer displayed.
7. Restore one of the sentences that you just removed from display back to the "Currently Reading" tab.
8. Find another sentence containing 好. Note that this was the previous study focus.
9. Now you want to return to a general review of all words you know, and aren't focused on studying a particular word. Switch to a general review of all words, and fetch 2 sentences for review.
10. Contribute a sentence.
Observations during User Testing
User 1:
This user grew up in a China for part of her life and moved to America when she was 12 years old. She said that she hadn't practiced reading in a long time but wanted to be able to review. Hence, the user could benefit from using this application, and thus matches the target audience.
Task 1
The user had no trouble creating an account. Although at log in, the user complained about waiting too long at the loading screen.
Task 2
The user selected the correct book and chapter. However, the user did not understand how to enable the words at first. At first she attempted to press the "Make study focus" buttons, but then realized that clicking multiple buttons undid the actions of previous button clicks. After realizing this, she then found the "Check All" button and used it.
Task 3
The user did not know how to complete this task alone. She did not know what the "Make study focus" button was supposed to do. When told what the button does, she did not know how to fetch a sentence. She complained that the panel below the 'fetch next sentence' button was blank and unintuitive what the panel was used for. She also complained about the wording used in the button, saying that it wasn't descriptive of the task of getting new sentences using checked words. She suggested teh wording "find sentences with checked words".
Task 4
The user had no problem with looking up the definition using the clickable buttons in the sentence and the translation. She focused the word by clicking the button on the popup.
Task 5
The user didn't have a problem forbidding the word from being shown.
Task 6
The user had no problem closing the sentences.
Task 7
The user had trouble locating the Closed Sentences tab and restoring the sentence. Although, she did complain about cosmetic issues. In particular, she thought there should be more horizontal spacing of sentences.
Task 8
The user had trouble noticing the study focus history button. So she searched for the desired focus word instead in order to change the focus. When asked why she didn't notice it, she said that it was too small and not obviously placed. This suggests that we need to design a way to grab the user's locus of attention better when the study focus is changed so that users will understand where the button is.
Task 9
The user had no problem with this task after having clicked the study focus history button in the previous step. However, the user suggested making the state of the system obvious. For example, she suggested using tabs to indicate that the state is in either "General Review" mode or "Study Focus" mode.
Task 10
The user had no problem with this step.
User 2:
This user grew up in a Chinese-speaking environment, and has basic reading proficiency. He was able to read the sentences that he encountered while doing the tasks for this user test. However, after the user test, when he enabled the remainder of the vocabulary, there were words that he needed to click on to discover their pronunciation and meaning. Hence, the user could benefit from using this application, and thus matches the target audience.
Task 1
The user had no difficulty going to the Create Account tab, entering his username and password (switching between the fields using tab), and having the account created and logged in after pressing enter. The user initially typed the repeat of his password in incorrectly (apparently deliberately), and was pleased to see the immediate feedback that appeared that the entered passwords were different.
Task 2
The user had no difficulty locating and using the Textbook and Chapter selection combo-boxes on the left side. However, after the chapter was selected, the user wasn't entirely sure what to do. He first clicked on the "Displayed" sorting button (which sorts the words according to whether they are allowed to be displayed or not). Then he checked the "May appear in sentences" option for some individual words. At that point he believed he had accomplished the task, and started fetching sentences. After being informed that he hadn't actually accomplished the task, the user noticed the "check all" button and clicked it (which is how we had expected the task to be done).
The user later cited that part of his source of difficulty was that the "May appear in sentences" label was cut off (displaying only "May appear") because the user's browser window was not maximized, and was at the time smaller than the minimum size we had designed for. However, the larger usability issue in this case was that the "check all" button's location did not make it discoverable, or that its label did not carry sufficient information scent.
Task 3
The user first tried pressing Ctrl-F to search for the word using the browser's built-in search. In our case, however, because we had implemented the site using Silverlight not HTML, the browser can't search the page. After noticing that he wasn't getting when searching via the browser's built-in search, he quickly switched to the search box. The user didn't have any issues locating the search box or using it. After typing in the pinyin for the word, he found the word in the search results, but initially he checked the "May appear in words" checkbox, and didn't click the "make study focus" button. Only after fetching a few sentences and noticing that the desired word was not present did he focus his attention back to the left sidebar, and clicked the "make" study focus button. He clearly noticed the change this time, as he hovered his mouse around the (now highlighted in green) combobox displaying the word in the "study focus" section and faintly mumbled "ah that's it". He then pressed the "fetch next sentence" button to fetch the sentence, as expected.
Task 4
The user clicked on the word in the sentence to display its pronunciation and translation. He clicked the "make this word the study focus" as expected to make it the study focus, and then pressed the "fetch next sentence" button to fetch the next sentence.
Task 5
The user typed the pinyin in the search box to locate the word, and unchecked the "May appear in sentences" button. He then pressed the fetch next sentence button a few times to verify that the word indeed didn't show up to convince himself that he had actually accomplished the task.
Task 6
The user had no difficulty locating the close button for the button and clicking it.
Task 7
Immediately after accomplishing the previous task and before the user was shown this task, the user switched to the Closed Sentences tab and pressed the "Remove sentence" button repeatedly until all the sentences were removed. Thus, because there were no sentences to restore, the facilitator asked the user to first fetch a new sentence and redo the previous task before proceeding. After fetching the new sentence and clicking the Close button, the user switched to the Closed Sentences tab, and pressed the Restore Sentence button. He then switched back to the Currently Reading tab, saw the sentence he had just restored, and repeated the process of closing it and restoring it from the Closed Sentences tab to verify that that its behavior indeed matched what his mental model.
Task 8
The user went to the Study Focus sidebar, clicked the combo-box, and selected selected the word in the drop-down menu as expected.
user immediately discovered how to sort sentences according to "displayed"
took user a while to figure out to click the check all button
user first attempted to use the browser's built-in search Ctrl-F to search for finding a word, but quickly discovered it didn't work and switched to the search box
user checked "may appear in sentences" to display words, instead of study focus. However after not seeing the word in the fetched sentences, he noticed the make study focus button and used it
User 3:
Reflection
Discuss what you learned over the course of the iterative design process. If you did it again, what would you do differently? Focus in this part not on the specific design decisions of your project (which you already discussed in the Design section), but instead on the meta-level decisions about your design process: your risk assessments, your decisions about what features to prototype and which prototype techniques to use, and how you evaluated the results of your observations.