Design
Describe the final design of your interface. Illustrate with screenshots. Point out important design decisions and discuss the design alternatives that you considered. Particularly, discuss design decisions that were motivated by the three evaluations you did (paper prototyping, heuristic evaluation, and user testing).
Login and Account Creation
The user begins by arriving at the login screen. We aimed for simplicity in this screen: there is a login and create account tab, each of which contain labeled fields. The create account tab has some features designed towards fast error detection, namely that a message will be displayed if the username is already taken, immediately as the user is typing; likewise if the repeated password doesn't match the original, a message appears when then user finishes typing (as opposed to only making these errors visible once the user clicks the create account button). During user testing, User 2 noticed the immediate detection of the mistyped password, and commented that it was a useful feature. Additionally, the login and create account allow for the keyboard to be used for efficiency; namely the user can tab through the input fields, and the user can press the Enter button as opposed to needing to click on the widget. During testing, Users 1 and 2 both created their accounts by pressing the Enter key instead of clicking the widget, so this feature turned out to be useful.
Main Page
Once the user logs in, he is greeted with the main page, which is split into a left sidebar for displaying and managing the study focus and vocabulary, as well as a right half which displays sentences. We chose to have both of these displayed at once, as opposed to haivng the vocabulary management and sentence viewing components in separate tabs, based on the results of the first paper prototyping iteration, where users encountered visibility issues due to having the vocabulary hidden away in a tab, and were generally dissatisfied about having to frequently switch between the two tabs.
...
The purpose of the study focus is to highlight the user-selected vocab word that will be used in each new example sentence generated. The design originally had the study focus as a text-display of the word currently being the focus of all the example sentences. However, due to user complaints about both visibility and navigation control, the study focus display was changed to a drop-down selection, which keeps a history of all the previous study focuses. Additionally, the popped-out drop-down selection also improves information scent. Finally, the study focus word is color coated as light blueteal, which extends to the color coating of the study focus word in the example sentences.
...
With regards to our sentence segmentation code, we use a lexicon-based segmentation approach. Thus, if a user contributes a new sentence, then provided that the words are in the lexicon, the sentence will be properly segmented, and the sentence can be found by making any of those words it is segmented into the study focus. Because our lexicon is extremely large and comprehensive (several megabytes), then the words in a sentence are usually found in the lexicon, so the sentence is properly segmented. Additionally, this means that when a user contributes a sentence, he doesn't need to segment the sentence himself, or provide any tagging for the words - he simply inputs the sentence, and it is automatically segmented and annotated with its vocab words. However, this automatic lexicon-based segmentation and word extraction also prevents users from being able to use out-of-lexicon terms in their contributed sentences (they can use them, but they will not be annotated with that vocab word).
The server-side code, which stores login credentials and the study focus, study focus history, words that are currently allowed to be displayed, sentences that are currently displayed, and contributed sentences for each user, is implemented as set of ASP.NET scripts which communicate with a MySQL database instance.
One design decision we made to simplify implementation was that all information - namely, the list of sentences, the words, and their translations and romanizations - would be retrieved from the server at login time. This was meant to ensure that no latency from communicating with the server would be seen once the user has been logged in and is using the main interface. However, one usability issue which resulted was that login times tend to be long due to all the downloading that is taking place. We managed to alleviate this by having shared portions of the data that would need to be downloaded by all the users (for example, the word database) be downloaded in the background while the user was still entering his login credentials at the login screen. Additionally, we have a loading screen with an animated progress bar to provide feedback for the user while the page is loading. Another issue which results from our decision to download all data from the server at login time is that if other users contribute a sentence, then the sentence won't be observed until the next time the user logs in.
Another design decision we made is changes made by the user (the study focus history, the words that are allowed to be displayed, the lists of displayed sentences, and the contributed sentences) are sent immediately to the server. This was done to ensure that the user doesn't have to press a "save" or "logout" button to preserve changes (which would have been a source of error if the user closed the browser without saving changes or logging out). However, because there would have been high latency (leading to an unresponsive interface) if the application waited for a response from the server in the GUI thread, then these requests are instead queued by the application and sent in a separate thread. An unintended consequence is that if the user makes a change (say, selects a new study focus) and immediately quits the browser before the server communication thread can send the update to the server, then that change will be lost when the user next logs in. However, because the system is still in a consistent state upon login (namely, it is in the state it was in immediately before the action occurred), and the user can easily observe the system state and redo the action, then this is not a severe issue.
Evaluation
Describe how you conducted your user test. Describe how you found your users and how representative they are of your target user population (but don't identify your users by name). Describe how the users were briefed and what tasks they performed; if you did a demo for them as part of your briefing, justify that decision. List the usability problems you found, and discuss how you might solve them.
We found our users We found our users by asking people who we knew were not experts in the language but had at least academic exposure. These people are part of our target population because they had a desire to review the vocabulary they already knew.
...
The user first tried pressing Ctrl-F to search for the word using the browser's built-in search. In our case, however, because we had implemented the site using Silverlight not HTML, the browser can't search the page. After noticing that he wasn't getting when searching via the browser's built-in search, he quickly switched to the search box. The user didn't have any issues locating the search box or using it. After typing in the pinyin for the word, he found the word in the search results, but initially he checked the "May appear in words" checkbox, and didn't click the "make study focus" button. Only after fetching a few sentences and noticing that the desired word was not present did he focus his attention back to the left sidebar, and clicked the "make " study focus" button. He clearly noticed the change this time, as he hovered his mouse around the (now highlighted in greenteal) combobox displaying the word in the "study focus" section and faintly mumbled "ah that's it". He then pressed the "fetch next sentence" button to fetch the sentence, as expected.
...
The user had no issue with this task - as expected, he went to the Contribute Sentences tab, and inputted a sentence using his computer's Chinese IME, the English translation, and submitted the sentence.
User 3:
Usability Problems Identified during Testing
As seen with User 2 on Task 3, our interface is inconsistent with the usual browser search functionality. This is of course an artifact of our decision to implement it in Silverlight rather than HTML. Although we can't have the browser search functionality work as expected without rewriting the interface, we could intercept the Ctrl-F shortcut and have it focus the search box, as searching for words is the most likely reason why the user would press Ctrl-F on the interface.
Reflection
Discuss what you learned over the course of the iterative design process. If you did it again, what would you do differently? Focus in this part not on the specific design decisions of your project (which you already discussed in the Design section), but instead on the meta-level decisions about your design process: your risk assessments, your decisions about what features to prototype and which prototype techniques to use, and how you evaluated the results of your observations.
This user lived in China until her middle school years. She is fairly proficient in reading Chinese, but she still regularly encounters words she cannot read. Thus, she could benefit from using this application to gain exposure to and master more difficult vocabulary, and is thus part of the target user population.
Task 1
The user had no difficulty with this task. She used tab to switch through the fields, and pressed Enter to create her account.
Task 2
The user quickly selected the textbook and chapter from the combobox. However, after doing this, she was at a loss as to what to do. She explored the interface, first checking the "May appear in sentences" checkbox for a few words, then pressing the "Make study focus" button for a few words. She then clicked the buttons controlling the sort options. Then, she fetched a few sentences, but she still (correctly) believed that she hadn't accomplished the task. At this point, she gave up on the task, and we proceeded to the next one.
After the user experiment, we asked the user what the "May appear in sentences" checkbox did, and she indeed had the model of what it did. However, she cited that "Check words below" / "Uncheck words below" buttons were rather far away from the checkboxes that they mainpulated, hence the relation wasn't apparent to her.
Task 3
Upon seeing the word we had asked for, 好, the user remarked that this was a very common word, so the user first began clicking "fetch sentences" repeatedly, knowing that a sentence containing 好 was eventually going to appear. After a few sentences, however, she realized on her own that this wasn't the way we had intended her to accomplish the task, so she clicked on the search box, typed in the pinyin, and pressed the "Make study focus" button for the word, and fetched the sentence.
Task 4
As expected, the user clicked on the word to find out its prononciation and definition via the popup. She then clicked on the "Make study focus" button in the popup to make it the study focus, and fetched a sentence.
Task 5
As expected, the user typed in the pinyin into the search box, and unchecked the "May appear in sentences" checkbox.
Task 6
The user clicked on the "Close" button for the sentence, as expected.
Task 7
The user went to the "Closed Sentences" tab, and clicked the "Restore" button as expected.
Task 8
The user remembered that she had previously fetched a sentence containing 好, and that it was in the "Closed Sentences" tab, so she went to the tab, located the word in the sentence, clicked it, used the "Make study focus" button to make it the study focus. Immediately after she did so, however, her attention was drawn to the top-left side where the study focus had just changed, and she noticed that the widget displaying the study focus was actually a combobox, so she clicked it, saw that the history of study focuses was displayed there, and (correctly) remarked that the way we were probably expecting users to do the task was using that combobox. Thus, the user was able to discover the intended way to display previous study focuses on her own.
Task 9
The user clicked on the study focus combobox and selected General Review as expected.
Task 10
The user went to the Contribute Sentences tab and contributed a sentence as expected.
Usability Problems Identified during Testing
"Check words below" button has poor information scent
The largest usability issue we encountered was that all 3 of our users had difficulty discovering the "check words below" button in Task 2 - the first two users took a long time until they located and used it, and the third user never found it at all and gave up on the task. During our earlier tests in the paper and computer prototyping stages, however, users didn't have such difficulties locating the buttons or discovering what their functionality was. There were 2 changes that we ascribe to this regression. Firstly, because evaluators had remarked that the label "Allow all words below to be displayed in sentences" was too verbose, we had shortened the label to "Check words below". However, this label has less information scent, because the button itself doesn't say what clicking it actually does, but rather which checkboxes it manipulates; the user has to read the label on the checkboxes, "May be displayed in sentences", in order to understand what the button itself does. The second issue is that we introduced the "Sort by" buttons, again following the advice of our evaluators that items should be sortable via metrics other than just Pinyin. However, because the sort by buttons were immediately above the list of words, then they further separated the "Check words below" button from the "May appear in sentences" checkboxes that they manipulated.
Our solution to this problem would thus be to adapt a label which has better infromation scent, and describes what clicking the button actually does (for example, "Allow all words below to be displayed in sentences", or perhaps a less verbose version of this), rather than requiring the users to figure out that the button manipulates the checkboxes, and then read the labels for the checkboxes below, before they can determine what the button actually does. We should also remove the "Sort by" buttons for the vocabulary - the buttons conflict with our goal of simplicity on the left sidebar, and they are only useful if the user already knows what word that he's searching for is, but in this case it would be more efficient for the user to simply type the word in to locate it via the incremental search functionality.
"Make study focus" button label assumes user knows what "study focus" means
As we saw from how User 2 didn't initially press the "Make study focus" button on task 3, he apparently didn't understand what "study focus" meant (namely, that the study focus must appear in fetched sentences). We could perhaps have replaced the term "study focus" in our interface with a term which better described what it meant for a word to be the study focus, for example, "word which must appear in fetched sentences".
Study focus history functionality has poor information scent
Although all our users were eventually able to discover that the study focus combobox could be pressed to display the study focus history (Task 8), Users 1 and 3 took a while to discover that it was a combobox which contained the previous study focuses as well, not just the current study focus. This is likely because the only affordance that is provided is a small arrow indicating that it's a combobox - this certainly doesn't pass the squint test. The arrow should be made larger and more apparent; additionally, the label "study focus" above the combobox should be changed to reflect that it can also be used to display the previous study focuses - for example, "current and past study focuses".
Ctrl-F is inconsistent with usual browser search functionality
As seen with User 2 on Task 3, our interface is inconsistent with the usual browser search functionality. This is of course an artifact of our decision to implement it in Silverlight rather than HTML. Although we can't have the browser search functionality work as expected without rewriting the interface, we could intercept the Ctrl-F shortcut and have it focus the search box, as searching for words is the most likely reason why the user would press Ctrl-F on the interface.
Reflection
Our design process was flexible, and revisited even our most core assumptions based on observations made in prototyping. For example, one of the decisions we made early in the design process was our model for which sentences would be displayed. Namely, initially we would not display sentences if they didn't have words that the user had selected as being allowed to be displayed; later we would allow sentences containing words the user hadn't selected as being allowed to be displayed, though these sentences would be displayed last, and the novel words would be highlighted - this noticeably reduced the frustration users had in fetching new sentences. Thus, our design process was holistic and considered all elements of the interface as up for revision, avoiding setting any elements of the design into stone.
The paper prototyping stage helped us establish our initial risk assessments and the features to focus our prototyping efforts on - namely, that the main difficulty was conveying to the user our model that sentences would be displayed only if the words they contained were specified as being displayable in sentences, and included the study focus if there was one. Thus, we focused our prototyping efforts mainly on the vocab and textbook selection, and less on logging in, reading and understanding the sentences themselves or contributing sentences, with which users had little difficulty in early prototyping stages. Of course, we still did include these tasks in our later user testing stages as well to ensure that no regressions were occurring.
The prototyping techniques we used were initially paper for the paper prototyping stage, due to the ease of use of paper. Because we got our computer prototype in a mostly-working state well before the initial GR4 deadline, we were also able to use computer prototyping often during the implementation stage, asking users to attempt to accomplish a specific task on the interface. Because we were using stock widgets and a WYSIWYG designer for the interface, we were able to iterate quickly even with the computer prototype, making this technique feasible and not costing a great deal of time. Sometimes we would make 2 implementations of the same C# interface with radically different GUIs (particularly in experimenting with the sentence viewer and sidebar), and compare them in tasks side-by-side. Had we been coding in HTML and Javascript, where the lack of decent WYSIWYG designers, lack of code modularity, and high time cost of implementation would have made such experiments prohibitively expensive, we would have been forced to use lower-quality prototyping techniques.
In evaluating the results of our user tests, we asked ourselves two questions: did the user understand the model, and did the user locate the appropriate widget for interacting with it. In the event that a user failed a task, our first goal was to determine which of these was the cause: was our model fundamentally being misunderstood, or was the widget simply placed in a bad location? We did this by asking users questions about our model after we finished the main tasks of the user test - for example, "what does it mean for a word to be the study focus"?
Perhaps one thing we should have focused on more during prototyping was specific labels for items. Because item labels are easy to change after implementation, we focused little on them during early stages of testing. However, though we had intended to revise them in later stages, we often pushed it aside for later, and thus many of the usability issues we encountered were partly related to users misunderstanding the meaning of labelsThe iterative design process helped our project to improve a lot. In particular, the paper prototyping stage allowed us to realize that the best design for our user interface would be one that had very few pages because users showed confusion and dislike to the idea. After we implemented the computer prototype, our design stayed the same for the most part, although we did discuss alternatives that we ultimately threw away because we realized that the ideas would clutter the screen or be inconsistent. For example, we discussed using a tree view to display words in the word look-up in the left side bar, but decided that tree views were mostly associated with outlining information rather than displaying information. Looking back, we realize that we should have paid more attention to and experimented more with the wording used in the interface. The problem was apparent from paper prototyping, but we had thought the problem was sufficiently addressed in computer prototyping. However, the problem surfaced again in later stages due to insufficient tests.