Reading Practice - Paper Prototyping

Photos of both iterations are on the bottom of the page, because they are used in explaining changes between the two iterations.

Briefing:

Thank you for participating in our user interface test. Your participation will allow us to find problems with our interface and will help us to build a more user-friendly interface.

In the following mock scenario, you will be a diligent and studious 11th grade student studying a variant of pidgin English language called Singlish. Your Singlish I class uses the textbook “Singlish for Dummies”, and your class has studied up to and is currently studying chapter 7 of this book. Being the hard-worker that you are, you have discovered “Reading Practice”, a web application that helps members to practice reading in their language of study by providing a database of sentences in that language to read. The paper prototype you are about to be shown is a possible design for the web application’s user interface. You will be using it to practice Singlish.

In order to help us understand the user’s experience, we ask that you think aloud and ask about any uncertainties that you may have. Remember, this test is entirely voluntary, and you may stop at anytime.

Scenario Tasks

1. Find sentences that use only Singlish vocabulary from all chapters up to chapter 7 of the “Singlish for Dummies” textbook, aiming for a review of all vocabulary.

2. Read the 1^st displayed sentence and find out its meaning.

3. Find out the definition of “gamen”.

4. Read more sentences that use the Singlish word “gamen”.

5. You believe you’ve mastered the word “gamen”, so return back to reviewing sentences containing all vocabulary (not just sentences containing “gamen”).

6. Find a sentence that uses the word "goondu", and read it.

7. You are offended by the word “goondu” and don’t want it to ever appear in a sentence again.

8. Contribute the Singlish sentence “He is atas”, which means “He is arrogant” in English.

Observations during User Testing

Iteration 1

User 1:

* The user felt that the language was difficult to understand. The user thought “My Vocab” meant only ‘already-learned-vocabulary’.

* The user complained about having to click a lot when going through each sentence. The user wanted hotkeys to speed up the process.

* On the ‘My Vocab’ page, the user wanted to drag and drop the words instead of using checkboxes.

* The user complained about the portrait view of the interface and said it would be easier view in a landscape view.

* The user doesn’t like looking at so many lists at once and would rather not have to scroll through them.

* The user wants to see only sentences for “gamen” but doesn’t like having to go back to ‘My Vocab’ to edit the lists in order for her to accomplish this.

User 2:

* On the ‘My Vocab’ page, the user wanted to drag and drop the words.

* The user doesn’t like managing vocabulary on a different tab. She feels like she is leaving the environment when she changes tabs.

* The user wanted a way to focus on one word at a time, but she didn’t know how to.

* The user does not understand why she needs to select a book. It is not apparent to her what it actually does. She thought that she was reading from a book.

* At the end, the user said she felt very anxious the entire time using the interface. She says she felt like all her changes were difficult to revert, and thus she was reluctant to edit any settings.

User 3:

* The user thinks that the book selection page is inefficient. She does not like selecting the chapter from the bottom because it isn’t visible.

* The user tried double clicking the word in the column lists in the ‘My Vocab’ page to try to start learning the word. This an error because this merely checks and unchecks the checkbox.

* The user though she was reading sentences from the textbook.

Overall:

One of the main issues with the first paper prototype was that users did not understand the purpose of selecting a textbook. Many people expected to be reading sentences from the textbook. It was not apparent to the users that they were only ‘importing’ vocabulary into their collection of learned vocabulary.

Another issue involved the wording used for the interface. Users felt that the wording was difficult to understand. For example, we used “not yet known” (instead of “ignore”, which was suggested by a user) to describe vocabulary that the user does not want to see. Two users thought “My Vocab” meant only ‘already-learned-vocabulary’

A third issue was that all users were unsatisfied with the efficiency of the interface. For example, a complete change of environment was necessary for the user to change the words being studied because the user would have to change to the ‘My Vocab’ tab to manage his/her vocabulary settings. This constant switch caused users to feel like they were losing the vocabulary they had just studied. The user would then have to manage his or her vocabulary and then switch tabs to begin practicing again.

All three test users wanted to be able to focus on one specific word and see many sentences for that one word. Only one user knew how to change the settings in the ‘My Vocab’ tab to accomplish this. After we told them an approach to accomplish this, all users thought that the proposed approach (which required going to ‘My Vocab’ and changing the word lists) was unreasonable because it required too many permanent changes in the word lists and would be difficult to revert once finished studying the word.

Iteration 2:

User 4

* The user feels offended by the title “Singlish for Dummies”

* At the welcome screen, the user tries to click the text ‘Select a Language’. This is an unexpected behavior. She says it would be easier if the selection was inline with the instructions.

* When the user is selecting chapters from a textbook, the user doesn’t want things to be happening until after she is finished selecting all the chapters. She is referring to the appearance of options and un-dimming of new available actions when a chapter is selected.

* She thinks that the “+” expand icon is inappropriate for the ‘Show English Translation’ operation. She associates it with outlines and hierarchies.

* She is unsure about the words “Display this vocab” because she thinks unchecking it might make the current sentence disappear.

* After she focused on a word, the popup disappears. She is unsure how she would undo this action.

* She forgot the definition of the word she focused. She suggests constantly displaying the focused word’s definition somewhere to remind users.

* After she stops focusing on gamen and goes back to ‘review all vocabulary’, she is worried that she will see gamen when she goes back to “review all vocabulary”, and she might feel like she didn’t change the settings properly.

* The user succeeded in the task to make the system no longer display “goondu”, but she had initially been looking for words like ‘ban this word’ and felt that unchecking ‘Display this vocab in sentences’ didn’t evoke the same emotions as banning a word from future sentences.

* She was able to find the ‘Contribute Sentence’ feature, but she said she expected to have been introduced to this function directly when first visiting the site.

* The user expected more options to be offered in the sidebar for the word being focused on, such as ‘stop studying this word’ and a definition of the word being constantly displayed.

User 5

* User was fully able to comprehend initial introduction to the objectives, commenting on "Oh my gosh, this is so cute!"

* The initial tasks were aced by the user, having no trouble with language selection (mild complaints about handwriting). However, user got lost briefly while trying to figure out how to select textbooks.

*When trying to select vocab from chapters 1-7 of Singlish for Dummies textbook, user kept trying to brute force by checking next to every chapter, and then checking every vocab. However, upon seeing the check all box automatically being checked after she checked all the vocabs in a chapter, she realized that she can use the check all boxes instead of checking each vocab at a time.

*When reminded that the objective was to view sentences, and not to just check vocab words, user commented "I don't know what to do!"

*Facilitator prodded the user, and user was eventually able to start up the sentence reviewing panel on the right side of the interface. When asked to find out the meaning of a particular word "gamen" in the sentence, user tried to match words using the show english translations option. User was reminded that the translation does not necessarily correspond to sentence word for word.

*User then thought that going through more sentences to "learn" the meaning of the word "gamen.". The user was then told to just be creative. User finally clicks on the hyperlinked word "gamen" in the sentence (the hyperlinked affordance was there). 

*The user was able to complete the task for reviewing sentence for all vocabs successfully (by checking the all vocabs box), search for "goondu" successfully by using the search box, and successfully eliminate sentences containing "goondu" from the group of practice sentences by unchecking the box next to"goonuo." 

*User was able to successfully contribute sentences after initially typing the foreign language sentence in the wrong sentence box (the translations sentence box).

User 6

*During the briefing, the user was very excited to try out the interface.

*In the first task, the user was immediately struck by both instructions from the tasks and instructions from the interface, found language selection hard to locate at the top right half of the interface when reading instructions for language selection in the middle of the right half of the interface.

*The user was confused by the textbook chapter selection task, because unlike during the language selection, information regarding how to select textbook chapters were not part of the initial instructions set on the right side of the interface. 

*After textbook selection, the user was overwhelmed by the number of possible entry points for selecting vocabulary words. There seemed to be a lack of cue as to what to click/type in next. 

*When going through the sentences task, the user expressed confusion about both the selection box and the button on the flash card whose names seemed to indicate redundant functions.

Overall:

Two of the three users did not have issues with selecting the textbook and chapter. The third user was trying to explore and forgot about the task at hand, but was able to succeed at the task when reminded about that her task was to read sentences and not to browse individual words.

The radio box of reviewing/focusing was less successful. Upon indicating to the system that a vocabulary should not be shown, users expected sentences to appear immediately and did not at first think to click the “Next Sentence” button.

Two of the three users were able to immediately use accordion menus. The third clicked on the word itself. Although our initial design only had accordion menus expand when the user clicked the “+” icon, we could still potentially have the entire surface be clickable to expand and collapse.

None of the users had issues with searching for words and making them the focus of study.

Although one user took some time scanning the page to find the “Contribute Sentence”, all the users did find the feature. One user attempted to contribute a sentence without filling out the necessary fields. We will fix this by disabling the contribute button until the required fields are complete.

Prototype Images and Changes over Iterations

First Iteration

On first login, the user sees a screen prompting him to enter a language to study, a textbook, and a chapter. Based on the textbook and chapter selection, the initial vocabulary lists are chosen. Because users were confused about what the textbook and chapter selection did (they thought they were doing so to read sentences from that textbook, not for the purpose of specifying vocabulary - there was a mismatch between the user model and the system model), and this screen appeared only once (hence the user would find it difficult to undo any mistakes made in textbook and chapter selection), we eventually removed this initial screen in the second iteration.

Afterwards, the user would be brought to the reading sentences tab. This is the starting tab for each subsequent login. It begins by showing a sentence using vocabulary that the user has learned (which has been initially determined via the user's textbook selection, and can be edited in the "My Vocab" tab). Clicking on a word in a sentence opens up a popup defining the word, and allowing its status to be changed (either currently being studied, already learned, or not yet known). In the first iteration, the status changing option took the form of a drop-down menu (which caused a visibility problem for one user, as she did not click on the drop-down menu, hence she couldn't see there there was an option in there to study the word).

The My Vocab tab categorizes words by 3 categories: "currently studying", meaning that any sentence that is shown must contain at least 1 item from this menu, "already learned", which are the words that can be displayed in sentences, and "not yet known", which are the remaining words; any sentence that contains a word that is not yet known will not be displayed. Words can be moved between the lists by checking them, and clicking the "move to" drop-down on the bottom, and selecting the target list to move to. Users had issues with this way of moving items between lists - some simply clicked words, expecting options for studying words or displaying example sentences to appear, and some tried to drag-and-drop words between lists. Another issue was that some users were unaware that changes on this screen would effect the sentences that would be displayed in the "Read Senteces" tab when they returned to it - thus, it had poor visibility.

The Contribute Sentence tab brings the user to a page where the sentence and its English translation can be entered and contributed. No users had issues with discovering or using this interface, though one did remark that having to go to a different tab and losing sight of the currently being read sentence was inconvenient.

Second Iteration

The primary changes made in this iteration related to simplifying the vocabulary manipulation interface (the "My Vocab" interface), and moving it into the sidebar. We also removed the initial textbook selection screen seen after login, because users had misinterpreted what its purpose was, believing that they were reading sentences from the textbook, rather than using it to populate the vocabulary list. Instead, users use the vocabulary manipulation interface in the sidebar to accomplish this task of selecting vocabulary from chapters in books to display in sentences.

This is what the user sees upon logging in. The sidebars are initially hidden, and the user is given instructions to select a language from the drop-down menu.

After selecting a language, the sidebar appears. Here, the vocabulary that the user wants to be displayed in sentences can be selected from an accordian menu with checkboxes. This shows the selection of textbooks that appears if the user expands the "Vocabulary from Textbooks" accordian menu. Note that in our first iteration, this functionality was split across 2 interfaces - first, the textbox selection which appeared only upon 1st login (which imported all vocabulary from that textbook, but could not be accessed again), and also separately in the "My Vocabulary" tab. Now, because in this second iteration we have a single interface for importing vocabulary, which is always present in the sidebar, it is more discoverable because it is always visible and the user uses it upon the 1st login.

Upon expanding the entry for a book in the accordian menu, the list of chapters appears. Checkboxes appear next to each chapter for controlling whether the vocabulary items in that chapter should be displayed in sentences. The chapters can themselves be expanded to show individual words, if the user wants more fine-grained control. Because on each level of the hierarchy, the checkbox is available, this interface is more efficient for adding new chapters or new textbooks - one simply checks the checkbox next to the chapter to allow all vocab in that chapter to appear in sentences (whereas in the first iteration interface, one needs to filter the "all vocab" section by textbook and chapter, then use the "select all" option, and move them to the "learning" list to accomplish this task). Because we have only a single list now (as opposed to 3 in the original), this also increases the simplicity of the interface.

The user next clicks on "review all vocab", and this causes the initial list of instructions to be replaced with a sentence. The second iteration interface has a radio box indicating "review all vocab" or "focus on a specific word" - this is to increase visibility of application state, because in the first iteration interface, one would see this only by going to the "My vocab" tab, and observing whether the "studying" list is empty (which is equivalent in this to reviewing all vocab in the 2nd iteration interface), or nonempty (equivalent to "focus on a specific word" in this 2nd interface). Because users found it confusing on what indicating multiple words to be studied would do, then we made it such that "focus on a specific word" can be used for only 1 word at a time, while providing "history" list of previously selected words so they can be referred back to.

Users encountered few issues with the sentence viewer itself during testing, hence it is similar to the one in the first iteration. Like with the first iteration, clicking on "Show English Translation" shows the English translation in the accordian menu.

Clicking on a word shows its definition, and an option to toggle whether it will be displayed, and whether it should be made the study focus. We changed the language from the first iteration's "studying", "learned", and "not yet known" (which users had found confusing, especially that "not yet known" would be used to prevent that word from appearing in sentences) into "displayed", "not displayed", and "focus on studying" (which implies displayed), to more directly express what the effect of the options had on whether the word is displayed in sentences. We also separated out the "focus on studying" into a separate button instead of having all 3 options be in a single drop-down, to reflect that there is 1 focus of study, but there can be many words that can be displayed, hence making a word the focus of study will cause the existing word that is the focus of study (if any) to no longer be the focus of study, but checking/unchecking the displayed option has no effect on the other words.

Making a word the study focus (as shown here, "gamen" was just made the study focus) will change the radio button in the sidebar to "focus on a specific word", add the previous word (if any, in this case there was no word) into the top of the history section, and make the new study focus appear next to the "current word" label.

If the user wants to make an arbitrary word the study focus, in this case "goondu", he types it into the "search" section in the sidebar (he can alternatively expand out "All Vocabulary" and find it, though this is likely more time-consuming). This is more discoverable compared to the initial iteration, as in the second iteration, this is always visible in the sidebar, whereas in the first iteration, one had to first go to the "My Vocab" before one could even see the vocab list or search button at all (which caused some users to have difficulty this task with the first iteration, because they didn't know to go to "My Vocab").

Once the desired word "goondu" is found in the vocab list, "study this word" can be clicked to make it the study focus, moving the existing study focus word "gamen" to the history. Previous items in the history can be clicked to make them the study focus again. This is to support easy undoability in case the user makes an error and makes an undesired word the study focus, as during testing of the first iteration a user had commented that when she wants to see sentences containing only a specific word, she found it annoying that she loses the word she had been studying up to that point.

Sentence contributions can be made by clicking on the "contribute a sentence" item, which pops up a dialog, and can be closed after contributing the sentence. A popup dialog is chosen instead of the tab from the 1st iteration, because it makes it more clear that the user hasn't lost the sentence he's reading at the moment, and it is more clear how it can be undone (just close the popup dialog, as opposed to go to the tab the user had been in before).