Keyboard - GR6 User Testing

1. Design

Throughout the course of this project, we progressed through several very different prototypes. Each prototype presented challenging issues during both the design and test phases. We describe features of our final design as solutions to problems encountered in earlier prototypes to illustrate the motivation behind each design decision.

In our first prototype, we presented the users with left and right arrows to choose “A-M” or “N-Z” (Fig. 1).

Figure 1. Navigating in an early prototype

Paper prototyping suggested that after learning how to use our keyboard, users would quickly be able to decide whether to initially go left or right. However, as we implemented this and tried using it ourselves, we realized that many users might never become efficient at this. Heuristic evaluations and user testing confirmed our suspicion. We improved this by displaying all characters up front so that rather than forcing the user to figure out which half of the alphabet contains his target letter, he can simply locate his letter visually (Fig 2.)

Figure 2. Navigating in the final design

After navigating left or right, our early prototype opened up a new set of choices that required the user to navigate left or right again (Fig 3.)

Figure 3. Submenu navigation in the early prototype

This proved to be extremely difficult for users to understand in all stages of evaluation. Users wanted to press down to select the newly opened choices, but they in fact needed to press left or right (down changed the mode). We built on our solution to the first problem by providing color cues and moving “arrow” buttons to teach the user how to navigate (Fig. 4)

Figure 4. Submenu navigation in the final design

Another source of great confusion in the early prototype was mode changing. Pressing down at any point would cycle between uppercase, lowercase, and digits/symbols. Surprisingly, up would not cycle the other way; instead, up would move focus up the tree of symbols or into the textbox itself. None of these controls were natural and were a source of frequent errors in all evaluations.

In our final design, we eliminated modes altogether and simply laid out each group of symbols in its own row (Fig. 5). We mapped the up and down arrow keys to move between the rows. We also eliminated the ability to focus on the textbox as it was rarely used and highly complex. The new, significantly simpler design was immediately understood by users.

Figure 5. Choosing from different sets of symbols

Since we removed the ability to focus the textbox, complex edits (such as moving the cursor back three characters and changing a letter) were no longer possible. Instead, we choose to include a backspace symbol as the sole mechanism to perform edits. We justified this decision with our observations during user testing - users almost never wanted to perform complex edits unless we explicitly instructed them to do so. To make backspace efficient to access, we placed it at the end of the bottom row (Fig. 6). The user need only press the right arrow once on the bottom row to delete characters. This feature was liked by many users.

Figure 6. Placement of the backspace character for efficiency

Navigating to the autocomplete box in our early prototype also proved difficult. Users could not figure out how to get to the autocomplete box to select an item from it, nor could they figure out how to get back to the text entry once in the autocomplete box.

Our final design aimed to simplify the controls and unify the appearance of the autocomplete box with the rest of the widget. We place the box directly below the keyboard with each suggestion in its own row (Fig. 7). Up and now now move between the rows of symbols and autocomplete suggestions.

Figure 7. Placement of autocomplete box

2. Implementation

On the software side, our implementation is entirely in HTML and javascript. We felt that the web platform provided the easiest tools with which we could rapidly prototype various ideas we had. Aside from the autocomplete box, the implementation is mostly static with a little bit a javascript to respond to key presses by the user. When the user presses up or down, we change the CSS attributes of the rows to reflect a change in focus (or if the user has already drilled down into one row, up or down undo the user’s previous action). Similarly, left or right adjust the CSS attributes of elements within the focused row to visually indicate what the effect of his next action will be.

On the hardware side, we took a Wii controller and connected it to a laptop via Bluetooth (Fig. 8). This allowed us to test our design in our target scenario: text entry with a limited input device. It also allowed us to do a direct comparison against the Wii’s keyboard, the widget we are trying to supplant.

Figure 8. FInal implementation with a Wii  controller

3. Evaluation

We asked six of our friends who are members of our target population to test our keyboard. They are all college students; five of the six are avid gamers and were already familiar with the Wii and its controller. Nevertheless, we included a warm-up exercise for all users.

We decided not to demo our keyboard to our test participants to more realistically simulate how a real user would encounter our keyboard. We observed how easily our keyboard controls were learned during the warm-up exercises.

We fixed a set of two phrases for the warm-up round and two phrases for the actual test. We tested both the built-in Wii keyboard as well as our keyboard (we randomized the order in which we tested the keyboards for each participant). We asked each participant to type the two warm-up phrases followed by two test phrases (they were informed that the first two phrases were warm-up). Finally, we asked each user to make a specific edit to the last phrase entered (the same edit for all users). We collected timing information, made observations during testing, as well as subjective feedback from the participants.

The results of the timing data is summarized in Figure 9. We see that typing these phrases with our keyboard takes roughly twice as long as it does with the stock keyboard. Additionally, the average time to edit “Call me maybe” to “Call me baby” with the control keyboard was 5.5s; the average time to edit with the experimental keyboard was 12.8s. Our observations during user testing suggest that the reason for this is that our participants were heavy gamers so they were very familiar with the control keyboard. Since they did not have as much experience with the experimental keyboard, they lacked muscle memory in navigating to letters. In the future, we would like the resources to do a more fair evaluation in which test participants have not had any experience with game controllers.

Figure 9. Average time to type each phrase for the Wii keyboard (control) and our keyboard (experimental).

After completing the assigned tasks, we asked each user which keyboard they preferred. Four of the six participants said they preferred the experimental keyboard - they believed that with a little more use they would become efficient enough with it that it would be faster than using the control keyboard.

Our final design presented very few usability problems for our users. From subjective feedback and our own observations, we found only one suggestion for improvement:

Our project is unique from many other projects in that it is highly amenable to keystroke level analysis. We are able to perform a highly quantitative analysis comparing the performance of our keyboard versus a standard keyboard. If no mistakes are made, our keyboard requires 32 keypresses to enter "BenB12" whereas it takes 39 keypresses with a standard keyboard. With practice, we are confident that our keyboard can be much faster to use than existing keyboards.

It is interesting to note that given this keystroke-level analysis, we can craft phrases in which our keyboard requires a fewer or greater number keypresses than a standard keyboard. We notice that standard keyboards are good for entering repeated letters and co-located letters, whereas our keyboard offers a roughly equal cost of 5 keypresses for all characters. In the future, given the resource to carry out a more in-depth evaluation we would like to if our keyboard can offer better performance for certain types of input.

4. Reflection

In all stages of testing (paper prototyping, computer prototyping, etc), user feedback was the most valuable piece of information we received. It was the primary guiding force in directing how our prototype evolved over the course of the project.

We quickly learned that what made sense to us may not necessarily make sense to our users. We thought the controls for our early prototype were natural, but gathering user feedback during the paper prototyping stage revealed that many of the controls were in fact un-intuitive. This helped us realize the importance of building cheap prototypes, quickly getting them in front of users, and letting user feedback guide our design decisions. If we were to do this project again, we would probably spend less time trying to decide for ourselves which of our prototypes was best and instead present cheap prototypes of all of them to users and empirically determine which prototypes to proceed with.

User testing also taught us just that we cannot predict how our users will attempt to interact with our product. While designing our interface, we tried to enumerate all the ways in which our users might make mistakes so that we could guard against them. However, we learned that some errors that we thought would be common actually did not occur at all, and other errors which we had not thought of in fact occurred frequently.

We also spent much time deciding which features to include in our early prototypes when a simple user test could have decided for us quickly and easily. For example, we put a lot of time into designing an editing mode in which the user could focus the textbox and navigate between characters already typed to insert or delete characters. However, when users played around with our prototype we observed that no one ever tried to edit the text - we had to tell the users to deliberately edit some text in order to evaluate our design of the edit mode. Had we gathered user feedback earlier, we would have decided to not include an edit mode.

After encountering many problems with our early prototype seen in Fig 1, we considered two alternatives. One is the final design seen in Fig. 2. The other is a variant of our final design that uses a tertiary search instead of a binary search. We thought this would further reduce the number of keystrokes, but were not sure how intuitive the controls would be. Here, we applied what we learned from our early prototype and immediately gathered user feedback and decided not to proceed with tertiary search.

Our evaluation also taught us the relative importance of subjective versus quantitative evaluations. We knew our keyboard required less keystrokes, so in one sense our design was better. Our keyboard was nevertheless slower to use, so in another sense it was worse. We still received much positive feedback about our design, suggesting that with more practice, our keyboard could become a more popular alternative.