GR6 User Testing - WebAnnotator

Back To Project Home

Design

Login Page

We designed our log-in page to be as simple as possible. It passes through all three evaluations without major changes.

Figure 1. The empty log in page

Figure 2. The filled log in page

Editing Page

1. Annotation Tool Bar. 

In our final design, the tool bar only shows up when the user selects certain text. We considered two alternatives. In the first alternative, we would always have the tool bar docked at the top of the page. We decided not to implement this based on many users' feedback. They seemed to dislike the idea of having go back and forth between a top tool bar and the text in order to accomplish an annotation. In the second alternative, the tool bar would always appear when the mouse hovers over some elements that could be annotated, such as a picture. We ruled this out because we decided that the tool bar would appear too frequently.

Figure 3. The annotation tool bar appears when the user selects some text (in this case "Originally specified in 1958").

2. Notes

After a text is selected and the "note" button is pressed, a light yellow box would appear after the selected text in a new line. Thereafter, the selected text is always dotted-underlined and the note always stays there. We put the note into a separate yellow box because we want the note to be visible but not disrupting the normal view of the page. We dotted-underline the selected text because we would like the text to be associated with the note: every time the mouse hovers over the underlined text, the yellow box will be highlighted. The other alternatives either hide the notes by making an extra pointer next to the text or display the notes at the cost of disrupting the content of the page.

Figure 4. The note in the yellow box in associated with the dotted-underlined text3. Deleting Highlights and Underlines

3. Deleting Highlight/Underline

If the user wants to delete a highlight or an underline, he can just click on the affected area. A red "close" button will appear next to the area. Once the user clicks on the button, the highlight or the underline will disappear. We consider the alternative of using an "eraser" button in the tool bar to carry out the undo function. We ruled it out because 1. eraser does not afford the idea of completely undo and 2. the current design allows the user to pinpoint the affected area much more easily.

4. Deleting A Note

As a direct and convenient consequence of associating a note with a dotted-underlined text, all the user need to do is clicking on the underlined text. A red close button will appear and we are back to the previous case.

5. Share Dialogue

In the share dialogue, the user can change both the name and the accessibility of the file. Moreover, if the viewer wants to revoke someone's accessibility, he can select "remove" from the drop-down menu next to that person and click "Done." Next time the user opens the share dialogue again, that person will not be on the list again. We stick with this design because it provide safety: a user will not accidentally remove another user and cannot revert.

6. Dashboard
Our final design lists all the pages and supports a search by prefix feature. An alternative design we considered was presenting all pages as thumbnails. A user raised a good concern that this might render the dashboard hard to navigate if the number of pages is large.

Implementation

Front-end

Technology

The front-end UI is implemented in javascript and html. It is developed and tested primarily on the Chrome browser. Firefox should also work, though not extensively tested. Some new CSS 3 features are adopted. Browsers that don’t support CSS 3 features such as box-shadow might not display the web application correctly.

On the Javascript side, jquery, jquery-ui and a third-party Javascript library called rangy are used.

Logic

The annotations are modifications to the HTML code made by javascript on the client side, so that there is no delay between pressing of an annotation button (such as “Highlight”) and seeing the effects (text selection highlighted with a bright yellow background). An update to the server is sent immediately after, hence the “auto-save” feature.

Back-end

Technology

The back-end server is implemented in Java. HTTP requests sent from the front-end javascript are served by Java servlets. We are using Apache Tomcat latest version (7.0) to host the server.

Logic

HTTP requests sent from client to server:

• /newpage?url=...: Sent when a new page is created from a user-provided URL ready for annotation. Server returns a URL to the new page to be annotated.
• /saveAnnotatePage: Sent when an annotated page needs to be saved. The message body includes the entire modified HTML of the page being annotated.

When a “newpage” request is received by the servlet, it first downloads the page using wget, then a number of modifications are made on the page, including disabling links on the original page and adding our annotation button HTML code and Javascript files.

The saved pages are stored as files in the file system on the server, i.e. there is no formal database at the back-end. Server keeps track of the paths of the saved pages and serves them to the client when asked.

Design Decisions that Affect Usability

Server uses “wget” to download the webpage to be annotated, which has limitations. Some webpages dynamically generated by Javascript seems not to be downloaded correctly, and there are other pages whose format and styling are not preserved well.
The main effect on usability is that the user seems to be limited on the range of webpages that they can annotate. Not loading pages correctly frustrates the user.

Evaluation

User Recruitment and Characteristics

We personally approached our recruited users and asked about their interest to a test user for WebAnnotator. All three of them agreed immediately so we did not have to look for a 4th candidate.

The first user is a CS researcher at MIT. The other two are undergraduates of different age and do not major in CS. We believe that they well represent our user population (recall: students and researchers). Moreover, Our users are quite diverse as their academic backgrounds are different.

We also asked our users about their previous experience with annotating document. All of them have read annotated document before but have rarely annotated themselves. The advantage of this is that we could better evaluate the learnability of our interface because of their limited experience.

Nonetheless, our users may lack the diversity in age as they are all under 25.

Procedure

Our users were asked to perform the following tasks. A short briefing of what WebAnnotator achieves was given at the beginning. However, no information about the interface was mentioned. We also encouraged them to speak up about their thought process. No demo was performed.

We helped our users at time when they were stuck. Fortunately, this did not happen frequently.

Tasks

1. Annotate a page on your favorite topic. In WebAnnotator:
   1. Open the page for annotation.
   2. Highlight, underline and add notes to pieces of text in the page.
   3. Delete an annotation you have just made.
2. Suppose you have annotated other webpages before using WebAnnotator.
   1. Find all your saved annotated pages. (Note: about 30 saved pages were created beforehand in order to test the search box)
   2. Reopen the one you just annotated.
   3. Open the one named Project 1: An ABC Music Player.
3. Share your page with your best friend.
   1. Suppose the page is already shared with some people:
      1. Remove one of them
      2. Change one user's access rights from "can edit" to view-only.
      3. Add your best friend in.

Observation

User 1 (CS Researcher)

• Task 1
  • After highlighting, did not notice the popup annotation buttons at first. Was looking for a tool bar at the top and searching throughout the webpage.
  • Adding multiple annotations to the same piece of text seemed difficult.
  • Was looking for a "save" button before noticing the message "All changes saved" at the top.
  • The task was successfully completed otherwise.

• Task 2
  • Did not realize that the search box implements only prefix search. Entered keywords like "abc" and "player".
  • Had to locate the entry for "Project 1: An ABC Music Player" by manually searching through all the entries.
  • Had some difficulty to go back to the dashboard from an annotated page. Did not remember that the button "Saved Pages" would bring her back to the dashboard. Recommended changing the wording to "All pages".
  • The task was successfully completed otherwise.

• Task 3
  • In the share dialog box, the "plus" button was not pressed before clicking "done".
  • The task was successfully completed otherwise.

User 2 (Math Undergraduate)

• Task 1
  • Entered the URL for the Wikipedia main page. Tried to use the search box there to find a desired page before finding that the search box was disabled.
  • Thought that highlighting a piece of already highlighted text would un-highlight it. Took quite a while to realize the delete button would show up when clicking on it.
  • Tried to delete a note by highlighting the text in the note box.
  • The task was successfully completed otherwise.

• Task 2
  • Mistook the search box in the dashboard page with the functionality to "add a new page".
  • Did not realize that the search box implements only prefix search. Entered keywords like "abc" and "player".
  • Then tried to enter a URL in the search box to get to the page created in the last task.
  • Finally, had to locate the entry for "Project 1: An ABC Music Player" by manually searching through all the entries.
  • The task was successfully completed otherwise.

• Task 3
  • The task was completed successfully and quickly.

User 3 (Mechanical Engineering Undergraduate)

• Task 1
  • Entered the URL for the Wikipedia main page. Tried to use the search box there to find a desired page before finding that the search box was disabled.
  • Then tried to use the URL box as the google search box by entering a keyword. At this point the facilitator told her that a URL was needed.
  • Tried to click the shortcut link to jump to some section in the Wikipedia article. Realized that hyperlinks are disabled.
  • Didn't know how to add notes. Was looking for the button without success.
  • Recommended that clicking (in addition to highlighting) should also make the annotation toolbar appear.
  • The task was successfully completed otherwise.

• Task 2
  • First used the search box thinking that it implements substring search. Then realized it actually implements prefix search. Found page successfully.
  • The task was successfully completed otherwise.

• Task 3
  • In the share dialog box, the "plus" button was not pressed before clicking "done".
  • The task was successfully completed otherwise.

Interestingly, all three of our users chose a Wikipedia page even though nothing about Wikipedia was mentioned in any verbal or written description throughout the testing. This may indicate that Wikipedia articles are among the most popular webpages to be annotated and more effort should be geared to make our interface more usable for Wikipedia articles.

Usability Problems and Solutions

Reflections

One important observation revealed from this final user testing is that users generally behave quite differently from what we expected for the annotation toolbar. What we thought to be “natural” turned out to be not so intuitive and it would take them quite some time to figure out, for instance, how to add a note.

Perhaps we should have tested the annotation toolbar more extensively in the paper prototype iteration. But we had not done so, mostly because the cost to simulate this was quite high - one needed to ask the user to simulate cursor movement and dynamically display the annotation toolbar. Simulating annotations such as underline and adding notes seemed quite difficult to do on paper prototype.

Nevertheless, given the nature of our application, this should be the most important feature to test and the higher cost to achieve this could probably be justified. We are, however, glad that our limited test of the annotation toolbar in the paper prototype did reveal important insights into its usability, though we could have discovered even more.

Perhaps one thing we should have done is to add another prototype iteration and testing between GR4 and GR5. The canned version of WebAnnotator should be of sufficiently high fidelity to allow
users to supply informative observations and insights during testing. This way we could have tackled the usability issues before rolling out our “final” version for GR5.

In short, we have learned that our decision on what to test in an iteration should be based on its cost as well as its benefit. In our experience, the benefit was subconsciously underestimated because of the relative high cost.