User tests

To access our app: http://people.csail.mit.edu/d_chang/smak/

User 1

The user was a 42 year old white male geek who occasionally travels for work and uses an ipad regularly.  

I instructed the user to pick Flourence as we had not implemented the ability to change cities in the splash page.

Usability Problems:

•  The user began by clicking on a particular place on the map to try to pick an event.  Finding that that did not work, he he looked around the map to pick places and then typed their names into the text field.  He said he wanted a search feature for the map where you can find restaurants.  
  • (major) This indicates not so much a flaw as a lack of a feature. An improvement would greatly increase the efficiency of the application.
• At the time of the evaluation, changing the start and end time caused the schedule to disappear. 
  • (catastrophic) It rendered the application unusable until the user pressed undo. This was fixed by demo time.
•   He was frustrated by the fact that when he typed the name of a location, it did not search for the location and place the pin at that place.  
  • (major) This is easy to overcome but decreases efficiency
• He automatically plans for walking time when the number of events in the schedule is small. However, when it is large, he left an unrealistically short amount of time between activities.  
  • (minor) This can be overcome as the user will learn not to do it, but it will make him late to things the first few days.
• There is no way to save the schedule. Thus, the user has to leave the application open the whole time. 
  • (catasrophic) Realistically, the user is going to be closing and opening this application.
• Clicking on the pin of a scheduled event, the nature of the info displayed is non-obvious and less than useful. 
  • (minor) This doesn't impede usability except by confusing the user why it is there momentarily.
• It took him a while to discover that you can lock something. The lock icon should be within something that looks like a button.  
  • (minor) This is discoverable, just after a while. 
• It is hard to lock things because clicking the lock is hard. It should be bigger.  
  • (major) This is a major flaw in that it causes the user frustration in using the feature.
• Redo does not completely redo operations, as it ignores locking. 
  • (minor) The user can still manually redo all operations.
• Selecting an event should pan such that the event is visible on the map. 
  • (minor) This just requires the user to spend more time panning around and remembering.

Usability Successes:

He finds the dragging of events from the todo list to the schedule very intuitive, as well as resizing events and moving them around to change event times.  He also finds it intuitive to be able to drag the map pin to the location of the event in order to place it.

Notes: 

He did not use autoschedule.

He expressed a desire for a button that just says "pick stuff for me to do" that picks the top-10 reccomended activities in the city. I am skeptical.

User 2

User 2 was a grad student from the media lab.  We chose this user because the user has never been to Florence but has some experience planning trips.

Usability problems:

• User tried tapping instead of dragging the resize handles.  (minor)
  • Possible solution: better grip icons
• Earliest start and latest end times were unintuitive and unused by the user.  When the grey bars appeared, the user was unsure of their meaning. (minor)
  • Possible solution: remove feature altogether. 
• On multiple occasions, the user tried to add a new activity while an activity was selected. Because the uses the same form to add a new activity as to edit an activity, a new activity could not be added until the activity was deselected. (major)
  • Possible solution:  Do not reuse the form to edit activities. Make the form inside the activity or as a popup box. 
• User tried pressing the “Schedule these activities for me” when there were no activities in the unscheduled activities column.  It is unclear to users that auto-schedule does not help optimize the current schedule, but instead optimally moves unscheduled items into the schedule.  (major)
  • Possible solution: Make more descriptive  label for button

Usability successes:

• User was able to quickly add all the activities he wanted to do that day
• User understood and used the dialogue for change the start and end time of the day
• User quickly discovered how to move location markers on the map since the marker said "Drag Me!"
• The user was able to select an activity by clicking on the corresponding location marker on the map. 
• The user immediately understood that the activity boxes were meant to be dragged.

User 3

User 3 was a college graduate (early/mid twenties age range) who had just finished applying to law school.  This user was chosen because his personality was meticulous and thought-out but had very little experience planning trips.

Usability problems:

• User had a different iPad with a different resolution: the "end of day" pull down was not on the screen and there was no indication of where it was supposed to be. (minor)
  • Possible Solution: set default day to be sufficiently small so that the entire (default) schedule fits on one screen with high guarantee (easy fix).  OR adjust based on dynamic iPad resolution (more difficult).
• As described in the list of tasks, finding the specific locations on the map was difficult.  It was confusing to have to constantly zoom in and out of the map to pinpoint locations. (major)
  • Possible Solution: allow the users to input street addresses to locate places, as opposed to pinning the location down on the map itself.
• User complained that activities disappeared all of a sudden (the user had accidentally clicked the X button for that activity).  Fortunately, there is an undo button but the user did not see it (the facilitator had to give a hint).  (major)
  • Possible solution: ask for confirmation when an activity is to be moved from the List of Activities column to the trash.
• There was no way to specify "end of day" when setting the latest start time---one can only specify exact times. (minor)
  • Possible solution: add both "end of day" and "beginning of day" to the start time/end time pull-downs.
• There is no way to make an activity with no location.  User wanted to have lunch anywhere around 12p-2p (minor)
  • Possible solution: if the user specifies no location for an activity, that activity will still take up space in the schedule column but not appear on the map for distance routing purposes
• User did not use the lock button for the opera. (minor)
  • Possible solution: add an element in the "Add Activity" flow for specifying specific starting times.
• User did not use the auto-schedule button. (minor)
  • Possible solution: Couple the auto-schedule button more tightly with the schedule column (see below for details).

Usability successes:

• What does the drag me button do?  While this caused confusion for a few seconds, the user quickly realized that the currently selected activity corresponded to drag me.
• The user correctly discovered how to drag and resize activities.

The biggest issue with this user was that he did not use the lock or auto-schedule buttons.  In the debriefing he said that their purpose wasn't clear when he could just manually create the schedule by moving activities around (i.e. what does it mean to "Lock" something if you don't know about the auto-schedule button?).  As a broad solution to this, the user suggested eliminating the auto-schedule button, and letting users fill up the schedule as they please.  Then, the user can say "finished" at which point the app pops up a dialogue that says "thank you for scheduling.  would you like SMaK to try to optimize your schedule for you?"  This flow would make the auto-schedule button take more of a center stage.

Report

Design

Structure and flow

The final design is three-part/column and has the following structure:

1. Left column (Add activity): user can add an activity, and change properties related to that activity such as its location, name, earliest start time and latest end time (e.g., museum closing times).  When an activity is added, it appears in the list of activities (center) column.
2. Center column (List of Activities): A list of all activities that the user wants to do, but have not yet been scheduled.  This is conceptually a TODO list: activities in this column have a duration (which can be changed by the user), but haven't been given a specific start time.  Activities move between this and the right column as the schedule changes.
3. Right column (Schedule): The concrete schedule of activities.  This column has an hour-by-hour calendar feel; activities placed on it have a duration and start time and abide by the following constraints. First, no activity may overlap with another activity (you can't be in two places at the same time).  Second, no activity can start before its earliest start time (specified in the left column) or after its latest end time (you can't visit a museum after it has closed).  Each activity has a "locked?" button which indicates whether the start time is flexible (this will be explained below).

The final design has the following flow.  The user adds a set of activities in the left column.  Then, the user constructs a final schedule through an iterative process of moving activities between the list of activities and schedule column.  When the user moves an activity from the list of activities to the schedule, some activity from the schedule may be displaced and moved back to the list of activities.  The application provides a "schedule these activities for me" button, which tries to schedule all of the remaining activities in the list of activities in one click.  When the user is satisfied with the state of the schedule column, the scenario is complete.  

Here is a screenshot of the main page in the application.  Both shots are the same screen. The right-hand side is annotated.

Platform

SMak is meant to be used on the go, when the user is traveling.  The UI is currently a web browser application which has been ported to an iPad for portability reasons.

Design evolution

Our original storyboard design (GR2) was a mobile app consisting of a series of forms that allowed users to specify “constraints” such as location, duration and ordering of events. 

During our paper prototype (GR3), we quickly learned that the constraints list was very confusing and the navigating through the forms was too complicated.  In addition, users wanted to interact with the schedule once it was created. 

For our computer prototype (GR4), we decided to abandon the mobile platform due to problems with drag and drop, and instead created a web app that shows all the different views from the mobile app on one screen.  In addition, we decided to display and edit all constraints visually, such as allowing users to resize activities to change its duration or dragging activities around in the schedule to rearrange them.  We used an edit distance algorithm to move conflicting activities around if the user drops an activity into an occupied time lot.  This allowed users to rearrange their day without manually shifting each activity. 

For our final implementation (GR5), we realized we needed to make the web app work on an iPad to make it portable for traveler, so we figured out how to make drag and drop work on the iPad browser.  This new form factor meant the resize handles were very difficult to grab (since our fingers are a lot fatter than the mouse), so we created much larger round handles outside of the activity box to allow users to easily resize the activity.  We also added many features suggested in the heuristic evaluations including an auto-completing form to allow users to either pick from a list of activities or create their own activities, undo, locking activities in the schedule, optimized auto-scheduling using traveling salesman, earliest start time and latest end time, and draggable map pins.

Implementation

View

The interface was implemented using Jquery UI which included libraries to create draggable and resizable boxes, as well as the autocomplete input box in the form.  

When the page first loads, it renders the calendar grid based on the length of the schedule and the starting time.  Each time the start or end times of the day changes, the entire calendar grid and all items in the schedule get redrawn.  Each time an activity is added to the “List of Activities” either from the form or from being unscheduled, a new activity box is drawn in the “List of Activities” column.  This means that the “List of Activities” column is modified with each change, but not completely redrawn.  The one time the entire “List of Activities” is redrawn is when the undo and redo buttons are used.  On the other hand, whenever any part of the schedule column changes, the entire schedule is redrawn because often times, more than one element in the schedule is moving and it is hard to keep track of what changed.

In addition to rendering the boxes for each activity, the view also dynamically updates the activity boxes as they are moved and resized so that they show error messages, map labels and the current duration and times.  A single activity can be selected, at which point, it greys out parts of the schedule in which the selected activity cannot be done between.

Form: The user first selects events using a form and map. The form has little more than a textbox for the name of the event and two dropdown menus to pick the start and end times of the event. The textbox suggests activities within each area from the google places API.  
Map: The map indicates both the location of an individual activity and the user's itinerary.  When an event is selected (or first created) a pin is placed on the map saying "Drag Me", which the user drags to indicate the location of the activity. When a user deselects an event, the pin becomes a normal google maps blue pin. A User can click on a pin to select it When the event is scheduled it becomes part of the user's itinerary. The walking path from event to event given by the google maps API is displayed.

Model

The fundamental object in our application is an activity: something that can be done at a particular location in between two particular times. There was simply an array of these, each with a unique id. This activity could be in one of four states: 

- Suggested, as one of the events that the google palces API returned or one of our created events. These were not visible except in autocomplete.
- Todo, as an event in the middle column that the user was interested in visiting but not certain.
- Scheduled, as an event placed on the schedule and itinerary.
- Locked, which was also placed on the schedule, but unable to be displaced without being unlocked.
Three things could be done to these events:
- They could change state, and there were a number of functions provided by the model to do this while preserving invariants. Of course, javascript is still a dynamic language and so no real invariants could be enforced.
- The event could be selected or deselected. Only one event at a time could be selected.
- Other properties such as start time, end time, opening, closing, location, or name could be changed.
Views each register handlers to such changes in an activity. Thus, when an event's time changed in the schedule, the schedule updates the model, which files all handlers for that change, which fires the handlers in the view, form, and map. The function call to propegate the change can take a parameter naming a view whose handler should not be called. This allows us to prevent the schedule from twice dealing with the movement of an event.
Model: The fundamental object in our application is an activity: something that can be done at a particular location in between two particular times. There was simply an array of these, each with a unique id. This activity could be in one of four states: 

- Suggested, as one of the events that the google palces API returned or one of our created events. These were not visible except in autocomplete.

- Todo, as an event in the middle column that the user was interested in visiting but not certain.

- Scheduled, as an event placed on the schedule and itinerary.

- Locked, which was also placed on the schedule, but unable to be displaced without being unlocked.

Three things could be done to these events:

- They could change state, and there were a number of functions provided by the model to do this while preserving invariants. Of course, javascript is still a dynamic language and so no real invariants could be enforced.

- The event could be selected or deselected. Only one event at a time could be selected.

- Other properties such as start time, end time, opening, closing, location, or name could be changed.

Views each register handlers to such changes in an activity. Thus, when an event's time changed in the schedule, the schedule updates the model, which files all handlers for that change, which fires the handlers in the view, form, and map. The function call to propegate the change can take a parameter naming a view whose handler should not be called. This allows us to prevent the schedule from twice dealing with the movement of an event.

Scheduling algorithms (Controller)

When an activity moves from the Activities column to the Scheduling column, an important decision to make is how to "make room" for the new activity in the Scheduling column.  We implemented separate algorithms for the following operations:

1. Moving an activity onto/within the schedule, or resizing an activity already on the schedule.
2. Changing the schedule's start or end times.
3. Clicking the "Schedule these activities for me" button (a.k.a. auto-scheduling).

Broadly speaking, the algorithms rely on the principle of least destruction: when adding to/resizing/etc the schedule, don't move activities already in place unless they have to be moved.  We chose this heuristic for usability reasons: if the user is planning a schedule and moves some activity A, he/she will not want the rest of the schedule to change in unnecessary ways. So, if an activity is moved into an empty space, no other activities should have to move.  If one activity A is moved on top of another activity B, activity B should "scoot" or somehow else move to make room for A.  If changing the schedule makes placing some activities impossible (due to time constraints, etc), the activities are moved to the Activities column.  The algorithms try out different possible schedule configurations based on these principles and choose the best configuration based on a weight function.

The auto-scheduling algorithm, in addition to the above heuristics, tries to minimize the average euclidean distance between each activity.  This was also done to generate better schedules: given a set of activities, the user wants to complete as many as possible in a fixed amount of time.  The algorithm internals are traveling-salesman-like: we have a set of unscheduled items which we add to the schedule in different ways (while observing the "principle of least destruction").  Then, we choose the best complete configuration based on the weight function we described above plus the distance metric.  Despite our small problem scale, the brute force approach did not work time-wise; so our algorithm is greedy, with intelligence to avoid getting stuck in bad schedules.

Evaluation

We chose users who had a desire to travel to foreign places and had different travel.  For example, young people who travel through Europe do so "by the heals of their pants," jumping from hostel to hostel.  Older travelers, however, probably want a more set-in-stone schedule.  Older travelers are probably more experienced in traveling, as well.

Procedure

We read the briefing (below) to each user.  We then gave each task on an index card to each user.  The next card was given when the user completed the task on the current card.  No demo was given to any user.  After the user test completed, we debriefed each user and let them know how to overcome problems they were having (if any) with the interface.  We have listed usability problems during the user tests (and potential solutions) at the top of the page under the User tests section).

Briefing

SMak helps you plan your schedule.  When people travel to foreign countries, they typically know what you want to see but not how to go about doing it.  With SMak, your job is to specify what your activities are and give hints as to how you want to carry out those activities.  Smak's job is to help fill in the details of your schedule: how to get from place to place, in what order to do your activities---these sorts of things.

Tasks

1. You want to visit Florence.
2. You want to wake up at 8am and go to bed at 10pm.
3. You want to do the following activities throughout the day: eat breakfast/lunch/dinner, visit the dome, and go to the opera.
4. Now, you want to visit tourist places in Florence *and want to do as little walking as possible.*  You want to visit the dome (at the piazza del duomo) at some point in the day for two hours; it doesn't matter when you go.  The dome opens at 10am and closes at 5pm.  You also want to go to the Medici chapel for a special opera showing that starts at exactly 3pm and lasts for 2 hours.

Reflection