GR6 - User Testing (EZ-FIX)

Design

The design of EZ-FIX allows housing managers the ability to quickly process and assign jobs to the house mechanics (mechanics in this context refer to anyone employed by the facility staff for repair related purposes). EZ-FIX was designed to allow users (house managers) to easily filter and view information about repair requests, which are known by our system as jobs.  As seen in the image above, a user is able to filter jobs by clicking on the left-most column, view a listing of jobs in the middle column, and view specific information about a selected job in the right-most column.

During initial user testing of our computer prototype (both the heuristic evaluation and in-class demo) we received feedback that the color choice should be re-examined.  Some people felt that the red of the buttons indicated importance or error.  As it is true that red is usually used to signal error or importance, we felt that the application had a general red look and feel; there is enough red on the page (logo, all buttons, shadow, etc) that the red does not pop out to the user and look as if it requires attention. The red and grey colors were chosen to give an MIT feel to the application.

Grouping

One major design decision that was influenced by the heuristic evaluation is the structure of the middle panel.   Initially, the jobs listed in the middle panel were grouped into three different categories: unassigned, assigned, and completed.  This allowed the user to quickly discern the status of a repair job and find unassigned and assigned jobs quickly and efficiently.  We found this to be an issue during the heuristic evaluation.  It became a problem trying to keep a consistent layout with a varying number of jobs in each section; listing all jobs in a group didn't work well, fixed heights of each group didn't work well, and finally hiding empty groups (zero jobs) didn't work well.  For this reason, we pivoted and built the middle panel to list all jobs and provided labels to allow the user to discern job status.  Every job is listed and has a colored label indicating the status of a job (green = assigned, blue = unassigned, orange = completed).  If the job has been assigned or completed, a picture of the mechanic responsible for the job is associated with the particular job. Or else, a default picture is chosen. This allows for quick scanning of a particular mechanic's jobs.

Updates

There are different events that trigger a job to be moved to the top of the job listing.  Adding an update on a job moves the job to the top, while assigning a job or marking it complete does not.  Typically, an update on a job happens by someone else (the mechanic) and indicates some form of status change (need for materials, etc).  Assigning a job is done solely by the house manager and because he is making the change, it's not important for the job to be moved to the top for more attention.

Consistency in Right Panel

Another design stemming from the heuristic evaluation was the consistency of the right panel.  Initially the design had action buttons (mark complete, assign, submit update) on both left and right aligned within the panel.  These buttons were not all the same style -- two were red while the mark complete button was a default grey and looked out of place.  The purpose of this was originally to not attract as much attention but it seemed to be detrimental as most users noted this as a major area for improvement.  As seen in the screen shot above, the right panel was modified to be more consistent and streamlined.  The mark complete button took on the custom styling, the assignment widget was improved to be more streamlined with more safety (un-assignment), and the submit button for updates was right aligned to maintain consistency.

Size of Location and Contact Info

The location and contact information of a job is smaller in font size than the description.  This was done to offer a squint differentiation and visually differentiate the information.  When a house manager clicks on the job, he is interested in the details for the job.  While it is important that he knows the location, it is not the biggest concern when assigning a mechanic to the job.  Ideally, the manager can quickly scan the job, assign it and move on.  The granular location data (floor, room number, etc) is more important for the mechanic responding to the job.  The house manager can quickly scan the information and determine what is important.

Assignment Section

The assignment and un-assignment of jobs is an integral part of our application for the purpose of safety.  It is important that this feature is very simple, but still gets the job done.  Assignment is easy; choose a mechanic and click on the assign button.  The more interesting case is how to un-assign a job or re-assign it to another worker.  Ultimately, we went with simple options of assign and un-assign.  If a job is not assigned, it can be assigned by choosing a worker and clicking "assign".  If a job is assigned, it can be un-assigned by clicking "un-assign".  We considered a re-assignment option, but this ultimately looked and felt overly complicated.  We wanted the button to do exactly what the text said. An "re-assign" button would have been actually just unassigning. So in the case a user actually needs to re-assign a job, they can simply un-assign and re-assign the job.

Implementation

EZ-FIX is implemented as a web application written in HTML, CSS, and JavaScript.  A choice was made to use well known, trusted frameworks as to mitigate cross-browser conflicts.  For this reason, we decided to use Twitter Bootstrap's CSS and JavaScript framework for major layout and jQuery for JavaScript functionality.

The layout of the UI is structured using Bootstrap's grid layout.  There are three major columns: one for tabs/filtering, one for listing jobs, and one for displaying the detailed information for the selected job.  In addition, at the top of the page, there is a navigation bar that includes a system search field and user options (logout).

The JavaScript code is very cleanly separated into modular components. EZ-FIX mainly consists of models to hold the data, event listeners to capture user actions, and functions to handle user actions.  There are 3 main models in our application: jobs, updates, and people.  Each model is constructed with available information retrieved from Firebase (described below) and contains appropriate mutator methods.

There is technically no "backend" to our application; rather, we're using Firebase to persist all changes.  Firebase provides a JavaScript library that gives an API to create, read, update, and destroy data.  Every addition, deletion and modification to the database triggers and event to be called on each client connected.  More succinctly, it allows us to easily handle creation and modification of data.  All jobs, updates, and people are loaded onto the client through Firebase as soon as the page is loaded.

The clean separation of concerns of our front end code allowed changes and new features to be quickly built.  The typical flow is: a user makes an update to an existing model instance (stars a job, adds a label, creates an update, etc), the client side code calls the appropriate mutator method on the object, the mutator method takes care of persisting the change by updating Firebase, finally the view renders the updated instance.  The simplicity of this flow allows for additional features to be easily implemented by simply adding the corresponding model methods and view changes.

Evaluation

EZ-FIX is designed for housing managers so it is ideal to do user testing on them.  Unfortunately, with end of the semester duties and the closing of Bexley Hall, most of the house managers who we were originally in contact with were busier than normal, and we were only able to do user testing on one house manager.  However, the specific house manager was part of our initial need finding and was familiar with the problem we were trying to solve.  

In addition to the one house manager, we were able to user test on one professional staff and two student desk workers.  The professional staff is close to our target user population because she is responsible for contacting various services (elevator and plumbing repair) for jobs within McCormick Hall and cares about the overview of the facility status.  Finally, student desk workers were chosen as the closest alternative because they are sometimes faced with repair requests and are well informed of the building status.  We noticed that the users had different technical abilities.  Some users were familiar with Google Mail while others were not. While this didn't affect the user's ability to test our system, it did have an effect on the learnability of various features (starring for example) since some of our features are similar to GMail.  

Users were briefed with the problem statement -- that house managers have a difficult time organizing and prioritizing repair requests and currently, the primary method is done by making mental notes.  Users were told that they are acting as the house manager of McCormick Hall and were not given a demo of the application. A demo was not given because we felt that showing users how to perform different tasks would prevent us from finding learnability issues. They were asked to speak aloud and note aspects that they find confusing or visually unappealing so that we could have a better sense of the problems that they were experiencing during the process. 

Users were asked to perform the following set of tasks, one after another.  In certain cases they would hit other areas of the application that were not being explicitly asked for.  For example, some users would create a label for "important" jobs, while others would use the star to signify importance.  The database was dumped and reloaded before each user session as an effort to control user testing.  

1. You just marked a repair job as finished, but you need to get information about that job again. Find the important information about that job.
2. Record the fact that the “Elevator broken” job is important.
3. Record the fact that the “Elevator broken” job is a mechanical job.
4. Assign the “Elevator broken” job to Billy Bob.
5. Record the fact that the “Door lock jammed” job is finished.
6. Find all repair jobs related to McCormick Hall.
7. Record the fact that the “Toilet clogged” job is not finished.
8. Let the mechanic working on the “Toilet clogged” job know that the toilet has not actually been fixed and needs to be fixed immediately.
9. You realize there’s a cockroach infestation in the Maseeh Hall 1st floor restroom. Record that this problem exists and needs to be fixed by Jenks Jenkinson.
10. Find Home Depot’s phone number.
11. Find all jobs that have not been completed but are being worked on by a mechanic.

House Manager

• Did not know to star a job in order to mark it as important
  • Severity: Major - Marking a job as important by using the star is a important and useful feature for house managers.
  • Solution: Better indicate ability to star jobs or some way to indicate that a star means important.
• Was looking down at the keyboard while typing in the search bar and did not realize that jobs were automatically being filtered while he typed.
  • Severity: Minor - This feature simply increases efficiency, but isn't necessary.
  • Solution: Better indicate this automatically filter feature through text.
• Attempted to get back to the All page and view all jobs but was unable to because there was text in the search bar.
  • Severity: Major - Not remembering there is text in the search bar will hide lots of jobs.
  • Solution: Clear the search box when any additional filtering action is taken to forcing filter then search as opposed to search then filter.
• Got confused about the default text in the textfields for creating a new job. He thought that upon clicking in the textfield the text would go away but it didn't. He tried to backspace to delete it, which didn't work, so he gave up and just started typing, which made the default text go away.
  • Severity: Minor - Eventually figured out how to get rid of text and will now know when creating jobs in the future.
  • Solution: Clear the textfield when the user clicks inside it; this would be more consistent with other applications.
• Tried to look up "Jenks Jenkinson" in the main page search bar when he was asked to find contact information for Jenks.
  • Severity: Minor - All the jobs assigned to Jenks appeared in the middle panel, but no contact information appeared. The user soon found the address book and was able to find Jenks' contact information there.
  • Solution: Include the assignee's contact information in the assignment section of the right panel, or make it more obvious that the search bar on the main page only searches through job entries and not contact information.
• Pressed enter in an attempt to submit text in the large textfields (i.e., updates section and description for creating job), which resulted in cursor moving to the next line.
  • Severity: Major - The house manager does not type that much in either of these textfields and would prefer to hit enter for submission for efficiency sake.
  • Solution: Listen to the house manager and implement this feature to submit on hitting enter. Should assume that the update will be short and will not need multiple lines.

Professional Staff

• Did not immediately realize the jobs could be clicked.
  • Severity: Minor - Figured out jobs were clickable then never experience problem again.
  • Solution: Affordance for clicking is already implemented, but trying making it more apparent. Maybe change the right panel's text of "no job selected" to say "no job selected. Select one on the left for more details"
• Could not figure out how to make the job "important", instead made an update noting that the job was important.
  • Severity: Major - Filtering important job, whether it is starred or labeled, is a crucial feature for house managers.
  • Solution: Better indicate ability to star jobs or create a default important label. This is something that we will have to iterate through and determine if star actually means "importance" to them.
• Had a very difficult time marking a job as "mechanical".  Tried to filter and assign a mechanic.
  • Severity: Major - Did not understand labels and how they affect jobs.
  • Solution: Have a way of teaching the users when they first log on to the page what it means to label something.
• A lot of trouble finding all jobs in McCormick, but eventually used search feature.
  • Severity: Minor - Once feature was used, it was quickly learned.
  • Solution: Possibly auto tag location and create tabs for them in the left-most column.
• Attempted to click all to remove "McCormick" text from the search bar -- did not realize text still remained.
  • Severity: Major - Not remembering there is text in the search bar will hide lots of jobs.
  • Solution: Clear the search box when any additional filtering action is taken forcing filter then search opposed to search then filter (similar to the feedback from the house manager).
• Difficulty finding the button to create a new job.
  • Severity: Cosmetic - The text "+ Job" was not clear to the user.
  • Solution: Consider changing the wording, "Create Job" or "New Job".
• Kept using "Submit" on update form for any change made to the job, whether it was an update or not (mark complete/incomplete, assign, un-assign). 
  • Severity: Major - User does not understand the update feature.
  • Solution: Possibly disable the submit button until there is text, additionally disable the assign button until a person is selected. Another option is to reword the button to say "submit update" or something to indicate the distinction between assigning and submitting update.

General actions: Kept on trying to do the labelling and assigning of the jobs in between us giving her tasks to complete. She seemed genuinely interested in seeing how useful the application could be and how it could be applied to McCormick Hall.

Desk-worker 1

• Could not easily discern the meaning of "important", looked to add an important label versus starring it.
  • Severity: Minor - Choose to use the application in a way unexpected to developers but still works.
  • Solution: Have a default important label that is linked to star.
• Attempt to add label clicking the filter for the given label -- click "Mechanical" to add label.
  • Severity: Minor - Quickly realized this filters and looked to find out how to add label.
  • Solution: Have more tooltips / ways of indication of adding labels.
• Confused as to how communication with mechanic was done through the update widget, especially when no previous updates from mechanic listed. "Does starring let him know it's important?  I don't know how to communicate him through this."
  • Severity: Major - Communication with mechanics is critical to this web application.
  • Solution: Show more indication that updates will notify mechanics.
• "This is really cool and addictive.  I want to assign everything to Jenks Jenkinson and Billy Bobby because I'm Michael McIntyre and I don't want to do anything."
  • Severity: Good.
  • Solution: Nothing.

Desk-worker 2

• Cannot find a way to mark the job as "important" -- surprised when found the star.
  • Severity: Cosmetic
  • Solution: Better indicate star's meaning.
• Had difficulty finding a way to add a label to a job.
  • Severity: Major - Adding labels is really important for house managers.
  • Solution: Make label adding more obvious.  Potentially add it to the right panel opposed to the middle panel.
  • User suggested having the "add a label" ability in the left-most column to be with the other labels.
• Could not easily find a way to find all jobs for McCormick -- did not realize search was possible.
  • Severity: Major - Ability to find all jobs by location is imperative.
  • Solution: Make search more obvious.  Automatically create labels for location.
• Creating a new job did not immediately show up in the list because "McCormick" was still in the search bar.
  • Severity: Major - New jobs typically take action (assign, updates, etc) and need to be visible.
  • Solution: Clear search terms when taking action not related to filtering. Automatically go to "all".
• Attempted to search for "Home Depot" in the job search bar.
  • Severity: Minor - Nothing appeared and user later found address book.
  • Solution: If term found in address book suggest, "Did you mean to search the address book?"

Reflection

As a result of this project, our team learned that using an iterative design process, specifically the spiral model, results in a better product and reduces stress and time wasted along the way.

If we had jumped right in and implemented our website before interviewing potential users or doing any user testing, we would have created a high fidelity product that might not have met users’ goals in terms of functionality, and the product probably would not have had a good user interface design. Also, if we were to conduct user testing after the high fidelity implementation, we may have been stubborn and not wanted to make changes that users suggest to us. Alternatively, we would have wanted to make the changes users suggested to us, but that potentially would have required us to completely change the design of our interface, which would take much time and effort.

Using an iterative design process allowed us to build a little; get some feedback; make changes, improve, and build more; get more feedback; and so on. It gave us the opportunity to work with users and make smaller changes along the way instead of needing to make larger changes and redo large parts of the implementation process as might be necessary if using the waterfall model. Using an iterative design process forced us to think about the user at all times, and this resulted in a highly usable interface.

Paper-prototyping was useful to us for several reasons. Users didn’t have much time with the prototype so the best feedback this type of user testing gave us was on learnability, a huge part of usability. Giving users specific tasks to complete allowed us to gain insight into the user’s understanding of the system before getting to know it well (i.e., how they expect the system to behave). Also, we didn’t need to worry about the exact shape or color of particular features; we could focus on size and location of individual features as well as the overall layout of our interface. Additionally, it was exciting to see users successfully complete tasks during the paper-prototype user testing sessions, and this motivated us to actually include those relevant features in our implementation. On the other hand, seeing users struggle with certain tasks motivated us to find good solutions to those problems.

Having other students do heuristic evaluations of our website was also useful because they spent a lot of time with the website and carefully explored all aspects of it, from aesthetics to learnability to user control, providing useful insight.

If we were to do this project again, there are a couple things we could do differently. One, we would have thought more about how to incorporate categories like job type (e.g., plumbing) and dorm name (e.g., McCormick) into our design during the paper-prototype stage. We focused more on getting the general layout of our interface figured out before including this extra feature, and it turns out we didn’t implement filtering by job type or dorm name until much later in the process (during the final implementation), so our only chance to test this filtering feature was during the final user tests. It was a little stressful when we tried to incorporate this filtering feature toward the end of the implementation process. So much of the design had effectively been made concrete at this point and there weren’t many options for how to incorporate the filtering or many locations for where to include the filtering elements (we were a little unsure where to include the dropdown for adding a label to a job). If we had thought about this major feature of our design earlier perhaps we could have better incorporated it into our design and we could have gotten more user feedback. A second thing we would have done differently would be to conduct more user testing along the way of individuals in our user population (i.e., MIT dorm house managers) instead of just students. Unfortunately we didn’t have easy access to house managers, but if we had, it might have been nice to get feedback from them more often.

We really enjoyed this project! Using an iterative design process was very effective, and we feel like we learned a lot.

Going Forward

The focus of building this application was to fit our users core needs; allow them to get done what they need to get done.  In this process, we avoided feature pollution and considered adding new features only as soon as we completely fleshed out the current ones.  Moving forward, there are a few features we would like to incorporate that will help improve safety, efficiency, and learnability.  Adding in these features would bring the application to a very complete, production-ready level.  While the application is really a minimum viable product, there are enhancements we would have liked to add given more time and resources.

One major feature we would like to add is the ability to select multiple jobs and take action on them as a group.  An example of this is selecting 2 or more jobs to be assigned to the same worker, or marked as completed, or given a label.  It might not be too hard to simply add a check box next to each job, then appear a menu when jobs are selected that allows for aggregate actions with the necessary ability to undo a given action.  We sacrificed this feature as we took into consideration all of the UI decisions and implementation time that we simply did not have.

Another important feature would be to allow multiple labels (i.e., job type, building) per job. It's possible that if the elevator is broken the house manager will want to mark it with both Mechanical and Electrical labels. Similar to the job aggregation feature discussed above, this multiple labels feature would have required some UI decisions we didn't have enough time to make.

It would also be nice to allow the house manager to assign multiple workers to a particular job, since the job might require a mechanic and an electrician. This would require some large UI decisions regarding how to show pictures for multiple workers and how to assign and un-assign workers individually.

A smaller enhancement includes integration with the address book into the application.  Bringing contact information into the update and assignment fields would allow managers to easily keep track of which services (Drain Doctor, Elevator Service, etc) have been contacted and check back in with them more efficiently.  

More information could be included in the job creation dialog.  It might be the case that the house manager knows who he would like to assign a job to during creation and he should have the freedom to do so.  This could be used as a source of learnability as well -- it could ask if the job is important with a star next to it.  This would allow a user to learn the fact that star means important opposed to applying their own meaning to the star  Just a note: this would only work if it were done early on, before the manager builds his own meaning for stars, etc.

Another small improvement would be the addition of an "Active" tab.  It's reasonable to assume the house manager is more concerned with "unassigned" and "assigned" jobs than with "completed" jobs.  For this reason, a filter that shows only "unassigned" and "assigned" jobs would be added.  More need finding and user testing would be done to asses the validity of this assumption.  

Finally, it would be nice to alter the update widget to be more inclusive of job activity and perhaps change it to an activity widget.  It's probable the house manager would like to know when he assigned a job, when (and maybe why) he re-assigned a job, etc.  More need finding and user testing would lead us to implementing this to accommodate the house manager's needs but it seems as if this would eventually be implemented given more time.