User Analysis
In gathering preliminary data for our project, we interviewed three subjects who have backgrounds in technology. We believe these subjects are reasonable since their demographic is likely to consider using a mobile device to look products up while shopping. Basically anybody with a smart phone capable of making informed consumer decisions will be apart of the user base.
For each subject, we presented three related products that they might go to a store to purchase. To A, we presented laptops. To B, we presented couches. To C, we presented clothing articles (jackets). We narrated a hypothetical situation where they were out shopping for a product of the type presented, and that they've narrowed their choices down to these three choices. We then asked about how they would go about determining which they would purchase:
Subject A, male, stated that they would purchase the smallest laptop, but because it is a larger purchase, they would have previously looked it up online. However, they stated that they often type in Amazon queries at the grocery store for certain expensive products (e.g. razors). When asked if they'd ever used the picture-based lookup applications from Amazon or Google, they stated that those have not worked well in the past, and that it was more reliable for them to type a text-based query despite its slow speed.
Subject B, female, stated that because furniture is also a relatively large purchase, she would have previously gotten an idea of what price ranges to expect by Googling products from home. When told that one of the three selected products was not initially found in their online searches, she said she would talk to the salespeople and later go home, look it up and come back. When asked if she ever looked up pricing and reviews from her phone while at the store, she said that she would consider doing it if she was alone, but not if she had already engaged a salesperson as she felt doing so would be rude.
Subject C, male, stated that to determine which coat to purchase, they would first want to try it on and then subsequently look up reviews for determining how durable it is. Their chosen method for doing so was to "awkwardly type in the product name into Amazon."
To summarize, A and C were familiar with image-based product lookup services, but were more inclined to "awkwardly type a query" than to take a picture, partly due to reliability reasons. B had not previously used mobile product lookup services in the store at all, reasoning that doing so seemed rude. On the other hand, she was perfectly willing to go home, look it up, and come back later.
These results suggest that looking up products on mobile devices is useful, but that it is currently slow and awkward. We believe that for certain types of products, image-based product lookups can be made more appealing and efficient in three ways: 1) by providing the user with an interface in which it is easy to specify what product is interesting (thereby helping computer vision algorithms), 2) by allowing the user to specify multiple products of interest (both helping computer vision algorithms, and providing interfaces for efficient comparison shopping), and 3) by providing search results that include price and links to reviews.
Task Analysis
Expediting picture-based product lookups can be broken down into three tasks:
1) How to specify an object of interest: Current systems either have no method for specifying the object (instead, they assume it is centered), or they have (obtuse) interfaces for drawing a rectangular bounding box. Due to the sensitivity of state-of-the-art computer vision algorithms, poorly placed localizations lead to significantly incorrect identification results. Both cause losses in efficiency which compel users to use typed queries, or to not use their mobile device at all. Indeed, we need more efficient methods for specifying objects of interest in a photo.
2) How to specify multiple objects of interest: Current systems do not support looking up multiple items in a single image, causing the user to have to repeatedly do the same task for multiple products. What is a visible, learnable and efficient interface for allowing users to specify multiple products in the same image?
3) How to convey results to the user: Current systems (e.g. Amazon Remembers, Google Goggles) will open a product page or search engine results page. Neither method would scale to multiple lookups. Even for individual lookups, it isnot clear that these are the best methods for conveying what a user is likely to be interested in for both prices and reviews. What is the best way to convey results to the user?