Final Design
Screenshot |
Explanation |
---|---|
|
Originally, the main screen looked like the extended song list below. Our final design adopts a splash-screen-like design in order to highlight the name of the app and reassure users that they're in the right place, while remaining functional with a prominent "new song" button. The background is intended to be aesthetically pleasing but unobtrusive. |
|
We limited the main screen to three saved songs to avoid overwhelming users. The "new song" and "about" buttons remain prominent. Tapping on a song launches the compose screen, shown below. Our original design included small "rename", "duplicate", "export", and "delete" buttons on each song, but our paper and heuristic evaluations noted that such nested buttons were confusing and difficult to use on a small screen. We therefore replaced them with a menu accessed by tapping and holding for a second or so; this method is nonobvious, so we also included an instructional note. |
|
Renaming a song works by replacing the name in the song button with an editable text box; when the user submits (presses enter) the song is saved. Duplicating a song creates a new song named "Copy of NAME" and enters rename mode on that song. Deleting a song asks for confirmation, then slides the song button rightward off the screen. (It would perhaps be clearer for the button to shrink to nothing vertically with the buttons below sliding up, but technical properties of the Android API make implementing this animation difficult.) |
|
Exporting uses native Android sharing functionality to display a list of possible transfer methods for the user to pick from. A MIDI file containing the song is sent via the specified method. |
|
If the user clicks the "see all" button, this screen is shown. It lists all saved songs, allowing scrolling and searching (the list updates as a user types in the search box). |
|
The "about" button takes users to this screen, which is just a static message. |
|
This is the compose screen for a song with no notes in it (e.g. a newly created song). Swiping horizontally scrolls the song and two-finger pinching zooms it, as is common for touch interfaces. |
|
A click track plays while the user is recording. The cursor turns the color of the current track, and moves rightward across the screen (the display scrolls so that the bar is never more than 3/4 of the way across). Notes are created at the cursor position when the app detects singing, and lengthen until they change pitch or stop; notes are deleted if the cursor hits them while recording into the same track. All buttons are disabled here, but "Stop" is available to stop recording. |
|
Here the user has stopped recording on the green track, and started recording on the purple track. All four tracks are drawn on top of each other; notes on the same track can't overlap, but notes on separate tracks can. |
|
Here notes have been recorded into all four tracks. Notes from the currently selected track are bright and large; other notes are dim and small. |
|
The 'new' editing interface. Tapping on a note causes these icons to appear. Tapping the trash can deletes the note. Dragging the side arrows left or right changes the start and end points. Dragging the middle icon vertically changes the pitch of the note, and dragging it horizontally moves the whole note without changing its length. As notes in the same track can't overlap, the note stops when it hits something, and a red glowy thing appears to signal that it can't move or extend anymore. |
|
The 'old' editing interface. Tapping on a note causes this overlay to appear. Tapping the trash can deletes the note. Swiping left or right on the left side of the screen moves the start of the note backward and forward in time; swiping horizontally on the right side does the same for the end of the note. Swiping up or down in the middle of the screen changes the pitch. There's no way to move the note without changing its length. If the note tries to expand but hits something, it stops and a red glowy thing appears; also, the arrows that don't do anything are crossed out. |
|
Tapping on the track tools button brings the user to this screen, which has color-coded controls for each track. A muted track doesn't appear in the compose screen and produces no sound during playback. |
|
The instrument menu allows the user to choose what musical instrument the track will be played as. These selections are saved with the song, so they persist across editing sessions. |
|
If there are unsaved changes to the song when the user tries to leave the compose screen, the app prompts the user to save. There are three options: save then exit, exit, or don't exit. |
|
When saving a newly created song, the user is prompted for a song name. After that, saving uses the existing name, and renaming from the song menu is necessary to change it. |
Implementation
Most of our app is written in Java using the Android APIs, but there are native libraries to perform playback and recording. (We used native code because there were no convenient Java libraries, and implementing the functionality ourselves would have been difficult and possibly resulted in slow and buggy code. We used a pure Java / Android API implementation for playback in the prototype, and the quality was fairly terrible.) There is also a third-party library (com.leff.midi) that implements the MIDI message protocol and file format. Recording grabs audio samples from the microphone and outputs pitches, from which the compose code can create notes. Playback takes notes and sends MIDI messages to a system audio library, which produces audio output.
Beyond those, the only important backend class is Song, which encapsulates the concept of a song with a name, a mapping of tracks to instruments, and a set of notes, each of which has a track, pitch, a start time, and a duration. Songs can be written to and read from file storage; the app creates a file in private storage for each saved song. Songs can also export themselves to MIDI files.
An activity is an Android concept described as "a single, focused thing that the user can do". It's typically a single screen or view that the user can see and interact with. Our app has four activities: the main screen, the about screen, the full song list screen, and the compose screen. The main and song list activities contain SongWidgets, which are the buttons in the lists; each one corresponds to a song and has code for interacting with that song.
The compose activity implements all the various button actions, interacting with recording, playback, and file storage. It has various state including cursor position, selected track, and zoom level. It also contains a list of notes. Each note is a custom GUI widget that draws itself onto the compose screen; it keeps track of its track, pitch, start time, and duration, and adjusts its color, position, and size as appropriate.
Each note listens for touch events; if the note is tapped, it starts the edit mode. In the 'old' interface, this is a full-screen semi-transparent widget that detects swipe events and modifies the note based on their starting point, direction, and length. In the 'new' interface, this makes the various icons visible; they listen for touch events and modify the note when dragged.
The compose activity contains the undo stack, which is a list of UndoItems. An UndoItem is added whenever a change is made, and the stack is modified when the undo and redo buttons are pressed. When the user saves a song, the compose activity converts its internal data structures into a Song, and saves it to file storage.
We planned to play back notes in other tracks while recording, à la overdubbing, so that users could more easily create harmonies when recording different tracks, but the microphone picked up the notes and it messed up the pitch detection. We considered something like disabling that feature unless headphones were plugged in, but concluded it wasn't worth it.
Evaluation
Our users were friends from our living groups who had varying levels of interest in music. While they're all MIT students and so might not be generally representative, MIT students seem to have a pretty wide range of musical inclination and so weren't a particularly bad sampling of our user population, though most of them were programmers (don't think any of them had programmed on Android before, though). Our app aims to be usable by people with varying levels of music experience and who are probably somewhat younger, creative and tech-savvy enough to carry a smart phone.
We conducted user tasks by telling the user the purpose of the app (record songs without having to figure out the musical notation and when you're on the go) and the target audience (should be usable by people without a lot of musical experience).
We mentioned that they should think out loud, and that we were testing the app's usability, not their intelligence.
We had two editing interfaces that we can't decide between. We tested this by having different users test different editing interfaces first, give us their comments on it, then switch and give us their comments on the other one. I should've been careful to not tell users who made which interface, but I think I failed at this for the first two.
We tried to stay silent during observation, only giving tasks, then at the end asking questions about certain things the user did that they didn't explain, or things they may not have discovered.
Tasks
1. Record something
2. Edit
3. Add a harmony line
4. Export
User summaries
User 1
Starting with (old edit) interface
Course 6
Android User
Experienced Chorollaries singer
User 2
Using (new-edit) interface first.
Course 6
iPhone user
Casual composer, doesn't often finish songs
User 3
Starting with (old-edit) interface.
Singing and playing instruments since he was 7
Doesn't own a smartphone, so I briefed him on Android usage. Function buttons, long tap, and zoom gesture.
"I'm definitely not a computer person"
We are sitting next to the piano and I mention that piano records nicely.
Bit worried about this user test because he seemed unwilling to hurt our feelings or worried about not making mistakes.
User 4
Starting with (new-edit) interface
"I warn you, I do usability testing for my lab, so I will, like, give you user interface feedback."
singing for as long as can remember in a wide variety of genres
"I just finished singing, so I'm warmed up"
Course 6
"I don't have any theory background, only performance."
Discovered usability issues
The "Users" column indicates how many (out of 4) of our testers had that issue.
Issue |
Users |
Possible fix |
Severity |
---|---|---|---|
Small notes are too difficult to tap; caused some cursing |
4 |
We added a "hit extender" to help resolve this, but it doesn't help in the x direction. We could potentially add next/prev note buttons. |
Critical |
Tries tapping arrows in old-edit rather than swiping them |
4 |
Usually they figure out how to use it fine after trying a few more things. |
Minor |
Accidentally start recording/play in the wrong place |
4 |
Maybe copy Finale's method of playing from beginning usually, not sure about record. Pretty recoverable. |
Minor |
App crashes after export |
4 |
Bug |
Critical |
Phone doesn't pick up melody anywhere near accurately |
3 |
?? Editing or re-recording is often sufficient. User often needs to hold mike closer to mouth or sing with a steadier voice (vibrato does not seem to help), maybe if they have to undo recording several times the program can suggest this. Recording seems to record just as well with words being sung as with random syllable, so it's not that. A user suggests that the app display your actual voice pitches so they can tune the lines to the voice. A user resorts to using piano to record, but this isn't our use case. |
Major |
Instrument change/track muting not visible |
3 |
1 person only knew where it was because he found it while looking for something else. These features need to be more visible. One user suggests double tapping track button to bring up options. |
Major |
(old-edit) should have move left/right function. 2 actions is "bizarre and awkward" |
3 |
Main/deciding reason people cited for preferring (new-edit), wouldn't be hard to just add. |
Critical |
Confusing back button with okay in old-edit interface, even after using many times |
3 |
Perhaps move them closer together, where you can see them all? |
Major |
If a song contains two notes on the same pitch, they often fuse, and there is no way to split notes |
3 |
Add a split note function |
Major |
Feels like it's impossible to coordinate harmony with melody |
3 |
1 was just really careful, 2 suggest playing the melody back while recording harmony, 1 suggests program do some AI to coordinate melodies and note lengths. |
Major |
(old-edit) Arrows obstruct note you're editing |
3 |
This is especially common on notes near start of song, since they can't be centered. Maybe if overlay were more transparent. |
Major |
Want to move/delete segment of notes en-masse |
2 |
a third person had an error in their music that would've been much more easily resolved with this function. We should have it. |
Major |
Wants to be able to have notes same length/multiples of same length like in sheet music |
2 |
Maybe we should do some AI for this? |
Major |
It's not obvious that notes are editable |
Unclear, at least 3 non-tested people |
?? |
Critical |
Accidentally presses play when they mean record. May not realize it even after trying several times to record. |
2 |
Not sure what's causing this? |
Minor |
Found export difficult to find |
2 |
?? |
Major? |
Stop is difficult to find while recording. |
2 |
Seems like users expect tapping the cursor to stop the song, or that stop should be red. We might try changing that. |
Minor |
Can't tell which buttons are disabled |
2 |
Might be more but I only asked 2. Should make buttons non-gray or something |
Minor |
Phone records an octave above where it should, only very briefly though |
2 |
This and other noise during recording are easily deleted in interface (Besides the fact that they're too short) |
Cosmetic |
Didn't notice undo button |
2 |
?? |
Minor |
Accidentally go into edit mode for a note when trying to move cursor |
2 |
?? |
Cosmetic |
Wants to zoom closer/farther than min/max zoom |
2 |
Minor |
|
Didn't notice zooming into scale was possible |
2 |
Might be more than 2, but 1 was instructed. ?? |
Minor |
(new-edit) Accidentally presses trashcan while editing |
1 |
Moved trash can after this user |
Fixed |
Wants a next/prev note button when editing |
1 |
Only 1 mentioned this, but others were also editing essentially linearly |
Cosmetic |
Don't realize what note length not being able to increase means |
1 |
This was fixed after a user test by implementing red bars when notes butt against each other. |
Fixed |
Does not realize that tracks record other overlayed melodies |
1 |
He actually recorded over his original melody, which was pretty bad. Saw his grayed-out melody on other tracks and thought they were other people's songs. This came up in heuristic testing too; it's unclear that people realize what tracks are until instructed to "record a harmony" |
Major |
Play doesn't begin from where expected |
1 |
Cursor did not follow editing, and played from seemingly random place. Cursor usually moves to thing you're editing in word editing, so maybe it should. |
Minor |
Wants to be able to add notes |
1 |
We could add this somehow |
Cosmetic |
(old-edit) Loses pitch while trying to change note and has to keep re-playing |
1 |
User suggested this would be fixed with next/prev note function. Another user hummed the pitch they wanted while adjusting to keep track of it. |
Minor |
Note disappears while editing (new-edit) |
1 |
Bug |
Critical |
Buttons at top of screen make user cover screen |
1 |
switch top and bottom buttons |
Minor |
Wanted to hold record to record |
1 |
?? |
?? |
Wanted to be able to pull notes past each other |
1 |
We could make notes on a track switchable. |
Cosmetic |
The default instrument does not sustain, doesn't match interface |
1 |
We could switch it |
Cosmetic |
(new-edit) wanted the delete button closer to note |
1 |
Conflicts with another user's accidentally pressing trashcan |
?? |
During playback, single note on another track played twice (replayed when a 2nd note on another track started) |
1 |
Bug |
Minor |
Looked for exported song in my files |
1 |
He was one of 2 actual android users, maybe other users would've tried this. Not obvious where to look for export. |
Major |
Feels like scroll is broken when recording is shorter than phone length |
1 |
Came up in a heuristic test too. Maybe make scrolling available anyway |
Minor |
(old-edit) Shortening/lengthening note was not working, note wasn't even visible on screen |
1 |
Bug? |
Major |
Reflection
"Our project was so shiny we used it as a mirror." - David Benjamin
"Despite the limitations of paper prototyping in representing our application, we learned that the spiral model of development was quite valuable, as our project saw a pretty large overhaul between user testings during that iteration of design. We learned that what users expect can often be pretty unintuitive; for example, there are a lot of expectations about where buttons are and how they will work that users probably couldn't tell you they expect to see, but that their behavior in repeatedly using the wrong button shows (e.g. using play as record and thinking it's broken). Personally, I learned some Java and Git from Alan; I hope I didn't frustrate him too much. If we had to do it again, we should probably have gotten GR4 in on time so that our testers didn't get reassigned. That was kind of unfortunate." - Di Liu
"We prototyped the easy things and not the hard things." - Alan Huang
"We evaluated the results of our observations by, I don't know, thinking about them." - Alan Huang
"Really hard." - Alan Huang
"Totally." - Alan Huang
It was fun! Thanks!
1 Comment
Unknown User (juhokim@mit.edu)
"Overall Wiki presentation
: best GR write-up from you guys.
Design description: Nice idea to have screenshots to discuss important design changes
Implementation: great
Evaluation: nice list-up of usability issues observed. Like the design of the usability issues table.
Reflection: Maybe a bit more thoughts on what you learned and how to improve the design process?
"