HFID :: User Testing

Selection of Variable

Throughout this project, user testing has been an invaluable resource in providing us with additional data to make informed design decisions. In our final user tests of the semester, we tested a variable impacting user confidence, our most important design goal. Users were examined to determine how confident they felt with modification of their privacy settings when using the interface of the product. Throughout most of our process, this issue was a high-priority issue, but nevertheless remained unresolved. Each of the different options we had developed had advantages and disadvantages with no clear best option. As such, when we began user testing, this unresolved issue was the largest question we wanted to analyze.

Test Design

After determining that we wanted to investigate autosaving, we identified three major different types of saving functionality we wanted to test. First, we tested the same autosave overlay we used when creating our initial prototype. This autosave overlay covered the entire profile when a pivot was modified for a brief pause. Additionally, to inform the user that a save action was taking place, this overlay displayed a checkmark and the word “Saving...”.

Autosave - Overlay

One of our primary concerns with the autosave overlay was that it was extremely invasive. The autosave overlay completely blocked the user from further interaction until it had cleared. To address this concern, we added a small animation next to the pivot. This animation would automatically display a progress bar when the user navigated away from the pivot. The progress bar would be replaced by a checkmark to indicate that the changes had been successfully saved.

Autosave - Pivot Indicator

Lastly, we wanted to test manual saving. Although it seemed to us that, in today's AJAX enabled world, autosaving would be preferred and expected, we wondered if manual saving would provide additional confidence to the user. As such, we decided to test a pop-up save dialog that would display at the top of the screen whenever the user made a change.

Manual Save

To maintain randomness throughout the testing process, we developed a script to randomly select the order that the three prototypes would be presented in. Users given first briefly shown an intro page and then directed to the prototypes.

User Intro Page

Each user would go through the three prototypes and finally be directed to a survey page. This survey asked the following questions:

  • Out of these three designs for saving, which one did you like the best?
  • In which design were you most confident that your changes had been saved?
  • With any of the designs did you ever feel like you did not know if/how your changes were being saved?
  • Without prior knowledge of the interface, did you expect it to automatically save changes or manually require you to?
  • Under autosave, were/would you ever be worried about accidentally making a mistake and needing undo functionality?
  • Which design was easiest to understand?
  • General Thoughts?

These questions were intended to allow us to analyze how the user felt about each of the three prototypes. Additionally, certain questions were designed to behave as sanity checks. For example, we expected the same prototype would make users feel most confident, be easiest to understand, and be most liked. Any deviation from this would signal that we needed to dig deeper to understand why users had apparently conflicted feelings.

Test Results and Analysis

When we compiled the results of our test, we found that no prototype was clearly preferred by the users. There was an almost equal three way split between the prototypes on the metrics of overall user preference, as shown in the following chart.

Additionally, we asked users which prototype provided the most confidence that changes had successfully been changed. Using this metric, users had a clear preference for manual saving.

Although this was useful data by itself, it also made us ask why users did not prefer the prototype which made them feel most confident. Our studies had showed that confidence in privacy settings was one of the key issues facing our users and this seemed to be an irregularity. However, upon analyzing the comments, we found that although users certainly felt more confident in the interaction with manual saving, there was an appeal to the automatic aspect of autosaving. Nevertheless, users generally expected a manual saving functionality and having to click “Save Changes” was an interaction that inspired confidence. Autosaving only provides a one-way flow of information. Manual saving is a two-way interaction that inherently creates trust and confidence. Given this result, we decided to use manual saving as we moved forward.