Zag Multiple Partner Platform Baseline Usability Study
conducted independently © 2008 by Susan Oslin : (323) 664-3880 : susan@MMMcreative.com

Multiple Partner Platform Usability Baseline Test Plan


Summary

Web analytics have been collected for the MPP platform and specific portions of pages have been A/B tested or evaluated, but there has been no user testing or overall look at the entire site.  Analytic data shows up to 70% fall off from the home page, decreasing that by half could potentially double sales.  A series of flow diagrams of the MPP platform have been created and indicate a number of areas for improvement from an interface design and flow perspective.  The purpose of this study is to gather objective data to create baseline usability measures, to identify areas of confusion, frustration, and satisfaction—to conduct a usability test that takes a holistic look at the site.


Overall objectives for the study

We will gather baseline data about the overall effectiveness of the MPP platform by testing www.driveyourdream.com . The goals of this study are to:

  • Assess the overall effectiveness of www.driveyourdream.com for different types of users purchasing a car.
  • Identify areas of confusion of the flow and obstacles to reaching the certificate page.
  • Compare and contrast the new and used car flows.
  • Understand the reasons for fall off from the home page and look for ways to improve the interface design.
  • Identify usage patterns that could be further investigated in follow-up research.

Research questions

In addition, in this study will try to answer these questions:

  • How well does the site support the paths and goals of the users?  That is, how closely does the organization and flow of the site match customers’ expectations?
  • What questions do customers ask as they work through their purchase?
  • What paths do customers take to reach the certificate page?  What obstacles do they encounter?
  • Are they able to find the information they are looking for that help them to make an informed car purchase, i.e. consumer reports, comparisons, technical data, etc.?
  • How do customers feel about the overall process, are they clear about the number and nature of each step?
  • How does the used car vs. new car process compare?  Are they similar?  Was one easier, clearer than the other?  What suggestions do they have?
  • How can we improve the home page?  Is the intention of the site clear?

Measures

At the end of the sessions, we will have mostly qualitative data.  Any quantitative data we gather will not be statistically significant but major trends may be garnered:

  • Pre-test survey to understand relevant buying patterns, home page survey to measure understanding and initial impression, and post-test survey to measure reactions to the site.
  • Number of times and location in the flow of negative commentary or apparent confusion recorded by the test administrator.
  • Number of times and location in the flow of positive commentary or apparent satisfaction recorded by the administrator.
  • The exact path each user took in the form of screen capture.
  • Whether participants were able to reach the certificate page and understood what their next step was. 
  • The recorded commentary that participants make as they think aloud – will give us indicators about what participants were confused by and why.
  • Debriefing interviews will tell us what stands out about the experience of using the site, which should help us set priorities on potential changes to the site.
  • Appropriateness of site’s functions to users’ tasks, overall ease of use, usefulness of terms and labeling.

Location and setup

The study will take place in the home/office work space of MMM Creative in Los Angeles, CA. Participants will use a Windows PC and FireFox 3.0 with a high-speed connection to the Internet.  The PC that the participant uses will also have Morae recorder installed on it, a web camera attached and the participant will be wearing a headset. The web camera will capture the participant’s face; the headset will capture the participant’s voice; the Morae software will record what’s happening on the screen (and can collect other data).  www.driveyourdream.com will be used as the test site for all participants.


Recruiting participants

We will select participants who have a variety of experience purchasing cars. They will be people who own their own car and are active drivers. MMM Creative will recruit 6-8 participants with the characteristics outlined in the following table.

Characteristic Desired number
of participants
Internet  average usage frequency
infrequently: 1–5 hours per week
moderately often: 6–15 hours per week
very often: more than 15 hours per week

2
2-4
2
Internet average purchasing frequency
infrequently: has purchased, but less than 1 purchase per month 
moderately often: several items regularly purchased per month
very often: would prefer never going to the mall again

2
2-4
2
Purchasing experience
currently own their own car
have purchased a new car in their life time
have purchased a used car in their life time

all
2-4
2-4
Age
25–39
40–54
over 55

1-3
1-3
1-3
Gender
female
male

2-4
2-4

We will exclude people who:

  • Don’t own a car
  • Don’t own a computer
  • Are affiliated with zag.com or the auto buying industry

Methodology

This usability study will be mostly exploratory but will also gather assessment data about the effectiveness of www.driveyourdream.com . Participants will each be asked to purchase both a new and a used car.  We will collect data about error and success rates as well as qualitative data about participants’ experiences using the site.

We will use a within-subjects design
In this within-subjects study, each participant will work through both the new and used path in counterbalanced order. I will conduct up to 8 individual 55-minute usability study sessions.  I’ll use 5 minutes of each session to explain the session to the participant, review basic background information with the participant, what the test is and is not, put them at ease, have them answer a brief survey to collect data on their relevant purchasing patterns and then conduct a 10 minute post-test debriefing interview and brief survey of their experience. During the middle 40 minutes of the session, participants will review the home page and answer a short survey before continuing on with purchasing a new and used car at a dealer local to their home address.

Moderator role
I will sit in the room with the participant while conducting the session. I will introduce the session and then introduce tasks as appropriate.  Because this study is somewhat exploratory, I may ask unscripted follow-up questions to clarify participants’ behavior and expectations. I will take some notes but focus more on observing.

Sessions will be digitally recorded on video and audio using TechSmith’s Morae product and the sessions will be more closely analyzed from the recordings.

Pre-test arrangements Have the participant:
Review and sign nondisclosures and recording permissions.

Introduction to the session. (2 minutes) Discuss:

  • Participant’s experience with usability studies and focus groups.
  • Importance of their involvement in the study.
  • Moderator’s role.
  • Room configuration, recording systems, observers, etc.
  • The protocol for the rest of the session.
  • Thinking aloud.

Background survey. (3 minutes) Captures the participant’s:

  • Car buying history.
  • Internet usage patterns.

Tasks (40 minutes)

  • Participants will start from the home page.
  • Prior to continuing they will be asked to answer a brief survey. (5 minutes)
  • Purchase either a new or used car according to the counter balance scheme for that participant. (not more than 20 minutes)
  • Purchase the remaining new or used car according to the counter balance scheme for that participant. (at least 15 minutes)

Post-test debriefing (10 minutes)

  • Answer a post-test questionnaire for general reaction data collection.
  • Ask broad questions to collect preference and other qualitative data.
  • Follow up on any particular problems that came up for the participant.

Session schedule
The table below lays out a daily schedule of sessions for the study.

Time Sunday, Oct 12 Monday, Oct 13
11:00    
12:00 David Emmanuel
1:00 Connie  Tara
2:00    
3:00 Michael Leah
4:00    
5:00 Marylinda  
6:00    
7:00    
8:00 Paul  

Production Schedule

What When
Recruit participants September 26-October 10
Install Morea October 6
Get feedback on test plan from Director of Business Intelligence at Driveyourdream.com October 7
Get feedback on session script from usability research colleague October 9
Technical set-up of test station October 9
Deliver final test plan and session script October 10
Dry run of session October 11
6 to 8 usability study sessions of 55 minutes each October 12,13
Deliver preliminary written report October 20
Install Axure October 20
Preliminary Home page redesign wireframe October 24
Wireframe prototype of revised flow and final home page redesign October 31
Presentation to Zag First week of November

Deliverables

  • A test plan (this document) describing the general approach and schedule for the study.
  • A final written report of complete findings and data analysis.
  • video highlights of significant findings of the Morae recordings of each of the 6-8 55 minute sessions.
  • A presentation of findings and recommendations to include: improved flow diagram  and home page design.

Test plan, session script and materials
I will get feedback on the test plan from a usability research colleague and someone from Zag. After those reviews, I will create a session script and data collecting materials that I will use during the sessions. This ensures that all of the participants receive the same instructions and that I gather data on the same issues throughout the study.

It is important that Zag review this test plan carefully and provide feedback. Major changes to the participant characteristics and the session tasks and activities after I create the session script could result in questionable data and findings.

Reviewing, tabulating, and analyzing data
Using the data collected on the surveys and the recordings, I’ll tabulate and analyze the data to answer the key questions (listed in the Goals section of this document) with findings and recommendations. I’ll provide these results in a final written report and a presentation at the Zag offices in Santa Monica.

Report contents
I will deliver a draft of the final report to my point of contact, Bill Lepler at Zag that:

  • Briefly summarizes the background of the study, including the goals, methodology, logistics, and participant characteristics
  • Presents findings for the original questions to investigate
  • Gives quantitative results and discusses specifics as appropriate to the question and the data
  • Provides visuals of pages of www.driveyourdream.com that are relevant to specific questions where they will help reviewers understand what we are talking about
  • Discusses the implications of the results
  • Provides recommendations
  • Suggests follow-on research