What does a startup company like Spout (www.spout.com) do to learn how well their site is working for customers and investigate what would increase its success? Their website has been called, “A genius hybrid of MySpace and Netflix,” and Spout’s founders believe, “If we can make it easy for people to share movies they love with others…really meaningful stories will find their audience.”

• Six months after the initial release of Spout.com, the company had many questions as they planned the evolution of the site:
• What will keep our target audience returning to the site?
• When will visitors become members and why?
• What kind of movie advice does the target audience want or need?
• How important is the community side of Spout.com? What causes people to become active members (writing movie reviews, adding personal tags to movie descriptions)?
• Overall, how useful does the target audience find the site and why?

Spout’s design partner, digital agency BBK Studios, recommended that Spout engage a neutral third-party research partner to help them understand how users interact with Spout.com. The designers at BBK Studios follow user-centered design methodology: they had conducted focus groups, a usability test, and ongoing user interviews to inform their design. Now BBK Studios judged it was time to involve user research specialists as they entered the next design phase for Spout.com.

Choosing an Integrated Approach

BBK was familiar with our firm Tec-Ed’s expertise in user research, and we began brainstorming in planning meetings with Spout. Unlike most startups, Spout and their design partner had laid out preliminary ideas for a year-long research program, so we had an excellent starting point for discussion. Like many startups, though, Spout had a limited research budget.

The Spout management team wanted to collect data with larger samples than had been used in the initial design research, so they first suggested that we conduct survey research. However, Spout’s primary goal was to gain more in-depth understanding of their potential user community, and several of their key questions asked why site visitors behave as they do.

We discussed the trade-offs with key stakeholders. A benefit of working with a small startup client is that Spout’s CEO and marketing VP were involved in the planning, as were BBK Studio’s management. Then we recommended a balanced program of qualitative and quantitative research that addressed Spout’s questions and stayed within budget. We proposed conducting qualitative usability testing twice a year, with each test to be followed by quantitative research to address questions that emerged from the qualitative research results.

Anticipated Challenges

During our early planning, we were concerned about achieving valid results from the quantitative research because even the most carefully crafted survey can only report what people say, not their actual behavior. Some surveys fail to predict user behavior because respondents are answering the questions outside their context of use, and no researcher is present to probe into the reasons behind their answers.

Therefore, Tec-Ed recommended improving the rigor—and success—of Spout’s quantitative research plan by conducting task-based surveys. Our goal was to introduce a behavioral component into the quantitative research phase by having participants perform tasks on the Spout website during the survey. This would require careful coordination not only between Tec-Ed and Spout, but also with the survey hosting vendor and with Spout’s partners responsible for the technical implementation of Spout.com.

In addition, we were concerned about how long it might take to recruit participants for both research phases whose characteristics matched Spout’s target audience: people who watch at least three movies a month and who regularly consult online information (such as Netflix, Blockbuster Online, Yahoo Movies) to decide how to invest their movie consumption time.

Qualitative Research Goals and Methodology

Spout’s high-level questions exemplify the integration of marketing and usability research—they are central to the marketing strategy and branding for Spout.com, and they’re rooted in the user experience.

When we planned the qualitative usability testing that began the project, we focused on questions that would be best answered by observing hands-on use.

• What features do visitors use on Spout.com and why?
• What is people’s impression of the “community” aspect of spout.com?
• What would encourage visitors to register?
• Why would people write a movie review?

To explore these issues, we asked the usability test participants to perform four tasks and we observed what site features they used and why. We asked participants to find out about a movie which we knew the site provided rich content for, and then to look for a movie we knew had no member-contributed reviews. Finally, we asked participants to write a review on the last movie they saw and noted their reactions to the review creation process.

Qualitative Research Results and What Spout Did

The qualitative usability test yielded many positive findings for marketing. The majority of the participants successfully used the reviews on Spout.com to help them decide on a movie, even when (in the first task) they had said they usually don’t do so. Although participants didn’t immediately understand the community aspects of the site, after exploring the features, most found them interesting and intriguing.

Two interrelated areas confused the usability test participants: they were uncertain exactly what Spout was and how to use the site to the best advantage. Participants wondered whether the primary goal of the site was to:

• Sell DVDs
• Provide information about movies
• Provide a blog space
• Advertise movies
• Build a database of people interested in movies

Participants also wanted to know why they should sign up—how would they benefit, and what disadvantages might there be (such as receiving too many advertising emails).

The second area of confusion was the terminology on the site. The terms used for different features (tags, talk, details, experts, people) were either ambiguous to participants or meant something different from what they expected. For example, participants were not sure who the experts were or how they became “experts.”

Spout immediately began addressing the two problem areas the qualitative usability testing uncovered. They added choices on the home page to help visitors see benefits. Clicking on “Learn more” (Figure 1) took visitors to a one-minute tour (Figure 2), which we then tested in the quantitative research.

Quantitative Research: Testing First Impressions

When we worked with Spout stakeholders to identify the objectives for the quantitative research, we learned that Spout wanted to focus on initial user reactions to a new site landing page, an updated Spout tour, and a new MovieMind tool (Figure 3) that helped people select movies.

In our initial design of the task-based survey, participants moved back and forth between Spout.com and an online survey site run by a third-party vendor. Participants started with background questions in the survey, switched to Spout.com to perform each task, then returning to the survey to answer questions about the task. However, during review we found snags in this approach: problems arose with synchronization of data between the sites, and the back-and-forth process made the survey take too long to complete.

Fortunately, users could experience the landing page and the tour with static screens, so we were able to simplify the research design by illustrating the survey questions with screen captures, plus just one branch to the live Spout.com site for trying the MovieMind tool. The survey questions addressed participants’ specific reactions to the screens. Taking the tour and using MovieMind gave participants a realistic and immediate context for the opinions we elicited about usefulness, appeal, and value of the site.

We also encountered some of the anticipated problems with participant screening. With our original criteria, the survey vendor would have to send out more than 132,000 invitations to receive 350 completed surveys from qualified participants, which required much more than the available budget. We compromised by adjusting the screening criteria (accepting participants who visit only one rather than two movie sites) and by reducing the number of participants to 250.

Results from the Quantitative Research

After analyzing the survey results, we concluded that:

• Participants were generally positive about Spout.com
• When participants made comments, the negative voices tended to be strong, the positive voices tepid
• Gender, parental status, and age differences had little effect on the findings
• Participants appeared to become more positive as they learned more about the site
• Half the participants said they were likely or very likely to return

Although the tour did not get rave reviews, it succeeded in describing Spout to the majority of new users. Also, after going through the tour screens, more of the participants said they would become Spout members. When we asked participants their opinions of the recommendations the MovieMind provided, half said the MovieMind recommendations were appropriate or very appropriate.

Overall, the quantitative research confirmed that Spout had addressed the usability test participants’ confusion about the purpose of the site. It also validated Spout’s approach to encouraging first-time visitors to join the community.

Integrated Qualitative and Quantitative Research for Spout and Others

Spout’s program of integrated qualitative and quantitative research was unusual because most startups don’t yet understand the value of user research, and fewer still will plan longitudinal research programs. All of Tec‑Ed’s other integrated qualitative and quantitative research projects have been for Fortune 500 companies in business over ten years with large in-house user experience groups.

Another memorable aspect of the project was Spout’s willingness to collaborate on innovative approaches such as our task-based survey methodology, which required cooperation from the IT specialists maintaining and enhancing Spout.com. Again, many startups would not invest scarce technical resources to learn more about their customers.

On the other hand, the Spout project was similar in many ways to other integrated qualitative and quantitative research projects we have performed recently. Some of the issues we consider for all such projects include:

• Deciding which method to use first; this is not a “no-brainer”. More often, we begin with qualitative research because the richness of observed behavior helps us design more relevant survey questions. However, sometimes a broad survey can help identify or prioritize what topics should be the focus of qualitative research.
• Defining and implementing participant screening criteria that work for both methods. It’s usually important to keep the same audience groups or customer segments for both the qualitative and quantitative portions of the research. Sometimes the findings from the first phase cause a change in the audience definitions for phase two, which raises the tricky question, “Is it more important to have research participants who accurately represent the target audience, or to compare more accurately the two kinds of research results?”
• What research questions should be addressed in which phase of the research? When Tec‑Ed defines a research project, our client’s “wish list” of research questions usually includes some that should be explored with qualitative research, others that are clearly suited to quantitative research, and still others that could usefully be addressed by either method. Part of our initial consultation is often discussing the trade-offs of qualitative and quantitative methodology, so our clients can use their research resources most effectively.

Our Spout project was especially rewarding because the many stakeholders at Spout, BBK Studios, and Tec-Ed worked effectively together in a successful partnership. This case history illustrates how you don’t have to work for a Fortune 500 company to carry out cutting-edge user experience research. Smaller organizations and startups can also benefit from conducting integrated qualitative and quantitative research.