One of the biggest challenges of moving from a yearly release cycle to smaller and more frequent updates is gathering both timely and actionable user feedback. The phases of an annual release cycle conveniently align with the seasons: spring is for design, summer and fall are for development and user testing, and winter is devoted to final packaging and release. In Agile environments, user testing often comes at the completion of development, when it’s too late to make any significant changes to the design or implementation of a given feature. It’s a cold, cold winter when beta testing reveals a plethora of usability issues that our teams have no time to address.
When the AutoCAD team at Autodesk adopted an Agile process, the UX team created a user research partnership model—customer councils—which help us validate the value, usability, and usefulness of specific feature areas on a regular and frequent basis.
What are Customer Councils?
A customer council is an online community of 50-75 users who are carefully selected based on experience and expertise in a given feature area. Customer councils are created when we plan to invest in substantial user research for areas that are difficult to understand, either because of the complexity of the various workflows they involve, or because they support new market opportunities. During the life of a customer council, which can consist of several months to a year, members frequently interact with product teams.
We seek feedback in our councils from the broadest range of users who would benefit from the feature under development, hence the general term “customer council.”
Recruiting for Customer Councils
Because we expect council members to engage with us over a long period of time and to help us understand specific features or workflows, it’s important to recruit carefully.
In preparation for recruiting, the product team—including product managers, user researchers, and interaction designers—defines a small set of targeted personas or profiles that are used to develop a screening survey. We send this survey to our existing user research pool with descriptions of the kind of person we’re looking for and what council participation involves. To select among the survey respondents, we weigh factors such as the balance of job roles and industry representation across the council. We over-recruit by about 300 percent to account for attrition.
Setting clear expectations about how much time members will be expected to contribute is essential. We go through several rounds of confirmation with invitees so that those who can’t commit enough time drop out early, allowing us to fill the council with people who can promise ongoing engagement. Generally, 10-20 percent of participants drop out within the first few weeks of the council, with a similar drop in participation after each set of tasks.
Managing Customer Councils
A successful council requires careful management of the communication environment and initial activities. A private customized site is created for each customer council (Figure 1). This is a place where members can talk with product team members as well as other council members. Later, the site hosts software downloads and other information for council activities.
Once a council is established, we immediately engage council members and establish their commitment to the longer-term responsibilities of membership. A typical sequence might build from simple to more in-depth activities:
- Share photos and bios of the product team and customer council members (shown in Figure 2)
- Hold webinars introducing new functionality and send out quick polls during the webinar to both engage participants and gather data
- Initiate discussions in the online forums and in one-on-one informal interviews
- Share work samples (such as point cloud scans of their projects)
Participants are not compensated for general participation in a council. We took this approach because we wanted to emphasize building a design partnership and promoting repeated small interactions rather than one-off transaction-based interactions. However, when a member participates in an hour-long one-on-one feedback session, we do offer our standard research compensation.
Council members are often sensitive to long gaps of inactivity and stale builds, so working with product teams to ensure active engagement is critical.
How the AutoCAD Team uses Customer Councils
Customer councils facilitate an ongoing conversation between customers and the product team. Although some research and testing methods discourage repeat participants, in our approach each member provides feedback in multiple ways over the lifespan of a council. They become more invested in the process as they see their feedback reflected in design changes. In return, product team members begin thinking in terms of concrete problems and real-world scenarios.
There are many creative ways to take advantage of the quick and easy access to customers that a council structure allows:
- Short surveys
- Directed topics in online discussion forums
- Webinars to introduce new features and take quick polls on first impressions
- Small group discussions on particular topics
- Structured usability sessions
- One-on-one informal interviews
- “In-the-wild” release testing
As these methods are reasonably self-explanatory, we’ll go into greater detail only on the latter two.
One-on-One Informal Interviews
During feature planning and development, we regularly schedule customer council members for informal one-on-one interviews. In these conversations, our team picks specific research techniques based on the current project focus. During the early stages of a project, conversations may cover concept testing or strategic needs assessments. Later stages of development might be more focused on workflow validation or usability testing. Ideally, each council member will be invited to at least a single one-on-one interview during the council lifecycle.
On a recent project, a series of informal one-on-one interviews helped us prioritize feature implementation. The product team had proposed a wide range of features that would support different kinds of product use cases. Our interaction designer created realistic flat mock-ups of key features, and in individual interviews we walked participants through the interactions and capabilities. They responded by describing each features’ relevance to their particular work needs, and gave us their prioritization of the different features. With this feedback, we revised our development plans, holding off on some features we really liked in order to build functionality that customers identified as best fitting their workflow needs.
In-The-Wild Release Testing
Once development is underway, we work with council members to collect feedback on what’s being implemented. Council members are given a copy of our software, which may only have partial functionality for some features, and are encouraged to spend time over a couple of weeks evaluating a particular feature. We define specific tasks, as we might for a moderator-led usability study, but members complete them independently and provide feedback through surveys and on our discussion forums. During this phase, product team members might ask specific questions to spark discussion, or respond to posts to ask for further details.
In planning tasks, using some creativity and applying team-building techniques can help council members get excited about the feature and the testing. For instance, when we needed to understand the impact and usability of a new remote communication tool embedded in our drawing software, we created a contest. Council members were divided into small virtual teams, provided a drawing of a house and backyard, and asked to work collaboratively (and remotely) to redesign the backyard.
As most of the participants worked in architecture or engineering construction, we expected this would yield competent design work. What we didn’t expect was the level of creativity that they brought to the exercise! Different teams built things like a golf course, an alligator pit, and a mother-in-law house surrounded by a moat. They exercised the limits of the remote collaboration tool in order to come up with crazy and fun ideas. This depth of uwsage gave us significant usability feedback as well as a better understanding of the relative importance of certain planned future features like image uploading capabilities.
Customer Council Tips
Here are some tips on how to get the most out of customer councils:
- Put faces to names and encourage conversations through webinars and online discussion forums.
- Where possible, leverage friendly competition and teamwork to inspire engagement.
- Contact customer council members as often as they are willing through a variety of means including discussion forums, quick surveys, and one-on-one conversations.
- Send weekly emails pointing out new information on the website, which can be very helpful in keeping engagement high.
- Mix informal methods and early concept testing with more traditional usability testing so you don’t have to rely on product or prototype readiness to gather useful feedback.
- Focus on specific features and workflows to ensure that you are evaluating real-world tasks in a realistic way.
We consider our customer councils to be successful when:
- Input is gathered early and often
- Built features are evaluated in a realistic environment
- Each feature is validated through real customer usage and feedback
- Customers see the results of their design and testing feedback
- Features are included in the release only when they meet defined targets
For our team, customer councils have helped to ensure quality user research while changing to an Agile development culture. Winter is coming…but our councils help us to prepare by providing both strategic insight and practical usability feedback throughout the year.
Retrieved from http://uxpamagazine.org/customer-councils/