One of the biggest challenges of moving from a yearly release cycle to smaller and more frequent updates is gathering both timely and actionable user feedback. The phases of an annual release cycle conveniently align with the seasons: spring is for design, summer and fall are for development and user testing, and winter is devoted to final packaging and release. In Agile environments, user testing often comes at the completion of development, when it’s too late to make any significant changes to the design or implementation of a given feature. It’s a cold, cold winter when beta testing reveals a plethora of usability issues that our teams have no time to address.

When the AutoCAD team at Autodesk adopted an Agile process, the UX team created a user research partnership model—customer councils—which help us validate the value, usability, and usefulness of specific feature areas on a regular and frequent basis.

What are Customer Councils?

A customer council is an online community of 50-75 users who are carefully selected based on experience and expertise in a given feature area. Customer councils are created when we plan to invest in substantial user research for areas that are difficult to understand, either because of the complexity of the various workflows they involve, or because they support new market opportunities. During the life of a customer council, which can consist of several months to a year, members frequently interact with product teams.

We seek feedback in our councils from the broadest range of users who would benefit from the feature under development, hence the general term “customer council.”

Recruiting for Customer Councils

Because we expect council members to engage with us over a long period of time and to help us understand specific features or workflows, it’s important to recruit carefully.

In preparation for recruiting, the product team—including product managers, user researchers, and interaction designers—defines a small set of targeted personas or profiles that are used to develop a screening survey. We send this survey to our existing user research pool with descriptions of the kind of person we’re looking for and what council participation involves. To select among the survey respondents, we weigh factors such as the balance of job roles and industry representation across the council. We over-recruit by about 300 percent to account for attrition.

Setting clear expectations about how much time members will be expected to contribute is essential. We go through several rounds of confirmation with invitees so that those who can’t commit enough time drop out early, allowing us to fill the council with people who can promise ongoing engagement. Generally, 10-20 percent of participants drop out within the first few weeks of the council, with a similar drop in participation after each set of tasks.

Managing Customer Councils

A successful council requires careful management of the communication environment and initial activities. A private customized site is created for each customer council (Figure 1). This is a place where members can talk with product team members as well as other council members. Later, the site hosts software downloads and other information for council activities.

Once a council is established, we immediately engage council members and establish their commitment to the longer-term responsibilities of membership. A typical sequence might build from simple to more in-depth activities:

• Share photos and bios of the product team and customer council members (shown in Figure 2)
• Hold webinars introducing new functionality and send out quick polls during the webinar to both engage participants and gather data
• Initiate discussions in the online forums and in one-on-one informal interviews
• Share work samples (such as point cloud scans of their projects)

Participants are not compensated for general participation in a council. We took this approach because we wanted to emphasize building a design partnership and promoting repeated small interactions rather than one-off transaction-based interactions. However, when a member participates in an hour-long one-on-one feedback session, we do offer our standard research compensation.

Council members are often sensitive to long gaps of inactivity and stale builds, so working with product teams to ensure active engagement is critical.

How the AutoCAD Team uses Customer Councils

Customer councils facilitate an ongoing conversation between customers and the product team. Although some research and testing methods discourage repeat participants, in our approach each member provides feedback in multiple ways over the lifespan of a council. They become more invested in the process as they see their feedback reflected in design changes. In return, product team members begin thinking in terms of concrete problems and real-world scenarios.

There are many creative ways to take advantage of the quick and easy access to customers that a council structure allows:

• Short surveys
• Directed topics in online discussion forums
• Webinars to introduce new features and take quick polls on first impressions
• Small group discussions on particular topics
• Structured usability sessions
• One-on-one informal interviews
• “In-the-wild” release testing

As these methods are reasonably self-explanatory, we’ll go into greater detail only on the latter two.

One-on-One Informal Interviews

During feature planning and development, we regularly schedule customer council members for informal one-on-one interviews. In these conversations, our team picks specific research techniques based on the current project focus. During the early stages of a project, conversations may cover concept testing or strategic needs assessments. Later stages of development might be more focused on workflow validation or usability testing. Ideally, each council member will be invited to at least a single one-on-one interview during the council lifecycle.

On a recent project, a series of informal one-on-one interviews helped us prioritize feature implementation. The product team had proposed a wide range of features that would support different kinds of product use cases. Our interaction designer created realistic flat mock-ups of key features, and in individual interviews we walked participants through the interactions and capabilities. They responded by describing each features’ relevance to their particular work needs, and gave us their prioritization of the different features. With this feedback, we revised our development plans, holding off on some features we really liked in order to build functionality that customers identified as best fitting their workflow needs.

In-The-Wild Release Testing

Once development is underway, we work with council members to collect feedback on what’s being implemented. Council members are given a copy of our software, which may only have partial functionality for some features, and are encouraged to spend time over a couple of weeks evaluating a particular feature. We define specific tasks, as we might for a moderator-led usability study, but members complete them independently and provide feedback through surveys and on our discussion forums. During this phase, product team members might ask specific questions to spark discussion, or respond to posts to ask for further details.

In planning tasks, using some creativity and applying team-building techniques can help council members get excited about the feature and the testing. For instance, when we needed to understand the impact and usability of a new remote communication tool embedded in our drawing software, we created a contest. Council members were divided into small virtual teams, provided a drawing of a house and backyard, and asked to work collaboratively (and remotely) to redesign the backyard.

As most of the participants worked in architecture or engineering construction, we expected this would yield competent design work. What we didn’t expect was the level of creativity that they brought to the exercise! Different teams built things like a golf course, an alligator pit, and a mother-in-law house surrounded by a moat. They exercised the limits of the remote collaboration tool in order to come up with crazy and fun ideas. This depth of uwsage gave us significant usability feedback as well as a better understanding of the relative importance of certain planned future features like image uploading capabilities.

Customer Council Tips

Here are some tips on how to get the most out of customer councils:

• Put faces to names and encourage conversations through webinars and online discussion forums.
• Where possible, leverage friendly competition and teamwork to inspire engagement.
• Contact customer council members as often as they are willing through a variety of means including discussion forums, quick surveys, and one-on-one conversations.
• Send weekly emails pointing out new information on the website, which can be very helpful in keeping engagement high.
• Mix informal methods and early concept testing with more traditional usability testing so you don’t have to rely on product or prototype readiness to gather useful feedback.
• Focus on specific features and workflows to ensure that you are evaluating real-world tasks in a realistic way.

Concluding Thoughts

We consider our customer councils to be successful when:

• Input is gathered early and often
• Built features are evaluated in a realistic environment
• Each feature is validated through real customer usage and feedback
• Customers see the results of their design and testing feedback
• Features are included in the release only when they meet defined targets

For our team, customer councils have helped to ensure quality user research while changing to an Agile development culture. Winter is coming…but our councils help us to prepare by providing both strategic insight and practical usability feedback throughout the year.在敏捷开发环境中，用户测试通常是在开发工作的完成阶段才进行的，但此时要对设计或技术实现进行任何重大改动已为时太晚。 客户委员会有助于确保在敏捷开发文化中实现高质量的用户研究。 委员会是一个在线社区，由50至75名经过挑选出来的拥有特定领域专业知识的用户所组成。 他们通过论坛、 问卷调查、 访谈以及可用性测试方法，为用户研究提供快速便捷的接触客户的渠道。

文章全文为英文版애자일 개발 환경에서, 사용자 테스트는 주로 개발이 완료될 때 실시되며, 이 때는 설계나 실행에 어떤 중요한 변화를 주기에는 너무 늦습니다. 고객 위원회는 애자일 개발 문화에서 질 높은 사용자 연구를 할 수 있도록 지원합니다. 신중하게 선발된 특정영역의 전문가 50-75명의 사용자로 구성된 온라인 커뮤니티입니다. 이 커뮤니티를 이용하여 토론, 설문조사, 면담 및 사용성 테스트 등의 방법을 통하여 사용자 연구를 수행할 수 있고, 이는 고객에게 빠르고 쉽게 접근할 수 있는 방법을 제공합니다.

전체 기사는 영어로만 제공됩니다.Em ambientes ágeis, testes com usuários muitas vezes são feitos na conclusão do desenvolvimento, quando já é tarde demais para fazer qualquer alteração significativa no projeto ou na implementação. Os Comitês de Clientes ajudam a garantir a pesquisa de qualidade em uma cultura de desenvolvimento ágil. São uma comunidade online de 50 a 75 usuários, cuidadosamente selecionados por sua especialização em determinada área. Fornecem acesso rápido e fácil aos clientes para a pesquisa com o usuário através de métodos como fóruns de discussão, enquetes, entrevistas e testes de usabilidade.

O artigo completo está disponível somente em inglês.

 アジャイルの環境では、往々にしてユーザーテストは、設計や導入に大幅な変更を加えるには手遅れな開発の完了後に行われるが、顧客協議会がアジャイル開発の環境におけるユーザー研究の質を確保することに役立つ。協議会はそれぞれ、該当する分野の専門知識に基づいて慎重に選抜された50～75人のユーザーから成る、オンラインコミュニティーである。ディスカッションフォーラム、アンケート調査、インタビュー、ユーザビリティテストを用いて行われるユーザー研究の対象となるカスタマーへのアクセスを短時間でたやすく提供してくれる。

原文は英語だけになります

 En entornos ágiles, los tests de usuario se suelen realizar cuando se finaliza el desarrollo, es decir, cuando ya es demasiado tarde para realizar cualquier cambio significativo en el diseño o la implementación. Los consejos de clientes ayudan a asegurar una investigación de experiencia de usuario de calidad en una cultura de desarrollo ágil. Éstos son comunidades en línea de entre 50 y 75 usuarios seleccionados cuidadosamente por su experiencia en una determinada área. Proporcionan un acceso rápido y fácil a los clientes para realizar investigaciones de experiencia de usuario a partir de foros de discusión, encuestas, entrevistas y pruebas de usabilidad.

La versión completa de este artículo está sólo disponible en inglés