The Usability Maturity Model

Inspired by similar models for software development and quality assurance, the Usability Maturity Model synthesized by Jonathan Earthy in 1998 defines six levels of capability in human-centered processes that can describe an organization. The six levels are:

• Unrecognized or “ignorance”: Usability is not discussed as an issue.
• Recognized or “uncertainty”: The organization sees that it has problems with usability but does not know what to do about these problems.
• Considered or “awakening”: The organization understands that UCD processes are necessary to improve usability and begins to implement some of these processes.
• Implemented or “enlightenment”: The full complement of UCD processes are used and deliver good results on a per product or module basis.
• Integrated or “wisdom”: UCD processes are included in the development lifecycle methodology, results are tracked, and performance goals are met.
• Institutionalized or “certainty”: The organization is human centered in both its internal function and its product design.

For this article, we have informally assessed the levels in use at Oracle Corporation; however, the complete methodology, as presented by Earthy, includes a more formal recording of form and scale.

The Problem

UCD professionals are highly trained in their individual sub-disciplines but are often unprepared for the challenge of fully implementing UCD to produce a finished product. Occupied with user research, design, prototyping, and usability testing, UCD professionals are often shocked when some or all of their work is not applied. This disconnect occurs when UCD professionals do not consider their organization’s readiness to accept their work or perform the non-UCD processes needed to ensure success.

A Case Study

In 2005, Oracle started a new development cycle to design and build a next-generation integrated enterprise application suite. The vice president responsible for the project had worked with the UCD process in the past, and he understood that UCD would produce a more usable product. As usability was an important differentiator for the product, he consolidated the enterprise applications UCD staff into one team that reported directly to him. As further evidence of his commitment to the team’s success, he also granted the organization a generous number of new positions.

Initial Analysis

At the start of this process, the UCD team analyzed the historical effectiveness of its various working styles and noted areas where they needed improvement. The newly joined teams agreed that they had, on average, achieved a Usability Maturity Model level of Implemented in their previous team’s practice: they had dedicated teams of UCD professionals who performed the complete process to support their product teams. In addition, all the teams adhered to company-wide user interface (UI) guidelines and standards. Some had managed to institute UI code reviews to ensure compliance to these standards.

However, these processes were applied inconsistently across products. Some development teams were still saying that they didn’t have time to incorporate all of the UCD team’s recommendations. Users were still complaining that products were too technology driven and that the software was hard to learn and use. There were also numerous places in the products where the UCD teams had identified missing functionality that could greatly improve user productivity, efficiency, and satisfaction.

While we did not consciously use the Usability Maturity Model levels in our thinking at the time, the push to function fully at the next level—Integrated—was a key factor in our overall success.

At the Integrated level, the following things occur:

• UCD processes are integrated with other processes in the development cycle, and UCD requirements are communicated in a way that developers understand and can implement.
• UCD processes are used to inform changes to the developing products in a timely manner, and these changes are implemented.
• Design and design solutions are treated iteratively, and goals for scope and quality are clearly stated and adhered to.

In the beginning, the technique that our organization adopted was to push harder on the UCD front. We knew that the teams needed more UCD to improve their products; however, the teams resisted the move toward this more unified approach to UCD, and we were having trouble understanding why. After all, everyone was saying that we needed to improve the products.

At this discouraging moment, we decided to use some of our headcount to hire program managers and developers. We had seen that one issue our UCD teams faced was employees spending too much time on process management or technical problems—not their area of expertise. We reasoned that by hiring product managers and developers we could free up our UCD people to focus on their specialty. This assumption proved true, but the greater value came when these new team members opened-up a way for us to reshape how our UCD organization functioned within the larger process.

Learning More about Our Internal Audience

We used the ethnographers in our organization who normally worked with users and customers to conduct an internal ethnographic study to help us better understand what our product management and development counterparts thought about UCD, how our team functioned, and  how UCD could help them make better products. From this we learned three key things:

• Product managers and developers wanted a stake in the research and design process. They wanted to know that they were contributing meaningfully to the UI design. They wanted the solutions to match the requirements and to understand how the UCD team arrived at these requirements.
• Product managers and developers wanted all user-experience guidelines, standards, and specifications to be in sync with delivered technology. Previously, syncing user-experience design standards and technology had been somewhat of a moving target, with UI designs running ahead of what was available in the technology. Developers wanted to build what they saw in prototypes and guidelines.
• Developer productivity was key; we scoured the bug database for the most annoying, time consuming, and frequently logged UI bugs. We found more than 1,000 bugs showing that developers were not inserting the key notation because it had to be coded separately. Consider that it takes two to three hours to file and fix a bug. To optimize their process, we identified that placement of the key notation string should happen automatically when a field was marked as required, thus saving hours of extra coding or bug filing and fixing and improving overall quality and consistency. By automating the process, we enabled developers to focus on building the parts of the product that would differentiate it from our competitors.

Effect of Our Program Managers and Developers

The new program managers on our team helped us understand what we needed to do to become fully integrated with the formal development process. Because the development organization was redefining this process in conjunction with a major release, the timing was ideal. In the beginning, a lot of what we had to do was explain our process and requirements to the team responsible for defining the new development process. Once others began to understand our role and include us in the process, the tables turned. We had to take a critical look at the quality and consistency of our deliverables to ensure that they were presented in a way that all audiences could understand, and addressed the concerns of all stakeholders. We also had to make sure that all members of our team understood how to create these deliverables consistently.

Because the development project included a new user interface for every product, our team would not be able to do all the work. We needed to document the mandatory user experience deliverables in such a way that product managers could complete them as well. Had we not done the prior ethnographic work, it would have been much more difficult to do this well.

Even with our up-front ethnographic work, we had to iterate the user experience part of the process a few times before ultimately defining a three-tiered support structure to which all teams had to adhere.

We also began to define standards for our team to adhere to, including properly using the existing bug filing and tracking process for all user experience issues. When there were issues on which we disagreed with the design direction or approach of a particular team, the development process itself ensured that we discussed the issues and resolved them appropriately and on time. We made our process visible to other functions using the same tracking and metrics as the rest of the organization.

Our program managers then helped us translate our UCD process to our executives and other team members and then translated their language back to us, ensuring that everyone was clear on the process.

Figure 1 shows what the process evolved to over time. We were instrumental in generating this diagram, which became an effective teaching tool for all the teams involved.

Our developer and UCD architect team members participated as members of functional and technical working groups. They helped us understand how the UI would be delivered and what we needed to provide to the technical teams in order to ensure that we could build the UI that our users needed.

Their work also gave us the opportunity to influence technical direction. Historically, our thoughts on technical issues had been poorly received, taken as the agitations of clueless outsiders. Now that we were participating on the relevant working teams, however, our input was taken seriously. We facilitated communication and cooperation among development, strategy, and product management and became, in many cases, the main conduit that led to the understanding of the solutions.

As a result of these actions, we have successfully carried out the UCD process and avoided wasting time and effort justifying our process and resolving inter-organizational conflicts. The company now better understands the value and use of UCD, and UCD has a voice and a role to play at all phases of the process. Most importantly, the design of the application suite itself now provides a significantly more integrated, satisfying user experience.

Ensuring that UCD Provides Maximum Value

Get Top-Level Commitment to UCD

Top-level commitment to UCD means that the organization recognizes that “usability” and UCD are strategic elements of the product offering. The organization may not understand all of the implications of applying these elements, but it knows enough to make an investment. Top-level commitment is demonstrated in three ways:

• Executive buy-in: Building executive buy-in takes time, unless you have a sponsor who comes in with previous successful experience with UCD. Part of your job is to increase the trust that your executives have in UCD through successful outcomes with more and more teams. The greater the number of people who have had a positive experience with UCD in an organization, the more momentum you can build around UCD.
• Adequate resources: An organization must appropriately budget for customer research, labs, test subjects, and staff for the number of necessary projects. If the major features of a product have not gone through some form of UCD, then the user experience will reflect the assumptions of the product builder, not the product users.
• Positioning of the UCD team: The UCD team must be positioned appropriately within the company to be effective. Both centralized and decentralized models can work. Our experience with an enterprise application suite is that the UCD team should be located in the development organization, as they are then seen to have the same level of buy-in as other members. As a centralized team, the UCD team can leverage expertise across the team, easily handle cross-suite issues, and load balance as needed.

Integrate with the Process

Integration with the process means that UCD is not a side activity whose outcomes may affect the product only if time permits at the end of the cycle. UCD is a full partner in the development process with all of the other players: strategy, development, product management, testing, and so on.

To ensure integration of UCD in the development process, a company must:

• Include UCD milestones and gating factors in its development process, standardize deliverables, and ensure that all stakeholders understand these deliverables.
• Hire the appropriate staff to support the total effort. In order to integrate with the process, you must have people on your team who understand the perspectives and challenges of the other members of the organization. You must go beyond UCD disciplines to include program management, developers, architects, and marketing specialists.

Conclusion

Instead of attempting to directly convert our organization to the language of UCD, we learned to communicate in the terms of product managers and developers by incorporating their disciplines on our team. We introduced by example more detailed aspects of UCD as a normal and accepted part of the process. As a result, UCD is well-represented in all stages of our process, from strategy, to planning, to execution. As we look forward, we see our organization beginning to function at the Institutionalized level, where the organization as a whole is human-centered in its focus.