Usability maturity models are methods for developing user-centered design processes in companies in order to facilitate usability methodologies for creating usable products. Usability maturity models help management understand the issues surrounding organizational opportunities and to improve the usability of its products. The models also benefit usability practitioners by pinpointing areas of improvement in usability processes and practices.
A maturity model is composed of two main elements:
- A set of organizational areas, such as development practices, processes, infrastructure, and skills
- The capability maturity levels for rating each of the areas
A usability maturity model can be regarded as an ideal model of user-centered design. The closer the company meets the ideal level of usability maturity, the higher rating it gets.
A common method of utilizing a usability maturity model is to perform an assessment of the company’s current capacity for developing usable products.
In a usability maturity assessment, an assessment team analyzes organizational areas by examining documentation and interviewing key stakeholders. The team maps the findings against a usability maturity model and determines the ratings.
Four Model Categories
The first usability maturity models were Trillium by Bell Canada (a general maturity model including usability engineering), Usability Leadership Maturity Model by IBM, HumanWare Process Assessment model by Philips, and User Centered Design Maturity by Loughborough University. All of these models were developed in the early 1990s.
In 2000, ISO 18529 was published based on work performed by European INUSE and TRUMP research projects. This model is remarkable in the sense that its format complies with the standard processes assessment model widely used in software engineering, such as CMM (Capability Maturity Model) and its revised version CMM-I. A further extended version is ISO 18152 which includes a larger set of processes.
Further developments for Usability Maturity Models include: Human-Centeredness Scale, DATech in Germany, SDOS in Japan, and KESSU in Finland. A more recent effort
is the Usability/User Experience maturity model as a result of a series of workshops at the UPA and HCII conferences.
Maturity models fall into four main categories:
- Standard process assessment models (ISO 18259, ISO 18152) use the format of the process assessment models used in software engineering.
- Non-standard models (Trillium, Philips, KESSU) examine processes, but with non-standard approaches.
- Generic models (ULMM, UCDM, UMM-HCS, DATech UEPA, Standardized Usability/User-Experience) include process aspects, but also larger issues such as management awareness, skills, and organizational position on usability.
- Specific models (HCD-PCM visioning) which have a limited focus.
The models are summarized in Table 1.
[greybox]
| Model | Origin | Category | Documentation | Empirical Research |
| Trillium | Bell Canada | Non-standard process | Relatively detailed | No research results reported |
| Usability Leadership Management Maturity (ULMM) | IBM (US) | Generic | Limited | No research results reported |
| HumanWare Process Assessment (HPA) | Philips (Netherlands) | Non-standard process | Limited | No research results reported |
| User Centered Design Maturity (UCDM) | HUSAT Research Institute (UK) | Generic | Limited | No research results reported |
| Usability Maturity Model: Human-Centeredness Scale (UMM-HCS) | European INUSE & TRUMP projects | Generic | Rather detailed | Developed iteratively based on trials |
| ISO 18529 | European INUSE & TRUMP projects | Standard process | Detailed guidance and training available | Developed iteratively based on trials and expert review |
| ISO 18252 | European INUSE & TRUMP projects | Standard process | Detailed guidance and training available | No research results reported |
| KESSU | Oulu University (Finland) | Non-standard process | Rather detailed | Developed iteratively based on empirical findings |
| DATech-UEPA | DATech (Germany) | Specific | Rather detailed (in German) | No research results reported |
| HCD-PCM design | Mitsubishi Research Institute, NTT, Otaru University of Commerce (Japan) | Standard process | In Japanese | No research results reported |
| HCD-PCM visioning | Mitsubishi Research Institute, NTT, Otaru University of Commerce (Japan) | Specific | In Japanese | No research results reported |
| Draft Standardized Usability/User-Experience Maturity Model | Workshop on user-experience maturity modeling, HCII 2009 | Generic | Limited | No research results reported |
Table 1. (top) Usability/UX maturity models (in rough chronological order). [/greybox]
[greybox]
| Level of Capability | Research |
| Level 5: Optimizing | Organization can reliably tailor the process to particular requirements. |
| Level 4: Predictable | Performance of the process is within predicted resource and quality limits. |
| Level 3: Established | Process is carried out in a manner specified by the organization and the resources are defined. |
| Level 2: Managed | Quality, time, and resource requirements for the process are known and controlled. |
| Level 1: Performed | Process achieves its purpose. Individuals carry out processes. |
| Level 0: Incomplete | Organization is not able to carry out the process. |
Table 2. (medium) Capability levels of standard processes assessment models. [/greybox]
Standard Process Assessment Models
As an example of standard process assessment models, ISO 18529 identifies seven processes, such as Context of Use Definition, Usability Requirements, and Usability Evaluation.
The capability levels range from Level 0 (incomplete) to Level 5 (optimizing) as defined in Table 2.
The result of an assessment is a capability profile as shown in Figure 1. Each process is rated separately: the higher the rating, the higher the capability of the process. Different processes may be at different levels of capability.
The process assessment model includes strict criteria for each level of capability. One should note, however, that an assessment is not a mechanical task. The lead assessor, in the end, makes final ratings through professional judgments. Therefore, the assessors should be experienced professionals.
What is Included in Usability Capability?
Different maturity models cover different organizational areas:
Performance of usability processes means examination of the extent to which usability activities—such as user analysis, task analysis, usability requirements determination, and usability evaluations—are carried out. Performance of usability processes is a basic area. If there is any effective usability in an organization, it should be visible in development projects. Practically all maturity models address this to some extent.
Requirements include:
- Existing competing products are assessed (Trillium)
- Mock-ups and prototypes are produced and evaluated (HPA)
- Understand and specify the Context of Use (UMM-P)
- KESSU has a specific focus on the quality and impact of the activities, stating: “usability activities are carried out professionally,” and “usability results have impact on design decisions.”
Management of usability processes in development projects means examination of issues, such as the inclusion of usability engineering activities in a project plan, follow-up of the implementation of the plan during the project, and configuration management of the documents produced. Addressing these issues decreases the potential for problems with scheduling, resources, and document control.
Usability in a quality management system is addressed in many models. The standard process assessment models (capability level 3) examine this issue. Requirements by other models include:
- User documentation is developed formally (Trillium)
- Existing procedures are extended to include HumanWare activities (HPA)
- Integrate HF processes with project processes (UMM-HCS)
- Usability is part of the Quality Policy and the Quality Handbook (DATech-UEPA).
Systematic improvement of usability processes examples include statements such as “Management’s actions to improve the current focus on usability,” (ULMM) and “Systematic improvement of quality in use,” (UMM-HCS).
The role of usability at a strategic level is explicitly addressed in some models:
- Activities at all organizational levels to insure a prominent focus on usability (ULMM)
- A coordinator drives the policy on HumanWare throughout an organization (HPA)
- Human-system issues in business strategy
(ISO 18152)
Statements on usability skills include:
- Usability engineer is educated, experienced, and certified in usability analysis, evaluation, and prototyping (DATech-UEPA)
- Development of appropriate skills in the human-centered staff (UMM-HCS).
The impact of usability is also addressed:
- HCI skills and tasks are viewed as important (ULMM)
- Acceptance of human-centered skills by the organization (UMM-HCS)
- Usability engineer is part of the design team and is responsible for design decisions (DAtech-UEPA)
- Skilled staff is involved and effective in all stages of development (ULMM)
The requirement “resources available for usability work,” (ULMM) addresses usability resources.
Organizational culture is also covered by some models. Statements include:
- Awareness at all organizational levels of the importance of usability in product development (ULMM)
- The degree to which the organization has a human-centered approach to its work and working culture. Staff members are made aware that end users’ skills, background, and motivation may differ from developers or system support staff (UMM-HCS).
Which Model to Apply
Standard process assessment models are well-established and well-documented. Established schemes and training exist for assessors. However, the limitation of the models is that they tend to focus too heavily on process management aspects, such as the inclusion of usability activities in a project plan, follow-up of the implementation of the plan during the project, and configuration management of the documents produced. The substance of usability should have higher consideration with less focus on processes alone.
The available documentation of ULMM, HPA, and UCDM is very limited, making these models difficult to use by anyone other than their developers. Practical implementation requires an extensive amount of interpretation which can lead to results that the creators of the models didn’t initially anticipate.
UMM-HCS, DATech-UEPA, Trillium, and KESSU are better documented, although not to the level of standard process assessment. The origin of KESSU was standard process assessment, but was developed for detailed analysis of the usability processes. UMM-HCS is for rough assessment, gaining an overall picture. Trillium is rather old, and the German language limits the use of DATech-UEPA. Language also limits the use of the Japanese models.
Most of the publications focus on describing the usability maturity models. Very little research is devoted to studying the real-life validity of the different models. Empirical research is reported on the development of ISO 18529 and KESSU. It is difficult to find any empirical research reports on the other models.
In summary, the maturity of usability maturity models is rather low. Standard process assessment is a more mature approach, but does not have the appropriate focus on usability in most cases. More empirical evidence on the real life validity and usefulness of different usability maturity models is needed, and concrete guidance of the existing UCM models is very limited.
An effective step to start improvements in usability capability is to carry out a usability maturity assessment. There are different usability maturity models that address different organizational viewpoints: usability processes, quality system, usability business strategy, awareness of usability, action improvements, usability skills and resources, and organizational position of usability.
To make to models more useful and valid, their documentation should be more detailed, and more empirical research needs to be carried out.
