Interaction with software applications is an integral part of our lives today. The quality of interaction is influenced by multiple factors such as usability of the interface, branding, how different the UI is from all others, and whether the UI “feels” delightful. An end-user compares their experience with similar applications they have used in the past. While a user’s preference is obvious to them, we have yet to come across an objective, globally acknowledged scale to stack-up and compare a product’s UX maturity in the marketplace.
Many of us are familiar with these commonly used means of usability assessment:
- Questionnaire for User Interface Satisfaction (QUIS), Kent Norman, 1988
- System Usability Scale (SUS), John Brooke, 1996
- Computer System Usability Questionnaire (CSUQ), Lewis, 1995
- Heuristic Evaluation
- Usability Testing
These methods provide a good measure of usability, and to some extent, the user satisfaction of an application. However, none of the assessments lends itself as a singular benchmark of the overall user experience of the application.
There are models that assess the maturity of user experience capability of an organization, such as Certified Practice in Usability™ (CPU) by Human Factors International. However, maturity of user experience as a product attribute itself would mean benchmarking its user experience on a scale of levels across a continuum. How can we do that?
At Tata Consulting Services, we are part of a group responsible for ensuring that our products meet the highest benchmarks of user experience. Due to the lack of established benchmarks, we developed the User Experience Maturity Model (UXMM). With UXMM, we measure the overall user experience associated with a software product or application against four distinct ascending levels of maturity:
- Level 1: Usable products have been designed in accordance with basic design principles and best practices. These products offer a fairly good user experience with emphasis on consistency and standards. However, they lack any nuances that could be worked into design based on context of use.
- Level 2: Useful products are those that enable users to accomplish their tasks and goals with ease, efficiency, and accuracy. These products are designed taking the end users and context of use into consideration.
- Level 3: Desirable products are market leaders. They provide a relatively better user experience when compared to current competitors, and this inherently gives them an edge in the market, other factors remaining the same.
- Level 4: Delightful products are simply awesome! Products that delight have the potential to create a cult following and inspire an exceptional sense of customer loyalty.
The basis of the UXMM is a set of key UX parameters (KUXPs), attributes that influence the user experience of software applications. These are:
- Ease of Use. Is the user interface intuitive enough to be used without external aids such as the user manual or human intervention?
- Speed of Use. Does the user interface facilitate a smooth and seamless interaction enabling users to accomplish goals in a minimal timeframe?
- Learnability. Does the user interface adhere to predictable visual and interaction patterns to reduce learning curve?
- Consistency. Do common elements work and behave in a consistent manner?
- Content. Does the user interface communicate information and address users appropriately?
- Accessibility. Does the interface follow basic accessibility guidelines and best practices?
- Flexibility. Does the interface use inbuilt intelligence to reduce manual errors for novice to expert users?
- Aesthetics. Is the user interface designed for readability and legibility and appropriate look and feel?
- Recovery from Errors. Does the system aid the user in rapid recovery from errors?
- Help. Does the system provide timely help? Is help easily accessible?
- Brand Recall. Is the overall experience on the site in line with the projected brand of the product?
- Persuasiveness. Does the product aid and encourage the users to perform certain actions that may not be primary user goals but increase the engagement factor?
- Differentiation. Does the product communicate a distinct advantage over competition? Examples include unique selling point, value added features, or integration across multiple devices.
- Greater Good. Does the product contribute to a “greater good” in the course of using the product, thus creating an emotional connection with end users?
Each assessment level includes several KUXPs and an associated assessment mechanism. The evaluation score at each level determines if the product is ready to be evaluated for the next tier.
Level 1 – Usable, is assessed through an expert review based on the first ten KUXPs. A UX practitioner reviews the application for compliance to the defined heuristics based on how well and how consistently it meets each review criterion. Actual context of usage is not considered for Level 1 assessment.
Level 2 – Useful looks at whether users are able to solve problems, complete tasks, and attain goals using this product. All Level 1 KUXPs are considered for Level 2 as well. Usefulness is evaluated through usability tests. Users are observed during the tests and parameters—such as speed of task completion and number of assists required—are recorded. Users are encouraged to “think aloud” and user comments are noted.
Level 3 – Desirable evaluates whether the product has a richer user experience when compared with peers. The desirability of the product is evaluated by competitor benchmarking.
The competitors considered for benchmarking could be direct competitors from the same or similar domains, or products from different domains that provide similar services. For example, a digital camera interface could be compared with a mobile phone while evaluating the goal of “clicking digital photos”.
Competitor benchmarking includes a KUXP based competitive analysis and usability tests with competitor products. The KUXPs considered for assessment include all those applicable for Level 1 and 2, plus:
- Brand recall
- Persuasiveness
- Differentiation
- Greater good
Level 4 – Delightful is the highest level of UX maturity. The quality of emotional responses elicited by a product is a measure of its delightfulness. A product may use the best hardware and be packed with the rich features and cutting edge technology, but if using it causes frustration and agony it will definitely fail in the market. The KUXPs considered for Level 4 are the same as Level 3.
Here again, usability testing is used This time, we observe the emotional responses of users rather than focusing on task completion.
The emotional responses are evaluated using tests such as the Faces Questionnaire (Benedek & Miner, presented at UXPA 2002), Word Matching, or Emotion Heuristics (de Lera & Garreta-Domingo, presented at BHCI 2007).
Defining Evaluation Benchmarks
The output of a UXMM assessment is a numeric score between 0 and 100. Higher scores indicate greater UX quality of the product.
A minimum required benchmark score is also generated based on the context of the application. The actual score is compared with the benchmark score to determine if the application passes or fails that assessment level.
The benchmark score is calculated through a benchmarking exercise based on a predefined questionnaire to be filled for the application that is being assessed.
Benefits of UXMM
We have used the UXMM for more than a hundred UX assessments over the last two years. The key benefit is obvious: UXMM is a reliable methodology of quantifying and benchmarking the degree of user experience of a product. It also delivers the following positive outcomes:
- The developer community welcomes the objective nature and ability to pinpoint specific areas of improvement. This speeds remedial measures and saves a lot of unnecessary back and forth.
- Using a standardized assessment mechanism helps establish transparency and credibility for UX as a practice, as well as the UX group itself.
- After a couple of assessments, product teams are better able to appreciate the importance of engaging UX teams right from the requirement-gathering phase. Due to this, we have been able to institutionalize robust requirement gathering and design methodologies in the product development lifecycle smoothly, and diffuse many of the traditional conflicts of interest between UX and development teams. The table below (Figure 3) shows how each assessment level is tied to UX activities to be conducted in earlier stages of product development for a better user experience.
- It becomes easier to define a tangible UX improvement roadmap. Product management teams begin aspiring for higher maturity levels, consequently improving the product quality in an incremental manner.
- The assessment criterion for each level has been designed in such a way that higher maturity levels require deeper engagement during early stages of product development.
Assessment level are tied to UX activities in order to better develop the user experience.
| Level | Associated UX, Development, and Business Activities |
|---|---|
| 1 – Usable | User interface design and development in accordance with established user experience heuristics. A part of the product development team should consist of user experience experts entrusted with the task of designing, developing, and testing products |
| 2 – Useful | Early involvement of end users in the product design. Iterative design based on user feedback by testing product using wireframes and mockups of varying fidelity: paper prototypes, electronic wireframes, visual mockups, and working HTML prototypes. |
| 3 – Desirable | Clearly defined strategy and market positioning. Competitor benchmarking to understand existing offerings. Workshops and focus groups with stakeholders, SMEs, and branding team to determine the unique selling point (USP). |
| 4 – Delightful | Ethnographic research methods, personas to understand users, their functional and life goals. Tightly integrated user centered design methodology with the product development lifecycle. |
We encourage you to try out the UXMM-based assessment. We are considering making the assessment framework available publicly and look forward to your feedback.
[bluebox]
More Reading
Persuasive Technology: Using Computers to Change What We Think and Do, B.J. Fogg
Usability Engineering, Jakob Nielsen
Corporate Usability Maturity. Jakob Nielsen’s Alertbox, April 24, 2006
[/bluebox]
[greybox]
Text from Figure 1 – Levels of UXMM
Usable (image: Swiss Army knife) – Provides for features that make it technically useful. However, lack of focus on the end-users leads to a one-dimensional interpretation of usability and people may not use it that much. A Swiss Army knife maps to all attributes of a knife, but is not particularly usable for everyday use in the kitchen.
Useful (image: toothpick) – Consumers find the product useful enough to meet end goals. It has been created keeping end-users in mind. However, it lacks the differentiator to take it to the next level. One toothpick is as good as another. One needs to look beyond basic usability to make it indispensible.
Desirable (image: models of cars) – Product is a market leader. Has an edge over competitors. Customers will choose this product over others, even if it means compromising on cost or features.
Delightful (image: designer chair) – Simply awesome! Cult following. Strong brand loyalty. Customers will choose this product due to consistently high quality of user experience associated with the product or brand.
[/greybox]用户体验成熟度模型(UXMM)是一种对用户体验进行评估和评分的方法，评估的范围是人们熟悉的，从只有基本可用性的不成熟的体验，到成熟的给用户带来愉悦的体验。文章介绍了这种模型和方法的使用，以及在组织中建立UXMM的益处。
文章全文为英文版UX 성숙도 모형은 사용자경험을 우리가 익숙한 연속형 축에서 설명하는 모형으로, 가장 낮은 수준의 기본 사용성에서부터 가장 높은 수준의 즐거움을 축으로 한다. 조직에서 이러한 UX성숙도 모형을 사용하는 방법과 이를 사용했을 때의 장점을 서술한다.
전체 기사는 영어로만 제공됩니다.
O Modelo de Maturidade da Experiência do Usuário (UXMM) é uma técnica para avaliação e mensuração da experiência dentro de um contínuo de familiaridade, com experiências ‘imaturas’ possibilitando uma usabilidade mínima e experiências ‘maduras’ conquistando usuários. O modelo e técnicas de uso são descritas, e os benefícios de estabelecimento do UXMM dentro de uma organização são transmitidos.
O artigo completo está disponível somente em inglês.UX成熟度モデル(UXMM)は、基礎となるユーザビリティを提供する未成熟な経験のレベルから、ユーザを嬉しくさせる成熟した経験までを評価し、その結果を良く知られた連続体の上に位置づけるものだ。そのモデルや利用技法について説明を行い、組織にUXMMを確立する利点について述べる。
原文は英語だけになりますEl Modelo de Madurez de la Experiencia de Usuario (MMUX) es una técnica para evaluar y puntuar una experiencia en un continuo familiar, mientras que las experiencias inmaduras representan la usabilidad base y las experiencias maduras a los usuarios encantados. El modelo y las técnicas usadas son descritos y los beneficios en la aplicación del MMUX en las organizaciones son ennumerados.
La versión completa de este artículo está sólo disponible en inglés
