Interaction with software applications is an integral part of our lives today. The quality of interaction is influenced by multiple factors such as usability of the interface, branding, how different the UI is from all others, and whether the UI “feels” delightful. An end-user compares their experience with similar applications they have used in the past. While a user’s preference is obvious to them, we have yet to come across an objective, globally acknowledged scale to stack-up and compare a product’s UX maturity in the marketplace.
Many of us are familiar with these commonly used means of usability assessment:
- Questionnaire for User Interface Satisfaction (QUIS), Kent Norman, 1988
- System Usability Scale (SUS), John Brooke, 1996
- Computer System Usability Questionnaire (CSUQ), Lewis, 1995
- Heuristic Evaluation
- Usability Testing
These methods provide a good measure of usability, and to some extent, the user satisfaction of an application. However, none of the assessments lends itself as a singular benchmark of the overall user experience of the application.
There are models that assess the maturity of user experience capability of an organization, such as Certified Practice in Usability™ (CPU) by Human Factors International. However, maturity of user experience as a product attribute itself would mean benchmarking its user experience on a scale of levels across a continuum. How can we do that?
At Tata Consulting Services, we are part of a group responsible for ensuring that our products meet the highest benchmarks of user experience. Due to the lack of established benchmarks, we developed the User Experience Maturity Model (UXMM). With UXMM, we measure the overall user experience associated with a software product or application against four distinct ascending levels of maturity:
- Level 1: Usable products have been designed in accordance with basic design principles and best practices. These products offer a fairly good user experience with emphasis on consistency and standards. However, they lack any nuances that could be worked into design based on context of use.
- Level 2: Useful products are those that enable users to accomplish their tasks and goals with ease, efficiency, and accuracy. These products are designed taking the end users and context of use into consideration.
- Level 3: Desirable products are market leaders. They provide a relatively better user experience when compared to current competitors, and this inherently gives them an edge in the market, other factors remaining the same.
- Level 4: Delightful products are simply awesome! Products that delight have the potential to create a cult following and inspire an exceptional sense of customer loyalty.
Figure 1: User Experience Maturity Levels. (Full text in this image at the end of the article)
The basis of the UXMM is a set of key UX parameters (KUXPs), attributes that influence the user experience of software applications. These are:
- Ease of Use. Is the user interface intuitive enough to be used without external aids such as the user manual or human intervention?
- Speed of Use. Does the user interface facilitate a smooth and seamless interaction enabling users to accomplish goals in a minimal timeframe?
- Learnability. Does the user interface adhere to predictable visual and interaction patterns to reduce learning curve?
- Consistency. Do common elements work and behave in a consistent manner?
- Content. Does the user interface communicate information and address users appropriately?
- Accessibility. Does the interface follow basic accessibility guidelines and best practices?
- Flexibility. Does the interface use inbuilt intelligence to reduce manual errors for novice to expert users?
- Aesthetics. Is the user interface designed for readability and legibility and appropriate look and feel?
- Recovery from Errors. Does the system aid the user in rapid recovery from errors?
- Help. Does the system provide timely help? Is help easily accessible?
- Brand Recall. Is the overall experience on the site in line with the projected brand of the product?
- Persuasiveness. Does the product aid and encourage the users to perform certain actions that may not be primary user goals but increase the engagement factor?
- Differentiation. Does the product communicate a distinct advantage over competition? Examples include unique selling point, value added features, or integration across multiple devices.
- Greater Good. Does the product contribute to a “greater good” in the course of using the product, thus creating an emotional connection with end users?
Each assessment level includes several KUXPs and an associated assessment mechanism. The evaluation score at each level determines if the product is ready to be evaluated for the next tier.
Figure 2: User Experience Maturity Pyramid
Level 1 – Usable, is assessed through an expert review based on the first ten KUXPs. A UX practitioner reviews the application for compliance to the defined heuristics based on how well and how consistently it meets each review criterion. Actual context of usage is not considered for Level 1 assessment.
Level 2 – Useful looks at whether users are able to solve problems, complete tasks, and attain goals using this product. All Level 1 KUXPs are considered for Level 2 as well. Usefulness is evaluated through usability tests. Users are observed during the tests and parameters—such as speed of task completion and number of assists required—are recorded. Users are encouraged to “think aloud” and user comments are noted.
Level 3 – Desirable evaluates whether the product has a richer user experience when compared with peers. The desirability of the product is evaluated by competitor benchmarking.
The competitors considered for benchmarking could be direct competitors from the same or similar domains, or products from different domains that provide similar services. For example, a digital camera interface could be compared with a mobile phone while evaluating the goal of “clicking digital photos”.
Competitor benchmarking includes a KUXP based competitive analysis and usability tests with competitor products. The KUXPs considered for assessment include all those applicable for Level 1 and 2, plus:
- Brand recall
- Persuasiveness
- Differentiation
- Greater good
Level 4 – Delightful is the highest level of UX maturity. The quality of emotional responses elicited by a product is a measure of its delightfulness. A product may use the best hardware and be packed with the rich features and cutting edge technology, but if using it causes frustration and agony it will definitely fail in the market. The KUXPs considered for Level 4 are the same as Level 3.
Here again, usability testing is used This time, we observe the emotional responses of users rather than focusing on task completion.
The emotional responses are evaluated using tests such as the Faces Questionnaire (Benedek & Miner, presented at UXPA 2002), Word Matching, or Emotion Heuristics (de Lera & Garreta-Domingo, presented at BHCI 2007).
Defining Evaluation Benchmarks
The output of a UXMM assessment is a numeric score between 0 and 100. Higher scores indicate greater UX quality of the product.
A minimum required benchmark score is also generated based on the context of the application. The actual score is compared with the benchmark score to determine if the application passes or fails that assessment level.
The benchmark score is calculated through a benchmarking exercise based on a predefined questionnaire to be filled for the application that is being assessed.
Benefits of UXMM
We have used the UXMM for more than a hundred UX assessments over the last two years. The key benefit is obvious: UXMM is a reliable methodology of quantifying and benchmarking the degree of user experience of a product. It also delivers the following positive outcomes:
- The developer community welcomes the objective nature and ability to pinpoint specific areas of improvement. This speeds remedial measures and saves a lot of unnecessary back and forth.
- Using a standardized assessment mechanism helps establish transparency and credibility for UX as a practice, as well as the UX group itself.
- After a couple of assessments, product teams are better able to appreciate the importance of engaging UX teams right from the requirement-gathering phase. Due to this, we have been able to institutionalize robust requirement gathering and design methodologies in the product development lifecycle smoothly, and diffuse many of the traditional conflicts of interest between UX and development teams. The table below (Figure 3) shows how each assessment level is tied to UX activities to be conducted in earlier stages of product development for a better user experience.
- It becomes easier to define a tangible UX improvement roadmap. Product management teams begin aspiring for higher maturity levels, consequently improving the product quality in an incremental manner.
- The assessment criterion for each level has been designed in such a way that higher maturity levels require deeper engagement during early stages of product development.
Assessment level are tied to UX activities in order to better develop the user experience.
| Level | Associated UX, Development, and Business Activities |
|---|---|
| 1 – Usable | User interface design and development in accordance with established user experience heuristics. A part of the product development team should consist of user experience experts entrusted with the task of designing, developing, and testing products |
| 2 – Useful | Early involvement of end users in the product design. Iterative design based on user feedback by testing product using wireframes and mockups of varying fidelity: paper prototypes, electronic wireframes, visual mockups, and working HTML prototypes. |
| 3 – Desirable | Clearly defined strategy and market positioning. Competitor benchmarking to understand existing offerings. Workshops and focus groups with stakeholders, SMEs, and branding team to determine the unique selling point (USP). |
| 4 – Delightful | Ethnographic research methods, personas to understand users, their functional and life goals. Tightly integrated user centered design methodology with the product development lifecycle. |
We encourage you to try out the UXMM-based assessment. We are considering making the assessment framework available publicly and look forward to your feedback.
More Reading
Persuasive Technology: Using Computers to Change What We Think and Do, B.J. Fogg
Usability Engineering, Jakob Nielsen
Corporate Usability Maturity. Jakob Nielsen’s Alertbox, April 24, 2006
Text from Figure 1 – Levels of UXMM
Usable (image: Swiss Army knife) – Provides for features that make it technically useful. However, lack of focus on the end-users leads to a one-dimensional interpretation of usability and people may not use it that much. A Swiss Army knife maps to all attributes of a knife, but is not particularly usable for everyday use in the kitchen.
Useful (image: toothpick) – Consumers find the product useful enough to meet end goals. It has been created keeping end-users in mind. However, it lacks the differentiator to take it to the next level. One toothpick is as good as another. One needs to look beyond basic usability to make it indispensible.
Desirable (image: models of cars) – Product is a market leader. Has an edge over competitors. Customers will choose this product over others, even if it means compromising on cost or features.
Delightful (image: designer chair) – Simply awesome! Cult following. Strong brand loyalty. Customers will choose this product due to consistently high quality of user experience associated with the product or brand.
Retrieved from http://uxpamagazine.org/ux-maturity-model/
Comments are closed.
