In early 2014, we found ourselves working for an organization with a large product portfolio built up over decades of acquisitions. To create a cohesive branded experience, it became clear that many of the products we were suddenly supporting existed at very different levels of UX maturity. Our challenge was to create a comparable scorecard for an array of products across UX frameworks, usability maturity, and technology platforms. Our first, somewhat obvious step, was to leverage existing UX maturity models to build a custom framework for our organization, following the advice of Rich Buttiglieri in his presentation on maturity models at UXPA Boston 2012. The model we built focused on the following for each level:
- What will the user see and experience?
- How will the experience impact brand perception?
- What level of UX process support is needed?
- How would product management, as a whole, react towards the framework proposed?
Table 1. Measures of Experiential Maturity
| What does this type of experience look like? | |
| Ad-Hoc | Fragmented & inconsistent experienceDisconnected from changing user needs
Measure satisfaction, but not experience quality |
| Intentional | Conducts customer research, but it does not drive the experienceCulture uses data to drive experience decisions |
| By Design | Uses structured research to understand needs by segment and designs directly address needs.Specify the differentiated experience the organization wants to deliver |
| Integrated | Identify operating model changes required to deliver that experienceMeasure and reinforce desired behaviors across the ecosystem (internal and external)
UX is fully integrated with the product lifecycle |
| UX-Driven | Engage customers, employees, etc. in collaborative innovationProvide a platform to personalize and configure experiences
Find ways to better ways to engage users by testing designs with live traffic |
| What is the resulting brand perception? | |
| Ad-Hoc | “This product/service works as programmed.” |
| Intentional | “This product/service is accurate & available.” |
| By Design | “This product/service is super easy to use and works like I think.” |
| Integrated | “This is a memorable product/service” |
| UX Driven | “Wow, this is fantastic!” |
| How does the organization think about UX? | |
| Ad-Hoc | Unintentional design“Users will be trained.” User will need training to use the product—usability is not self-evident. |
| Intentional | Starting to use data in design decisionsGetting to know users better through structured data |
| By Design | Activity-focused designUI designed by UX professionals |
| Integrated | User-centric design“In the field to study users” and UX metrics also used for product planning. |
| UX Driven | User-centric DNAData-driven UX is simply the way the organization learns and acts to transform its products/services. |
| What is the associated UX process maturity? | |
| Ad-Hoc | UX methods are not consistently applied and work is often outsourced.Heuristic reviews and limited usability testing, usually at process end or conducted on products already in production. |
| Intentional | Hire more staff to do more work; although volume increases, work remains more tactical.Heuristic reviews & limited usability testing performed at the beginning and end of the process.
Mock-ups and prototypes used to iteratively improve designs |
| By Design | Well-defined UX process, consistent quality and performanceContextual analysis and greater focus on managing ‘usefulness’
Discovery research (e.g. personas) are regularly used to make design decisions. |
| Integrated | User-centric design“In the field to study users.”
UX metrics used for product planning |
| UX Driven | Designs that perform in a predictable wayUX metrics are formalized to capture baseline vs. improvement
Competitive analysis, personas, and field research Quantitative studies (baseline and comparative) |
We began introducing our initial experience maturity model to leadership, both within the product organization and throughout IT. This proved invaluable in helping our internal stakeholders understand why our processes were a little different for each product. It also served as a communication tool to help our UXers understand how to relate to internal product management teams.
In addition to introducing our maturity model to key stakeholders, we also needed to provide senior leadership with an executive summary so they could make comparisons across products. This would allow them to gain insight across a large portfolio of varying user experiences and to identify where to prioritize UX in the organization.
We experimented with multiple methods to create this type of summary for products and kept returning to the idea of a UX debt scorecard. What if we could combine the concept of UX debt with our experiential maturity model? This could not only help us communicate across the organization on a level playing field, but would also provide us with a way to measure how products were decreasing their debt, thereby increasing their experience maturity.
In this article, we discuss UX debt as a way to measure how easy and intuitive a product is to use so we can use the UX debt score as a way to determine the organization’s level of experiential maturity. For example: when the Ad-Hoc UX Debt=0, the organization is ready move to Intentional UX. The same formula works for the move from each level, all the way to a UX-driven organization.
We are now calculating UX debt relative to the product’s current maturity level to provide senior leadership with a product-by-product comparative dashboard of UX debt specific to a given product’s maturity level.
Due to the organization’s numerous, diverse products, we created a calculation worksheet meant to consistently calculate UX debt across our portfolio. As we began to communicate this concept throughout the organization, it was apparent that this was an effective communication and planning tool across multiple stakeholders: sales, product teams, technology teams, and senior leadership.
Defining UX Debt
The term “technical debt” was coined by Ward Cunningham as a metaphor used to describe the increased cost of maintaining technology due to shortcuts taken during the development process. Joshua Kerievsky later extended the metaphor to user experience design using the term “User Experience (UX) Debt.” Kerievsky explained that, like technical debt, UX debt will eventually come due, usually in the form of less customer satisfaction and possible customer defects.
In an enterprise setting, studies show that the decision to accumulate technical debt is often made by non-technical stakeholders. In our current enterprise, we wondered if the same is true for UX debt? If we develop a practical way to measure UX debt for products in the enterprise, what would that look like and how would it be received?
From Theory to Practice
We were brought into a company that had previously tried to ”perform UX,” but teams were unable to make much visible progress across multiple diverse, distinct product lines. Non-UXers (largely product managers) were often in charge of making UX decisions. Across this wide variety of products, customer satisfaction scores were showing that user experience, along with a large UX debt, was hindering products’ usability and creating unhappy customers in the process.
How could a small UX team of five people begin to describe at a high level how more than 100 products compared to each other? How could we communicate to senior managers the amount of UX work that we might need to perform to improve them? We quickly shifted our conversations to whether the notion of UX debt could help not only prioritize our UX work, but could pave the way to communicate and develop a more data and user-centric approach to product development at an enterprise level. Our team was familiar with the term UX debt from a theoretical perspective, but we soon began to wonder if we could actually create a long-term UX debt calculation and track it over time for our products. We wanted to create a comparable scorecard across product lines, and to help explain to our senior leadership how much work needs to be done. This was the point where we began to develop the UX debt calculator spreadsheet.
Table 3. A summary of UX Debt by Maturity Level
| Ad-Hoc | Intentional | By Design | Integrated | |
| How to calculate UX debt | Heuristic analysis |
Heuristic analysis or usability data | Usability data | Usability data |
| Main type of UX debt to resolve | ConsistencyBranding
Baseline metrics |
Meets user needsFindable
Usable |
PersonalizedPersuasive
Accessible |
Persuasive |
| Common areas to examine | Map inconsistencies | Analyze product against known user needsLevel of consistency, based on UX standards | Personalization to user needsAlignment of current experience with organization’s visionary experience | Use of persuasive design elementsDesired user behaviors |
The UX Debt Calculator
UX debt can be calculated with data from either heuristic evaluations or prior usability testing.
Calculating UX debt with heuristic data
We noticed that many products have little or no formative usability data available; products are mostly at the Ad-Hoc level of maturity. To calculate UX debt in these instances, we use heuristic analysis. The UX team performs a heuristic evaluation of the product using the following categories: Findable, Accessible, Clear & Communicative, Useful, Credible, Controllable, Learnable, Overall Aesthetics, Persuasive, and whether the product is currently tracked using our web analytics tool.
In the Template-Heuristics tab in the calculator, the categories correspond to a heuristic. As we complete the evaluation of a product or service, we update the value of the Assessment field for each measure. This field controls the Actual Score and Score Percentage. Possible values for the measure include:
- Unusable: imperative to fix – Unusable
- Major issue: important to fix with high priority – Critical
- Minor issue: should be given low priority – Moderate
- Cosmetic issue only – Minor
- Meets Criteria No Issues (default value)
If the product has failed to take accessibility into account, we assign a severity of Major issue. In addition, if we do not currently track the product’s usage, do not complete formative and/or summative testing, and do not implement company branding, we assign a severity of Major issue. This helps drive a culture making product decisions based on user data and beginning to help “paydown” UX debt.
We use the fields Problems and Steps to Resolve to communicate what needs to be addressed in the product for it to move up in its UX maturity. Once all columns are completed, we calculate the current UX debt and any projected paydown.
Calculating debt with prior usability data
In our organization, usability tests were previously conducted on a few products at higher levels of UX maturity. Our initial proof of concept for our UX debt calculator was to translate these third-party studies into a standard debt score. This approach quickly became our preferred approach to calculating debt, as it takes direct user feedback into account. However, when there is no formative data (as was the case with many of our products at lower levels of UX maturity), we default to heuristic evaluation to calculate UX debt.
Next Steps
This first phase of the UX debt calculator has been useful in helping us assess products. As a small UX team intending to create a branded experience across over 100 products, we must first determine where that product lies in our experiential maturity model and, second, calculate a debt score. As we began to use this calculator for many of our products, we discovered where our gaps lie in terms of user data, user experience, and customer satisfaction. From this vantage point, we can help product teams move the product forward in its UX maturity, as well as lower its long-standing UX debt.
As we move forward with these concepts across our organization, we expect that our company’s products will advance in their UX maturity and, consequently, pay down their debt. The calculator is a great tool for modeling this CX/UX transition in a product and reinforcing a user- and data-centric product culture.
Download the UX debt calculator
[bluebox]
More Reading
- Technical Debt – Wikipedia
- An Enterprise Perspective on Technical Debt – MTD ’11 Proceedings of the 2nd Workshop on Managing Technical Debt.
- “To Pay or Not to Pay Technical Debt Software” by Andrew Wright, IEEE (2013 Volume: 28 , Issue: 6 ).
- “Applied UX Strategy” by Yury Vetrov (UX Matters)
- “How UX Evolves at Companies: A New Look at Maturity Models” by Rich Buttiglieri
[/bluebox]
创建一个用户体验缺陷计算器，能够帮助您确定在哪些方面需要优先安排用户体验工作。将用户体验缺陷概念与用户体验成熟度模型相结合，可以跨越不同的用户体验框架、可用性成熟度和技术平台成功评估多种产品。创建可比较各种不同产品的记分卡，不仅有助于高级管理层了解需要执行的工作量，还可帮助产品团队更快地提高产品的用户体验成熟度，减少每种产品长期存在的用户体验缺陷。
UX 부담 계산기를 만들면 UX를 위해 우선적으로 노력해야 할 부분을 손쉽게 알아낼 수 있습니다. UX 부담 개념을 사용자 경험 성숙 모델과 병합하면 다양한 제품을 UX 프레임워크, 사용성 성숙도, 테크놀로지 플랫폼 등 여러 측면에서 성공적으로 평가할 수 있습니다. 다양한 상품을 비교하는 점수표를 작성하면, 작업 분량이 어느 정도인지 상급 의사결정자들의 이해를 도울 수 있고, 각 제품의 오랜 UX 부담을 낮춤으로써 제품 관련 팀들이 제품의 UX 성숙도 향상을 더 신속하게 추진할 수 있습니다.
A criação de uma Calculadora de Dívida de Experiência do Usuário ajuda a identificar onde você deve priorizar os esforços de experiência do usuário. A combinação do conceito de dívida de experiência do usuário com os modelos de maturidade de experiência do usuário permite uma avaliação bem-sucedida de vários produtos em diferentes estruturas de experiência do usuário, níveis de maturidade em usabilidade e plataformas de tecnologia. A criação de um scorecard que compara vários produtos ajuda a alta direção a compreender a quantidade de trabalho a ser realizada, e ajuda as equipes de produto a avançar o nível de maturidade em experiência do usuário dos produtos mais rapidamente, reduzindo desta forma a dívida de experiência do usuário do produto.
O artigo completo está disponível somente em inglês.
UXへの取り組みを「どこで優先させるべきか」を判断するには、UXの負債計算機を作成すると良い。「UXの負債」というコンセプトと「ユーザーエクスペリエンスの成熟度」のモデルを組み合わせることで、UXのフレームワーク、ユーザビリティの成熟度、技術的プラットフォームといった項目で複数の製品を正当に評価することができる。様々な製品を点数化して比較すれば、必要とされる仕事量を上層部が理解しやすくなり、製品チームが製品のUXをより速く成熟させ、各製品の長期的なUXの負債を減少することにつながる。
Crear una calculadora de deuda en la experiencia de usuario ayuda a identificar las áreas en las cuales deben priorizarse los esfuerzos de experiencia de usuario. Combinar el concepto de deuda en la experiencia de usuario con los modelos de madurez de la experiencia de usuario permite la evaluación exitosa de varios productos en diferentes plataformas de tecnología, niveles de madurez de la usabilidad y entornos de experiencia de usuario. Crear una tabla que compare diferentes productos ayuda a los líderes con mayor antigüedad a comprender cuánto trabajo debe realizarse y a los equipos de producto a avanzar hacia la madurez de los productos en cuanto a la experiencia de usuario. A su vez, esto reduce la deuda que existe desde hace mucho tiempo con la experiencia de usuario.
La versión completa de este artículo está sólo disponible en inglés
Susan Teague Rector is an IBM design principal with 20 years of experience crafting information architectures and online experience strategies for a variety of public and private sector organizations. She is currently working on design for cognitive systems at IBM Watson Health.
Dr. Kimberly Dunwoody is an experience executive with more than 20 years creating innovative cross-channel experience strategies and policies. Kimberly has conducted customer research on five continents and is currently leads design for the Payer/Provider markets at IBM Watson Health.