In early 2014, we found ourselves working for an organization with a large product portfolio built up over decades of acquisitions. To create a cohesive branded experience, it became clear that many of the products we were suddenly supporting existed at very different levels of UX maturity. Our challenge was to create a comparable scorecard for an array of products across UX frameworks, usability maturity, and technology platforms. Our first, somewhat obvious step, was to leverage existing UX maturity models to build a custom framework for our organization, following the advice of Rich Buttiglieri in his presentation on maturity models at UXPA Boston 2012. The model we built focused on the following for each level:
- What will the user see and experience?
- How will the experience impact brand perception?
- What level of UX process support is needed?
- How would product management, as a whole, react towards the framework proposed?
Table 1. Measures of Experiential Maturity
|What does this type of experience look like?|
|Ad-Hoc||Fragmented & inconsistent experience
Disconnected from changing user needs
Measure satisfaction, but not experience quality
|Intentional||Conducts customer research, but it does not drive the experience
Culture uses data to drive experience decisions
|By Design||Uses structured research to understand needs by segment and designs directly address needs.
Specify the differentiated experience the organization wants to deliver
|Integrated||Identify operating model changes required to deliver that experience
Measure and reinforce desired behaviors across the ecosystem (internal and external)
UX is fully integrated with the product lifecycle
|UX-Driven||Engage customers, employees, etc. in collaborative innovation
Provide a platform to personalize and configure experiences
Find ways to better ways to engage users by testing designs with live traffic
|What is the resulting brand perception?|
|Ad-Hoc||“This product/service works as programmed.”|
|Intentional||“This product/service is accurate & available.”|
|By Design||“This product/service is super easy to use and works like I think.”|
|Integrated||“This is a memorable product/service”|
|UX Driven||“Wow, this is fantastic!”|
|How does the organization think about UX?|
“Users will be trained.” User will need training to use the product—usability is not self-evident.
|Intentional||Starting to use data in design decisions
Getting to know users better through structured data
|By Design||Activity-focused design
UI designed by UX professionals
“In the field to study users” and UX metrics also used for product planning.
|UX Driven||User-centric DNA
Data-driven UX is simply the way the organization learns and acts to transform its products/services.
|What is the associated UX process maturity?|
|Ad-Hoc||UX methods are not consistently applied and work is often outsourced.
Heuristic reviews and limited usability testing, usually at process end or conducted on products already in production.
|Intentional||Hire more staff to do more work; although volume increases, work remains more tactical.
Heuristic reviews & limited usability testing performed at the beginning and end of the process.
Mock-ups and prototypes used to iteratively improve designs
|By Design||Well-defined UX process, consistent quality and performance
Contextual analysis and greater focus on managing ‘usefulness’
Discovery research (e.g. personas) are regularly used to make design decisions.
“In the field to study users.”
UX metrics used for product planning
|UX Driven||Designs that perform in a predictable way
UX metrics are formalized to capture baseline vs. improvement
Competitive analysis, personas, and field research
Quantitative studies (baseline and comparative)
We began introducing our initial experience maturity model to leadership, both within the product organization and throughout IT. This proved invaluable in helping our internal stakeholders understand why our processes were a little different for each product. It also served as a communication tool to help our UXers understand how to relate to internal product management teams.
In addition to introducing our maturity model to key stakeholders, we also needed to provide senior leadership with an executive summary so they could make comparisons across products. This would allow them to gain insight across a large portfolio of varying user experiences and to identify where to prioritize UX in the organization.
We experimented with multiple methods to create this type of summary for products and kept returning to the idea of a UX debt scorecard. What if we could combine the concept of UX debt with our experiential maturity model? This could not only help us communicate across the organization on a level playing field, but would also provide us with a way to measure how products were decreasing their debt, thereby increasing their experience maturity.
In this article, we discuss UX debt as a way to measure how easy and intuitive a product is to use so we can use the UX debt score as a way to determine the organization’s level of experiential maturity. For example: when the Ad-Hoc UX Debt=0, the organization is ready move to Intentional UX. The same formula works for the move from each level, all the way to a UX-driven organization.
We are now calculating UX debt relative to the product’s current maturity level to provide senior leadership with a product-by-product comparative dashboard of UX debt specific to a given product’s maturity level.
Due to the organization’s numerous, diverse products, we created a calculation worksheet meant to consistently calculate UX debt across our portfolio. As we began to communicate this concept throughout the organization, it was apparent that this was an effective communication and planning tool across multiple stakeholders: sales, product teams, technology teams, and senior leadership.
Defining UX Debt
The term “technical debt” was coined by Ward Cunningham as a metaphor used to describe the increased cost of maintaining technology due to shortcuts taken during the development process. Joshua Kerievsky later extended the metaphor to user experience design using the term “User Experience (UX) Debt.” Kerievsky explained that, like technical debt, UX debt will eventually come due, usually in the form of less customer satisfaction and possible customer defects.
In an enterprise setting, studies show that the decision to accumulate technical debt is often made by non-technical stakeholders. In our current enterprise, we wondered if the same is true for UX debt? If we develop a practical way to measure UX debt for products in the enterprise, what would that look like and how would it be received?
From Theory to Practice
We were brought into a company that had previously tried to ”perform UX,” but teams were unable to make much visible progress across multiple diverse, distinct product lines. Non-UXers (largely product managers) were often in charge of making UX decisions. Across this wide variety of products, customer satisfaction scores were showing that user experience, along with a large UX debt, was hindering products’ usability and creating unhappy customers in the process.
How could a small UX team of five people begin to describe at a high level how more than 100 products compared to each other? How could we communicate to senior managers the amount of UX work that we might need to perform to improve them? We quickly shifted our conversations to whether the notion of UX debt could help not only prioritize our UX work, but could pave the way to communicate and develop a more data and user-centric approach to product development at an enterprise level. Our team was familiar with the term UX debt from a theoretical perspective, but we soon began to wonder if we could actually create a long-term UX debt calculation and track it over time for our products. We wanted to create a comparable scorecard across product lines, and to help explain to our senior leadership how much work needs to be done. This was the point where we began to develop the UX debt calculator spreadsheet.
Table 3. A summary of UX Debt by Maturity Level
|How to calculate UX debt||Heuristic
|Heuristic analysis or usability data||Usability data||Usability data|
|Main type of UX debt to resolve||Consistency
|Meets user needs
|Common areas to examine||Map inconsistencies||Analyze product against known user needs
Level of consistency, based on UX standards
|Personalization to user needs
Alignment of current experience with organization’s visionary experience
|Use of persuasive design elements
Desired user behaviors
The UX Debt Calculator
UX debt can be calculated with data from either heuristic evaluations or prior usability testing.
Calculating UX debt with heuristic data
We noticed that many products have little or no formative usability data available; products are mostly at the Ad-Hoc level of maturity. To calculate UX debt in these instances, we use heuristic analysis. The UX team performs a heuristic evaluation of the product using the following categories: Findable, Accessible, Clear & Communicative, Useful, Credible, Controllable, Learnable, Overall Aesthetics, Persuasive, and whether the product is currently tracked using our web analytics tool.
In the Template-Heuristics tab in the calculator, the categories correspond to a heuristic. As we complete the evaluation of a product or service, we update the value of the Assessment field for each measure. This field controls the Actual Score and Score Percentage. Possible values for the measure include:
- Unusable: imperative to fix – Unusable
- Major issue: important to fix with high priority – Critical
- Minor issue: should be given low priority – Moderate
- Cosmetic issue only – Minor
- Meets Criteria No Issues (default value)
If the product has failed to take accessibility into account, we assign a severity of Major issue. In addition, if we do not currently track the product’s usage, do not complete formative and/or summative testing, and do not implement company branding, we assign a severity of Major issue. This helps drive a culture making product decisions based on user data and beginning to help “paydown” UX debt.
We use the fields Problems and Steps to Resolve to communicate what needs to be addressed in the product for it to move up in its UX maturity. Once all columns are completed, we calculate the current UX debt and any projected paydown.
Calculating debt with prior usability data
In our organization, usability tests were previously conducted on a few products at higher levels of UX maturity. Our initial proof of concept for our UX debt calculator was to translate these third-party studies into a standard debt score. This approach quickly became our preferred approach to calculating debt, as it takes direct user feedback into account. However, when there is no formative data (as was the case with many of our products at lower levels of UX maturity), we default to heuristic evaluation to calculate UX debt.
As we move forward with these concepts across our organization, we expect that our company’s products will advance in their UX maturity and, consequently, pay down their debt. The calculator is a great tool for modeling this CX/UX transition in a product and reinforcing a user- and data-centric product culture.
- Technical Debt – Wikipedia
- An Enterprise Perspective on Technical Debt – MTD ’11 Proceedings of the 2nd Workshop on Managing Technical Debt.
- “To Pay or Not to Pay Technical Debt Software” by Andrew Wright, IEEE (2013 Volume: 28 , Issue: 6 ).
- “Applied UX Strategy” by Yury Vetrov (UX Matters)
- “How UX Evolves at Companies: A New Look at Maturity Models” by Rich Buttiglieri
Retrieved from http://uxpamagazine.org/ux-debt-in-the-enterprise/