Skip to content Skip to sidebar Skip to footer

UX Debt in the Enterprise: A Practical Approach

In early 2014, we found ourselves working for an organization with a large product portfolio built up over decades of acquisitions. To create a cohesive branded experience, it became clear that many of the products we were suddenly supporting existed at very different levels of UX maturity. Our challenge was to create a comparable scorecard for an array of products across UX frameworks, usability maturity, and technology platforms. Our first, somewhat obvious step, was to leverage existing UX maturity models to build a custom framework for our organization, following the advice of Rich Buttiglieri in his presentation on maturity models at UXPA Boston 2012. The model we built focused on the following for each level:

  • What will the user see and experience?
  • How will the experience impact brand perception?
  • What level of UX process support is needed?
  • How would product management, as a whole, react towards the framework proposed?

Table 1. Measures of Experiential Maturity

What does this type of experience look like?
Ad-HocFragmented & inconsistent experience

Disconnected from changing user needs

Measure satisfaction, but not experience quality

IntentionalConducts customer research, but it does not drive the experience

Culture uses data to drive experience decisions

By DesignUses structured research to understand needs by segment and designs directly address needs.

Specify the differentiated experience the organization wants to deliver

IntegratedIdentify operating model changes required to deliver that experience

Measure and reinforce desired behaviors across the ecosystem (internal and external)

UX is fully integrated with the product lifecycle

UX-DrivenEngage customers, employees, etc. in collaborative innovation

Provide a platform to personalize and configure experiences

Find ways to better ways to engage users by testing designs with live traffic

What is the resulting brand perception?
Ad-Hoc“This product/service works as programmed.”
Intentional“This product/service is accurate & available.”
By Design“This product/service is super easy to use and works like I think.”
Integrated“This is a memorable product/service”
UX Driven“Wow, this is fantastic!”
How does the organization think about UX?
Ad-HocUnintentional design

“Users will be trained.” User will need training to use the product—usability is not self-evident.

IntentionalStarting to use data in design decisions

Getting to know users better through structured data

By DesignActivity-focused design

UI designed by UX professionals

IntegratedUser-centric design

“In the field to study users” and UX metrics also used for product planning.

UX DrivenUser-centric DNA

Data-driven UX is simply the way the organization learns and acts to transform its products/services.

What is the associated UX process maturity?
Ad-HocUX methods are not consistently applied and work is often outsourced.

Heuristic reviews and limited usability testing, usually at process end or conducted on products already in production.

IntentionalHire more staff to do more work; although volume increases, work remains more tactical.

Heuristic reviews & limited usability testing performed at the beginning and end of the process.

Mock-ups and prototypes used to iteratively improve designs

By DesignWell-defined UX process, consistent quality and performance

Contextual analysis and greater focus on managing ‘usefulness’

Discovery research (e.g. personas) are regularly used to make design decisions.

IntegratedUser-centric design

“In the field to study users.”

UX metrics used for product planning

UX DrivenDesigns that perform in a predictable way

UX metrics are formalized to capture baseline vs. improvement

Competitive analysis, personas, and field research

Quantitative studies (baseline and comparative)

We began introducing our initial experience maturity model to leadership, both within the product organization and throughout IT. This proved invaluable in helping our internal stakeholders understand why our processes were a little different for each product. It also served as a communication tool to help our UXers understand how to relate to internal product management teams.

In addition to introducing our maturity model to key stakeholders, we also needed to provide senior leadership with an executive summary so they could make comparisons across products. This would allow them to gain insight across a large portfolio of varying user experiences and to identify where to prioritize UX in the organization.

We experimented with multiple methods to create this type of summary for products and kept returning to the idea of a UX debt scorecard. What if we could combine the concept of UX debt with our experiential maturity model? This could not only help us communicate across the organization on a level playing field, but would also provide us with a way to measure how products were decreasing their debt, thereby increasing their experience maturity.

In this article, we discuss UX debt as a way to measure how easy and intuitive a product is to use so we can use the UX debt score as a way to determine the organization’s level of experiential maturity. For example: when the Ad-Hoc UX Debt=0, the organization is ready move to Intentional UX. The same formula works for the move from each level, all the way to a UX-driven organization.

We are now calculating UX debt relative to the product’s current maturity level to provide senior leadership with a product-by-product comparative dashboard of UX debt specific to a given product’s maturity level.

Due to the organization’s numerous, diverse products, we created a calculation worksheet meant to consistently calculate UX debt across our portfolio. As we began to communicate this concept throughout the organization, it was apparent that this was an effective communication and planning tool across multiple stakeholders: sales, product teams, technology teams, and senior leadership.

Defining UX Debt

The term “technical debt” was coined by Ward Cunningham as a metaphor used to describe the increased cost of maintaining technology due to shortcuts taken during the development process. Joshua Kerievsky later extended the metaphor to user experience design using the term “User Experience (UX) Debt.” Kerievsky explained that, like technical debt, UX debt will eventually come due, usually in the form of less customer satisfaction and possible customer defects.

In an enterprise setting, studies show that the decision to accumulate technical debt is often made by non-technical stakeholders. In our current enterprise, we wondered if the same is true for UX debt? If we develop a practical way to measure UX debt for products in the enterprise, what would that look like and how would it be received?

From Theory to Practice

We were brought into a company that had previously tried to ”perform UX,” but teams were unable to make much visible progress across multiple diverse, distinct product lines. Non-UXers (largely product managers) were often in charge of making UX decisions. Across this wide variety of products, customer satisfaction scores were showing that user experience, along with a large UX debt, was hindering products’ usability and creating unhappy customers in the process.

How could a small UX team of five people begin to describe at a high level how more than 100 products compared to each other? How could we communicate to senior managers the amount of UX work that we might need to perform to improve them? We quickly shifted our conversations to whether the notion of UX debt could help not only prioritize our UX work, but could pave the way to communicate and develop a more data and user-centric approach to product development at an enterprise level. Our team was familiar with the term UX debt from a theoretical perspective, but we soon began to wonder if we could actually create a long-term UX debt calculation and track it over time for our products. We wanted to create a comparable scorecard across product lines, and to help explain to our senior leadership how much work needs to be done. This was the point where we began to develop the UX debt calculator spreadsheet.

Table 3. A summary of UX Debt by Maturity Level

Ad-HocIntentionalBy DesignIntegrated
How to calculate UX debtHeuristic
analysis
Heuristic analysis or usability dataUsability dataUsability data
Main type of UX debt to resolveConsistency

Branding

Baseline metrics

Meets user needs

Findable

Usable

Personalized

Persuasive

Accessible

Persuasive
Common areas to examineMap inconsistenciesAnalyze product against known user needs

Level of consistency, based on UX standards

Personalization to user needs

Alignment of current experience with organization’s visionary experience

Use of persuasive design elements

Desired user behaviors

The UX Debt Calculator

UX debt can be calculated with data from either heuristic evaluations or prior usability testing.

Calculating UX debt with heuristic data

We noticed that many products have little or no formative usability data available; products are mostly at the Ad-Hoc level of maturity. To calculate UX debt in these instances, we use heuristic analysis. The UX team performs a heuristic evaluation of the product using the following categories: Findable, Accessible, Clear & Communicative, Useful, Credible, Controllable, Learnable, Overall Aesthetics, Persuasive, and whether the product is currently tracked using our web analytics tool.

In the Template-Heuristics tab in the calculator, the categories correspond to a heuristic. As we complete the evaluation of a product or service, we update the value of the Assessment field for each measure. This field controls the Actual Score and Score Percentage. Possible values for the measure include:

  • Unusable: imperative to fix – Unusable
  • Major issue: important to fix with high priority – Critical
  • Minor issue: should be given low priority – Moderate
  • Cosmetic issue only – Minor
  • Meets Criteria No Issues (default value)

If the product has failed to take accessibility into account, we assign a severity of Major issue. In addition, if we do not currently track the product’s usage, do not complete formative and/or summative testing, and do not implement company branding, we assign a severity of Major issue. This helps drive a culture making product decisions based on user data and beginning to help “paydown” UX debt.

We use the fields Problems and Steps to Resolve to communicate what needs to be addressed in the product for it to move up in its UX maturity. Once all columns are completed, we calculate the current UX debt and any projected paydown.

Calculating debt with prior usability data

In our organization, usability tests were previously conducted on a few products at higher levels of UX maturity. Our initial proof of concept for our UX debt calculator was to translate these third-party studies into a standard debt score. This approach quickly became our preferred approach to calculating debt, as it takes direct user feedback into account. However, when there is no formative data (as was the case with many of our products at lower levels of UX maturity), we default to heuristic evaluation to calculate UX debt.

Screenshot of the author’s UX debt calculator
Figure 1. The UX debt calculator is a spreadsheet with measures and scores

Next Steps

This first phase of the UX debt calculator has been useful in helping us assess products. As a small UX team intending to create a branded experience across over 100 products, we must first determine where that product lies in our experiential maturity model and, second, calculate a debt score. As we began to use this calculator for many of our products, we discovered where our gaps lie in terms of user data, user experience, and customer satisfaction. From this vantage point, we can help product teams move the product forward in its UX maturity, as well as lower its long-standing UX debt.

As we move forward with these concepts across our organization, we expect that our company’s products will advance in their UX maturity and, consequently, pay down their debt. The calculator is a great tool for modeling this CX/UX transition in a product and reinforcing a user- and data-centric product culture.

Download the UX debt calculator

[bluebox]

More Reading

[/bluebox]

 创建一个用户体验缺陷计算器,能够帮助您确定在哪些方面需要优先安排用户体验工作。将用户体验缺陷概念与用户体验成熟度模型相结合,可以跨越不同的用户体验框架、可用性成熟度和技术平台成功评估多种产品。创建可比较各种不同产品的记分卡,不仅有助于高级管理层了解需要执行的工作量,还可帮助产品团队更快地提高产品的用户体验成熟度,减少每种产品长期存在的用户体验缺陷。

文章全文为英文版

UX 부담 계산기를 만들면 UX를 위해 우선적으로 노력해야 할 부분을 손쉽게 알아낼 수 있습니다. UX 부담 개념을 사용자 경험 성숙 모델과 병합하면 다양한 제품을 UX 프레임워크, 사용성 성숙도, 테크놀로지 플랫폼 등 여러 측면에서 성공적으로 평가할 수 있습니다. 다양한 상품을 비교하는 점수표를 작성하면, 작업 분량이 어느 정도인지 상급 의사결정자들의 이해를 도울 수 있고, 각 제품의 오랜 UX 부담을 낮춤으로써 제품 관련 팀들이 제품의 UX 성숙도 향상을 더 신속하게 추진할 수 있습니다.

전체 기사는 영어로만 제공됩니다.

A criação de uma Calculadora de Dívida de Experiência do Usuário ajuda a identificar onde você deve priorizar os esforços de experiência do usuário. A combinação do conceito de dívida de experiência do usuário com os modelos de maturidade de experiência do usuário permite uma avaliação bem-sucedida de vários produtos em diferentes estruturas de experiência do usuário, níveis de maturidade em usabilidade e plataformas de tecnologia. A criação de um scorecard que compara vários produtos ajuda a alta direção a compreender a quantidade de trabalho a ser realizada, e ajuda as equipes de produto a avançar o nível de maturidade em experiência do usuário dos produtos mais rapidamente, reduzindo desta forma a dívida de experiência do usuário do produto.

O artigo completo está disponível somente em inglês.

UXへの取り組みを「どこで優先させるべきか」を判断するには、UXの負債計算機を作成すると良い。「UXの負債」というコンセプトと「ユーザーエクスペリエンスの成熟度」のモデルを組み合わせることで、UXのフレームワーク、ユーザビリティの成熟度、技術的プラットフォームといった項目で複数の製品を正当に評価することができる。様々な製品を点数化して比較すれば、必要とされる仕事量を上層部が理解しやすくなり、製品チームが製品のUXをより速く成熟させ、各製品の長期的なUXの負債を減少することにつながる。

原文は英語だけになります

Crear una calculadora de deuda en la experiencia de usuario ayuda a identificar las áreas en las cuales deben priorizarse los esfuerzos de experiencia de usuario. Combinar el concepto de deuda en la experiencia de usuario con los modelos de madurez de la experiencia de usuario permite la evaluación exitosa de varios productos en diferentes plataformas de tecnología, niveles de madurez de la usabilidad y entornos de experiencia de usuario. Crear una tabla que compare diferentes productos ayuda a los líderes con mayor antigüedad a comprender cuánto trabajo debe realizarse y a los equipos de producto a avanzar hacia la madurez de los productos en cuanto a la experiencia de usuario. A su vez, esto reduce la deuda que existe desde hace mucho tiempo con la experiencia de usuario.

La versión completa de este artículo está sólo disponible en inglés