What are Outcome-driven UX Metrics?

How do we measure the improvements our UX efforts make in people’s lives?

As UX professionals, our work should steadily and regularly improve people’s lives. With every release of our products and services, our customers, users, and fellow employees should be better off. 

If we’re not doing that, why are we shipping these releases? We certainly don’t want to be making things worse. 

Yet, it happens. Have you ever upgraded an app you loved, only to find the new version is worse? It’s frustrating. 

How would you discover if you did that to your users?

Ideally, when our newest release improves customers’ lives, they’re excited to pay more for our products and services. When we deliver a new tool or process that enhances our fellow employees’ lives, they are more productive and can add more value into their work — value that gets passed on through the service they deliver.

Our work of improving the user experience should improve the lives of the people who use our products and services. Yet, very few UX teams track those improvements or how much new value they bring to their customers and organization.

This thinking led us to develop the framework for outcome-driven UX metrics, a radical approach to measuring the life improvements UX teams make for their customers, users, and fellow employees.

Conventional UX metrics don’t actually measure UX

When UX teams look to measure their effectiveness, they often turn to conventional UX metrics, such as task completion, time-on-page, conversion rates, adoption rates, customer satisfaction, or (a strong candidate for worst metric of all) Net Promoter Score. They try to use these metrics to demonstrate how their work is making a difference.

The challenge is that nobody understands what these metrics are trying to say. When a user’s average time on the home page grows by 40 seconds, what happens here? Is this truly a better experience for the user?

When the average CSAT (customer satisfaction) score jumps from 66.71 to 71.09, what triggered that jump? Was it the recent changes? Why didn’t those changes make it jump more? What could the team have done differently to get a higher score?

Great metrics tell a story of change. The best metrics explain not only what the change is, but why that change is meaningful.

However, because the conventional UX metrics are difficult to tie to what’s happening when people use our product or service, these metrics won’t tell us if our users’ experiences are improving. They won’t tell us how much value people get from our improvements. These conventional metrics won’t tell the story of why our work is essential. We need something different.

Metrics that demonstrate UX’s value and contribution

UX leaders everywhere face the challenge of demonstrating the value of their teams and their work. They need stories that communicate the contribution of UX back to the organization’s bottom line.

Other parts of the organization use their own metrics to show their contribution. The Sales team uses its (hopefully) increasing revenue numbers to show its effectiveness. The Marketing team tracks their audience reach and lead generation numbers. Human Resources tracks their staffing efforts.

Senior executives look to all these metrics to understand the contributions from each part of the organization. These metrics must be easy to understand and track so executives can instantly understand them. UX teams need informative metrics for executives that communicate the contribution of UX work.

Conventional UX metrics don’t meet that standard because nobody knows why these metrics change as they do. If the numbers go up, the UX team can only hypothesize why. If the numbers go in the wrong direction, nobody can explain what didn’t work.

As a result, these conventional metrics do not tell the story of how the UX efforts contribute to the organization’s success. When that happens, the executives start to wonder if a UX team is necessary.

Searching for alternative metrics is how we came to develop our outcome-driven UX metrics framework.

Starting with UX Outcomes

We wanted to tie our metrics to improvements in our customers’, users’, and fellow employees’ lives. To do that, we needed a method for identifying those improvements. 

We turned to UX outcomes. A UX outcome describes the improvement in someone’s life. It answers the question:

If we do a great job, whose life do we improve, and how do we improve it?

For example, let’s say the team is working on new functionality for booking recurring appointments with medical professionals, such as a therapist.

If we do a great job on the recurring booking functionality, whose life gets improved and how?

The team researched how people tried to book multiple appointments using their existing booking system. They came across one person, Paige, struggling to book six weeks of physical therapy appointments, two per week.

The issues that Paige encountered were numerous:

• She frequently switched between her therapist’s availability and her personal calendar, which felt awkward on her phone.
• She had to re-enter all of her information for each appointment, as the system didn’t carry it forward from one booking to the next.
• The booking system didn’t recognize her referral from her doctor (even though they’re in the same network) and would give her an error message that was challenging to bypass.
• Because it didn’t have an approved referral, the system wouldn’t let her submit the appointment to her insurance for reimbursement, which forced her to call the billing office to resolve the issue.

Paige found the entire experience frustrating, frequently swearing at the booking application. With more research, the team also saw that other patients experienced the same issues when booking their recurring appointments. 

However, their observation of Paige’s experience was particularly memorable, primarily because she was expressive as she repeatedly encountered each issue. They chose Paige for their UX outcome.

If we do a great job on the recurring booking functionality, Paige will find booking her recurring physical therapy appointments much simpler because…
… it will integrate with her personal calendar,
… require her only to enter her information once,
… take advantage of the information we already know about her, including her payment and insurance details,
… would recognize her in-network doctor, and
… automatically submit her appointment to insurance.

The team’s UX outcome defines the project’s goals by discussing how they’ll improve Paige’s experience. And, since Paige’s experience was similar to many other people’s experiences, achieving this outcome will improve things for those other folks, too.

Defining the project’s outcome-driven UX metrics

The team used the UX outcome they defined to create metrics that showed the improvement in Paige’s life.

In our framework for outcome-driven UX metrics, there are three classes of metrics: UX Success metrics, UX Progress metrics, and Problem-value metrics. The team created metrics in each class to tell their story of improving customers’ lives.

UX Success Metrics — measuring when we’ve achieved our outcome

UX Success metrics tell the story of the precise moment we’ve improved the life of our users, customers, or fellow employees. 

You can think of our UX efforts as if we’re running a marathon. While we’re in the process of running the race, we haven’t completed it yet. Then, the moment one foot crosses the finish line, we have.

UX Success metrics work the same way. It’s a binary, yes-or-no measurement. 

Before we achieve the UX outcome, the metric’s numeric value is zero. Once we achieve it, the value is now one. (If we want a value larger than one, we can count the number of people for whom we’ve achieved the outcome.)

To determine when we’ve achieved the outcome, we create the party moment checklist. We use a party moment to know precisely when we can celebrate the achievement of the outcome. We make a checklist to determine that precise moment.

For our UX outcome for the recurring booking functionality, the party moment checklist becomes:

• Integrates with the patient’s personal calendar.
• The patient only has to enter their personal information once.
• The system uses information already known about the patient (including payment and insurer).
• The system recognizes in-network doctors.
• The system automatically submits appointments to insurance.

We’ve crossed the finish line for the UX Success metric when we complete all of these items on the checklist. When Paige (or someone like her) can book recurring appointments with everything happening on the checklist, the UX outcome is achieved, and we can change the value of the UX Success metric from zero to one.

We can enhance the UX Success metric by counting everyone who achieves the outcome. (Just like a marathon’s organizers can count the runners who complete the race.) Or, if our organization makes money for each booked appointment, we can count the additional revenue we’ve earned because of the booked recurring appointments. These are related stories of our success, as told through the UX Success metric.

UX Progress Metrics — measuring how far we’ve come

UX Progress metrics come in handy when we haven’t achieved our outcome completely, yet we need to report what’s been done or how far away we are from achieving success.

I tell clients to think of UX Progress metrics as partial success metrics. Imagine the executive team sitting in the back seat of your car asking, every few minutes, “Are we there yet?” UX Progress metrics provide the answer of, “No, but we’re getting closer.”

There are a couple of ways we can calculate UX Progress metrics. One common way is to turn our party moment checklist into individual milestones to track. As each checklist item is completed, we’re making progress towards success. Give each item one point and report progress as a percentage of the total points.

In our recurring booking example, the team had two UX progress metrics. One measured the moment they could integrate with patients’ calendars and access previously-entered patient information (including their payment and insurer details). The other metric measured when the booking functionality recognized in-network doctors and allowed them to submit appointments to insurance.

Another approach to UX Progress metrics is to measure the number of people benefiting from the enhanced functionality as it comes online. This approach is the equivalent of reporting how many runners have just passed the five-mile marker in the marathon. 

If it had more meaning to the team’s executives, the team could’ve counted and reported the number of bookings that used the new integrations. Teams can choose the approach that best delivers the message they want the executives to take away.

UX Progress metrics work in conjunction with the UX Success metric. They tell stakeholders the story of the UX effort’s forward movement.

Problem-Value Metrics — measuring the costs of poor UX

Problem-value metrics tell a story about what happens if we don’t improve the users’ lives. This third class of metrics is often where teams start.

All products and services have a user experience, whether or not UX people worked on them. However, if no UX love was given before the product was released, chances are strong that users find the release’s experience frustrating.

Almost always, user experiences that are frustrating come back to haunt the organization. Either the company ends up losing subsequent sales, racks up support costs, requires excessive training, reduces the users’ productivity, or incurs unnecessary development costs. The costs incurred by these results will accumulate until something is done to resolve the underlying issues.

We report Problem-value metrics in dollars, euros, or whatever the organization’s currency is. The metrics immediately resonate with executives and senior stakeholders because they are deeply familiar with money and budgets. They can compare the costs of poor UX to other ongoing organizational costs.

That’s why many teams start with Problem-value metrics. These immediately capture the attention of their executives and can compel them to invest in delivering better-designed products and services to reduce the ongoing costs.

The team noticed two significant Problem-value metrics associated with recurring bookings. When Paige and others found the existing booking functionality onerous, they often called the therapist’s office and attempted to book over the phone. The team could count how much time (and from that, how much money) was spent handling these phone calls. 

They also noticed that when the insurance reimbursement failed, the patients contacted the billing department to fix the problem. They could track how much time (and, again, how much money) was spent resolving the reimbursement issues. Over one year, these two metrics proved to be a shockingly large number that caught the attention of senior management.

Problem-value metrics demonstrate what happens when we don’t invest in delivering great UX to our customers, users, and fellow employees. They tell powerful stories about the value of UX to the organization. It’s a compelling contrast to the investment costs of delivering something that improves people’s lives.

Outcome-driven UX Metrics tell the story of our efforts

UX metrics don’t have to be complicated or obscure. We can directly measure the improvement we make in people’s lives. 

Great metrics tell a story of change. When we use outcome-driven UX metrics, we’re telling the story of how we’re improving the experiences of our customers, users, and fellow employees.

In the case of our booking team, they started with their UX outcome:

If we do a great job on our recurring booking functionality, we’ll improve Paige’s life by making the appointment booking process more seamless.

The team ended up with a strong UX Success metric:

We’ll be successful with our recurring booking functionality when our patients can:

• Easily compare available appointments to the patient’s calendar.
• Only has to enter their personal information once.
• No longer needs to enter information already known about the patient (including payment and insurer).
• No longer receives errors about in-network doctors.
• Submits appointments to insurance for reimbursement automatically.

For their UX Progress metrics, the team tracked:

Phase 1 work will be complete when:

• Integrates with the patient’s personal calendar.
• The patient only has to enter their personal information once.
• The system uses information already known about the patient (including payment and insurer).

Phase 2 work will be complete when:

• The system recognizes in-network doctors.
• The system automatically submits appointments to insurance.

And, the team collected these for Problem-value metrics:

The money spent on handling recurring bookings over the phone.

The money spent fixing reimbursement issues caused by errors due to improper booking problems.

The combination of these metrics gave the team a robust way to communicate the improvements they were making to the lives of Paige and their other patients.

We’ve shared other stories showcasing how teams leveraged the outcome-driven UX metrics framework to demonstrate their impact and value to organizational priorities:

• How a UX Team Discovered an Underserved Audience and Made Millions for Their Organization tells the story of a subscription service that used the framework to solve a retention issue.
• How the Right UX Metrics Show Game-Changing Value shares how a team identified and tracked a $33 million UX fix.
• Measuring Experiences, Not Product Use details how teams can measure changes in their users’ experiences.

Our outcome-driven UX metrics framework is a powerful approach to showing the value that UX makes to people’s lives. This metrics approach makes it easier for UX leaders to demonstrate their teams’ contribution to their organizations’ success to stakeholders and executives.