How the Right UX Metrics Show Game-Changing Value

There’s a story I want to tell you. It’s a story that emerged out of my recent work with a UX team. They used this story to excite their executives on how recent UX improvements made a substantial business impact.

Since it’s not my story to tell, I’ve changed many of the details. The events in this story aren’t what actually happened. However, the events I describe are very much like what happened. 

While it’s a great story, I want you to pay close attention to how the UX team tells their story. Look at the metrics and the narrative that connects those metrics. It is an inspiring story, and how they tell it is just as educational as the story they tell.

Stories like this one are potent tools for UX leaders. When you tell stories like these, you’ll grab the attention of your executives and senior stakeholders, demonstrating the immense value of great UX in a whole new way.

Here’s the background to understand this story: The UX team works on an invoice-tracking application for small business owners who send invoices and collect payments. The application has been on the market for many years, and for most of its existence, customers paid the business owners’ invoices with checks.

Ok. With all that buildup, here’s the story:

Eighteen months ago, the UX team, collaborating with their Product and Development team partners, released a Pay Now feature to expedite customer payments to business owners.

At that time, their research showed that when customers paid with checks, more than 65% of payments took 20 or more days to land in the business owner’s bank account, with 15% of payments often taking 60 days or more. Everyone hoped the Pay Now feature would dramatically reduce payment times.

When the feature shipped, there was good news and bad news. The good news was that it worked. 60% of business owners who added a Pay Now button to their invoices saw their payment times drop to under two days.

The bad news was that only 5% of business owners were adding the Pay Now button to their invoices. Most business owners were not adopting the new feature. 95% of business owners were still sending invoices, just like they’d always done, expecting payments by check many weeks later.

When talking with business owners, they told the team their cash flow was their biggest headache. They were desperate for anything to reduce collection times, yet most business owners had no idea what was already built into the product.

The UX team researched why business owners weren’t using the Pay Now capability. They learned that most customers couldn’t find the capability because it was hidden too deep inside the user interface.

There were a few business owners who had somehow discovered and tried the functionality. However, many of these business owners thought the feature’s implementation created a poorer user experience with their customers.

The design of the Pay Now workflow was forcing business owners to send a separate email from any pre-existing email threads with customers. Their customers often missed this special email and didn’t pay the invoice, requiring the business owner to chase down the money — exactly the opposite of what Pay Now was supposed to do.

Using what they’d learned from the business owners, the team released an improved version a year later. In this release, it was much easier to discover the Pay Now functionality and better integrate it with the business owners’ existing invoicing workflows. The results after this release were dramatic and immediate.

In the four months after the new release, business owners sent out approximately 900,000 invoices, up from 800,000 in the four months before the new functionality. Those 900,000 invoices came to roughly $2,700,000,000 in payments.

585,000 (65%) of the 900,000 invoices went out with the Pay Now button, up from only 5% before the release. Customers sent roughly $1,755,000,000 in payments through the Pay Now system. 

All these payments generated $35,100,000 from fees and interest for the company. This revenue is up from $2,400,000 in the months before — a 13½ times increase of almost $33 million. 

486,000 (84%) of those Pay Now-enabled invoices were paid in less than two days. Over the four months, 24,600 business owners were paid within an hour of sending their invoices.

Best of all, the business owners told the UX team they loved the Pay Now functionality. Their customers were paying faster, which gave the business owners additional cash flow to handle more work. They told the UX team this feature was life-changing for them.

The UX team is now working on other cashflow and collection functionality. Their current research shows these are still big challenges for business owners, even with the improved Pay Now functionality.

That’s the story. When UX leaders first told it, it swept through the organization.

The $33 million revenue increase immediately captured the attention of the executives and senior management. They were amazed to learn that this highly profitable release contained no new functionality — only user experience improvements that fixed flaws in the original Pay Now implementation.

These executives and senior stakeholders now understood how UX contributes to the company’s success differently. Like top-level management everywhere, they had believed UX was the “make it pretty” group. Now, they saw the team as strategic partners, essential for achieving the organization’s objectives.

This change in how executives think about UX is why you also need stories like this. Let’s take apart how this story worked so you can learn how to create your own.

It’s all in the metrics.

The metrics in this story are what make it inspirational and attention-grabbing. Had the UX leaders not included the metrics, the story wouldn’t have had the impact and made the impressions it did. 

When we take apart the story, we can see that there are seven quantitative metrics that the UX team tracked and reported on:

• The total number of invoices sent
• The number of invoices that included the Pay Now button
• The accumulated dollar amount of those invoices
• The number of payments customers made through Pay Now
• The accumulated dollar amount of those payments
• The accumulated dollar amount of the fees earned from the payments
• The number of days customers took to pay each invoice

At first glance, these may not look like UX metrics. They’re not task time or customer satisfaction scores, which are what many UX professionals consider when choosing their metrics.

However, the UX team chose these because they measure the experience business owners have with the new functionality. They are outcome-driven UX metrics. 

The UX team identified these metrics by building a well-researched understanding of how business owners deal with invoicing and collections. They learned more about invoicing and collections from every business owner they talked to, grew their understanding, and generated ideas to make it substantially easier and burden-free for business owners and their customers.

The UX team collaborated with development, product management, and the company’s data team to collect data for these metrics. The development and data teams had to build custom instrumentation because no analytics vendor would build these metrics into their off-the-shelf tools. They merged behavior data with financial data, producing a complete picture of how the new release performed.

Looking further inside the story, we also find several qualitative metrics. (These are metrics that do not need to be numeric quantities to tell their story.) When telling this story, the UX team brought out these four qualitative metrics:

• The change in the pain that business owners felt for collections that took longer than they liked.
• The change in how disruptive the Pay Now process was to the business owners’ workflow.
• The change in how difficult it was for users to discover the Pay Now functionality.
• The change in how the business owners felt towards the product.

These qualitative metrics were essential to showing executives and stakeholders the improvements the UX team made to the business owners’ experience. The team collected the data for these metrics by having regular conversations with business owners before and after the release’s rollout. Those conversations let them track changes in how the business owners perceived the Pay Now functionality.

Where are the conventional UX analytics?

Interestingly, the UX team doesn’t use standard UX analytics when telling this story. The story doesn’t mention commonly reported metrics such as time on page, time to complete a task, click-through rates, or conversion rates.

The UX team’s leaders chose not to include these metrics. Why? These metrics won’t make the story interesting.

UX folks typically gravitate to conventional metrics, not because they are the best for measuring UX but because they’re easy to collect and report. The metrics come bundled with the analytics tool, so no custom instrumentation is needed.

Yet, despite their ease of collection, conventional analytics make it almost impossible to tell a compelling story.

Sometimes, it’s unclear whether an analytic shows improvement by increasing or decreasing. Is spending less time on each page better because that means the user found what they needed quickly? Or should it be longer because the user is more engaged with the content? 

Who knows? I don’t. And your executives sure won’t either.

The UX team skipped analytics like these and instead focused on metrics tied to business success and how the new release would improve users’ lives. It’s exciting when you can report that your UX change brought in almost $33 million while reducing your customer’s payment waiting time from more than 20 days to under 2.

Where are the attitudinal metrics?

Similarly, the story doesn’t mention satisfaction scores (CSAT, OSAT, or CES), Net Promoter Scores (NPS), or System Usability Scores (SUS). These are the frequent go-to metrics that UX folks use to show that their users have become happier with a new design.

Again, UXers choose them not because they are good indicators of how customers feel but because they are easy to collect. Just pop up a little survey and collect a number.

Unfortunately, these measurements are essentially performative — they are the business equivalents of astrological readings. They are garbage metrics that don’t measure what they promise to measure. 

At best, these metrics are a quick mood check of the user, yet they are unrelated to anything happening with the design. Reporting that you increased the customers’ average satisfaction from 3.6 to 3.9 is meaningless to executives. Stakeholders can’t see how that increase relates to their most important objectives or how it will solve any of the organization’s most challenging problems. Nobody throws a big party when they discover their Net Promoter Score is now 71 because nobody knows what 71 means.

To measure someone’s happiness, measure whether you’ve improved their life. When your newest design improves someone’s life, they become happier with you. Focus on improving their life, and their happiness will come along for the ride.

Through extensive research, the UX team discovered what would make their business owner users happier: getting paid faster. Therefore, they measured that and avoided the inscrutable numeric customer satisfaction scales.

Having the hard conversations.

There’s a little more to this story. After the celebrations died down, the UX team leader met with the team’s most supportive executive champion to have an uncomfortable conversation:

This release generated almost $33 million in revenue over the past four months. That means had we built the initial Pay Now release well, we could’ve generated almost $100 million in revenue a year earlier.

When we first worked on the new Pay Now capability, we prioritized shipping “something” out quickly. We deferred our plans to research the business owners’ needs.

I understand that we wanted this functionality in the market immediately. However, that decision cost us $100 million. How would things have turned out if we’d pursued the research plans?

That conversation generated an agreement that the executive team had made the wrong decision to push the functionality out too quickly. This executive champion now saw value in conducting thorough UX research to understand users’ needs. 

In subsequent projects, the UX team played a much more strategic role in researching their customers’ and users’ needs. By using outcome-driven UX metrics that emerge from that research, they are now showing even more value from the UX team’s contributions.