Conducting Inherent Value Testing

by Jared M. Spool

In last week’s article, I summarized the benefits of Inherent Value Testing, a simple usability testing technique that can help you measure how your site communicates your product’s value.

If you’ve conducted a usability test in the past, it’s simple to put together an Inherent Value Test. As with all testing, you start with recruiting the right participants. You’ll need two groups: people experienced with the site and people who you will introduce to the service during the test.

Let’s say you want to understand the value of your HR department’s intranet site. You would look for two groups of employees: those who are using the site on a regular basis and those who have never (oralmost never) used it before. You’d want to recruit a minimum of six from each group, to ensure you have enough users to see important behavioral patterns.

We start with the experienced group in the first phase. We want them to interact as “naturally” as possible with the site, so we may ask them to bring some of their own work or data to use.

For example, when testing your HR site, you might look for people who currently have questions about their benefits or need to make changes to their retirement plan distributions. Resolving their own issues gives the participants a “focus” for their work.

We’ve found having a focus works better than just a random tour without any focus. We get to see how users work with the site as they discuss it. Their commentary is easier to understand and more accurately represents true issues with the site.

Don’t be afraid to guide the users to visit specific portions of the site. Feel free to ask them what they think is important about it. In some types of usability tests, “leading the user” is inappropriate. However, in Inherent Value Testing, it’s a useful technique for discovering valuable qualitative perspectives on a variety of portions of the site.

You also can ask the participants to visit a competitor’s site in addition to the site you are studying. Having them take you on a tour of the competitor’s site, describing what they like and what they don’t like, gives them a point of comparison and highlights important benefits and deficiencies of your site.

Phase I: Extracting Benefits from Participants

As their working through the site, you want to ask them what they like and don’t like about the service. Focus your discussion away from the design specifics, such as fonts or colors, and more towards the utility of the service.

In our study of the HR site, we might ask: What parts of the site do they like the best? What part is most helpful? What have they used before? What did they do with that portion of the site?

Leverage any discussion on their comments. If the user says, “I like how complete the information is on this page.” You can respond with, “What do you think makes it complete information?” It can be difficult for participants to really analyze the site at first, so persistence will pay off.

At the end of the test session, just before you send them away, you can ask two useful questions: “What are two things you really liked about the site?” and “What are two things you really disliked about the site?” The most important answers are the ones that just roll off their tongue. In our experience, the more they have to think about it, the less important the like or dislike turns out to be.

When you’re done with the first phase, you should easily have identified patterns in what people like and dislike about your offerings. You’ll also have seen which features people aren’t discovering and how they react to them the first time. This feedback can help you brainstorm ideas on how you let users know about these hidden features and benefits.

Phase II: How Do New Users View the Benefits

For the second phase, you’ll bring in your ‘new’ users. As with the experienced participants, you’ll want to use test tasks that are as natural as possible. If you can ask them to bring the same work as the experienced users, that will work great.

However, if bringing in work isn’t feasible, you can ask the participants to ‘role play’ in scenarios similar to those performed by the experienced users. We’ve found that quality role playing can reveal many issues about a design.

For example, on the HR site, you could ask them to pretend that they are about to have a new baby and ask them to investigate the company’s family leave policy and procedures. Make up any appropriate but necessary details, such as the baby’s due date, but let them use their own situation as much as possible, such as their real manager and upcoming deadlines.

As their working with the site, you want to look at each page closely. Do they visit the same pages as the experienced users? Do the pages reveal the same benefits that you heard from the experienced users? If not, what benefits are *they* seeing from the site?

Viewing the same site through these new users’ eyes is often enlightening. For many sites we’ve studied, it’s a very disappointing experience the first time, as we realize how poorly the site communicates the inherent values. However, subsequent testing is encouraging as we see how simple design changes can completely change the new users’ perspectives of the site.

Analyzing the Results

Once you’ve completed both phases, you want to look at the behavioral patterns in each phase. You’ll also want to look at the differences between to the two phases.

You’ll want to ask yourself, “If the phase 2 users saw the same inherent value as the phase 1 users, what would be different?” You mayrealize that you’d see more sales or reduced call center costs.

It’s very likely you can come up with a fairly accurate estimate of the increased revenues or decreased costs, by estimating the number of people who fit into the ‘new user’ category. Multiplying that estimate by the average increase in revenues or savings in expenses will give you a powerful estimate.

You’ll also have new information to guide future development priorities. You’ll know how much to invest in new benefits and features and how much to invest in making the existing features more visible. Often, the latter can take a smaller investment to yield good results.

Inherent Value Testing is a straight-forward variant of a basic usability test. But don’t let its simplicity fool you. It’s an extremely powerful addition to your usability toolbox.

Published here on February 25, 2004.