Scheduling Hard-to-Find Users

Developers may hesitate to start usability testing because they worry that their product poses special problems in finding, scheduling, or compensating the right users. This shouldn’t stop them. We successfully find and test hundreds of users a year and about 10% of these require special tactics for scheduling. Some of these tactics include:

• Keeping a user database. This is the first place we look to find possible recruits
• Not focusing on money as an incentive. The best incentive is curiosity about new products and the chance to affect future versions.
• Starting early. Difficulties will show up right away, and this gives us time to adjust.

Here are some other things we’ve learned.

Some Roles Have Warning Signs

Even with detailed information on 800-plus users, we continually encounter new profiles we need to fill. Although our basic rules are the same for all kinds of users, some requirements need special treatment.

• The role is an occupation with an unpredictable schedule. Network administrators or emergency room doctors or can’t leave their workplace, and they commit spare time reluctantly. PC technicians can also have unpredictable work demands that will take precedence over any commitment to us.
• The product depends on domain expertise that only a few people have, such as molecular modelers or developers of distance-learning courses, or is used by only a small group of specialists.
• The product is not linked to any specific profession or skill, so even defining target users is problematical.

Some Sources Are Better than Others

After 10 years of finding users, we’re pretty confident we’ve tried every method there is. One thing we know is that there are some sure-fire ways to find users and some that we never want to try again.

Good Sources

We’ve had lots of luck with the following sources:

• People we’ve tested before. This is our best source of new users. They know what’s involved in testing, and will vouch for the fact that it’s fun and valuable. We reward people who find us users with T-shirts and lots of gratitude. These previous users are also our best users, and we re-use them when we can—a more substantial reward.
• User groups, professional associations, and other organizations. These groups often give us five minutes at their meetings to talk about testing and to recruit users. Members of these groups are interested in new products and product improvement, and, because they regularly go to meetings outside of work, are excellent at managing their own time and meeting our schedule.
• Real estate agents. When we’re really stuck, we contact friends in the real estate business. Typically these people know what everybody does for work and what kind of technological toys they have in their houses—all sorts of information we can use to identify the users we need.

Sources that Just Don’t Work

We’ve also found that some sources aren’t worth the time or effort. You might want to avoid the following recruitment strategies:

• Newspaper ads or job center postings. It’s tempting to use these, but we haven’t found them effective. One ad netted us 200 responses from interested people, but many did not match the qualifications we specified in the ad or responded after we’d finished the testing.
• Mass e-mailings. These also pose problems. Even assuming that the recipients expect e-mail and don’t mind the mild spamming, we’ve never had much luck. Once they reply, each recipient expects to be able to test and expects a response. Scheduling can take four or five messages back and forth.

Using Surrogate Users

In early tests, we sometimes test surrogate users when the real users are difficult to find. This can help eliminate gross imperfections in a product that might irritate the real users and waste their time. Testing with surrogates is okay as long as you realize that they may bring biases that your actual audience would not have.

For example, when we needed to test ER physicians, we first used medical students. When we tested subsequently with real physicians, we found that the students had helped us refine the design, but their lack of knowledge about real life did send us down the wrong track in one or two places.

We sometimes use existing customers as surrogates. Some of the biases the bring include knowing a lot about the product or company, and having a predefined way of working. In addition, when we test existing customers, we always involve sales and marketing in the testing and contact. We also ensure that customers understand the exploratory nature of the testing, and that features they like or suggest may not turn up in the final product.

Employees of your company are at best adequate surrogates for testing your product. They can catch obvious design gaffes before real testing begins, but because they work for the company, many factors may inhibit their honesty and frankness. Also, they probably know a lot more about your target audience that your actual users do.

Making the Pitch

We’ve found it invaluable to learn something about the users we want before contacting them. Knowing a little of their jargon and their time and work constraints makes it easier for us to offer something they can agree to. For example, once we learned that bond-trading desks usually close at 3:00, we scheduled the testing for 4:00 and had much more success finding willing participants.

Although specialized users may be hard to locate, we’ve found them easy to schedule because of their interest in the product.

We do pay users—usually $50 an hour—but they don’t generally do it for the money. This goes double for hard-to-find users, who are generally highly paid in their work. However, sparing the user inconvenience, such as paying for parking and gas money, if it seems warranted, always helps.

On the other hand, we don’t usually go out of our way to accommodate users’ schedules. Most take our timetable seriously and meet it—or recommend someone who can.

The Best Inducements

Most qualified users want to test. However, to commit their time, busy people want (and like) to hear exactly why we want them for this testing. They also want to know what they’ll get from testing this product. Among the things we tell them:

• “We need your input.” We show this by the way we ask qualifying questions.
• “Your colleagues are testing this product.” This works well in small specialist groups.
• “You’ll get to meet the developers, the people who produce the software you depend on.”
• “You can see—and even influence—what’s new in the field.”

If these don’t work we don’t schedule the user. Users who test for other reasons, such as money, can’t be counted on for the kind of participation we need for improving a product.