Avoiding Demographics When Recruiting Participants: An Interview with Dana Chisnell

User research works best when you match your participants to the people who will use your designs. It makes sense that teams would try to use the demographics, often compiled by the organization’s market research team, as the basis of their recruiting efforts. However, this can be problematic.

To explore why this might not be such a good idea, I recently talked with usability expert, Dana Chisnell. Dana is the co-author of the recently published second edition of the Handbook of Usability Testing, and runs UsabilityWorks, a San Francisco usability research consultancy. Dana’s organization recruits hundreds of participants every month for teams all over the world, so she is well familiar with the traps of using demographics. Here’s what she had to say about it:

UIE: Recently, we’ve had a bunch of clients come to our doorsteps thinking they know who they should be recruiting for their usability tests. But what they really have are demographics, such as “70% are males between the ages of 18 and 24.” I thought we could talk about why demographics are the wrong way to think about getting test participants.

Do clients come to you with demographics as a description of their ideal usability test participant?

Dana Chisnell: This happens all the time. Especially if marketing or market research is sponsoring the study you’re recruiting for, it can be really hard to break out of matching market segments.

Recently, a client came to us (my recruiting consultant, Sandy Olson, and me) wanting to bring people into the lab to compare PC security software. They gave us a screener with percentages. How do you get percentages of individuals?

They wanted 25% of their 24 participants to be between the ages of 30 and 39. They wanted 50% of their participants to be female, 50% to be male. They wanted participants who were professionals, broken down into nine categories. One category was “Banking, investments, and real estate.” Another was “Engineering, other.” There was also “Education, training, students.”

If they’ve done the research that tells them a quarter of their audience are in their 30s, there is a logic to wanting to recruit participants that way. Why did you feel this approach wasn’t going to work for them?

None of those percentages had anything to do with what they wanted participants to do in the study. None of those percentages said anything about how motivated the people we selected might be to do the tasks that were part of the study.

For example, age is not necessarily an indicator of behavior, performance, or expertise—the attributes that make a difference in the results of a usability results. Just because someone is in her 60s, doesn’t mean that she’s more or less technologically savvy or more or less security conscious than someone in his 30s or 40s. Being 20 doesn’t make you an expert computer user. Being 70 doesn’t mean that you don’t know how to use a computer.

If we wanted to see how different behaviors, performance, and expertise affects how people will use this application, we needed to look beyond age. Professions won’t help either. The client gave us a law enforcement category, but otherwise it’s unlikely that someone who is a teacher is more likely to be security conscious than someone who is in real estate.

But, in many organizations, the market research groups invest a lot of time and money in developing these demographic descriptions. Isn’t using this for recruiting exactly the reason why they were doing this? Shouldn’t this be the starting point?

Market research groups generally develop segmentation descriptions to split their market up into sub-markets to address advertising needs. Often, the segments get used to recruiting for focus groups. This is fine if you’re doing focus groups to explore market- or advertising-related issues. And, it works well if you’re conducting lots of focus groups so you can generalize preferences. However, it’s not so good if you want to learn about how well a design works for real people.

At the beginning of a user experience research project, it’s okay to talk about what the segments look like. The segments are one view of a user profile; personas are another. But they’re based on totally different data. Segments come from self-reported survey data, usually. Personas should be developed after observing users in their own contexts doing their own tasks.

In some organizations, the User Experience people work to mesh those types of data together, which can sometimes work well. However, if I have to choose between self-reported segment data or information from observing real people, I’d rather bring in the latter as that’s more likely to get me the results in a usability test that will help the team improve the design.

What kinds of problems do teams run into if they focus on demographics?

A few years ago, I did a study for AARP on the AARP.org web site. AARP is an organization for Americans over age 50. Among other things, AARP was interested in learning about how well their message boards, of which there were dozens active, worked for typical older adults.

We conducted a usability test in three different locations with 20 participants in each location. In the first location, we recruited based on segments. We recruited 6 people in their 50s, 8 people in their 60s, 4 people in their 70s, and 2 in their 80s. AARP is about age, after all. We did not select for what people did online.

When we got to the section of the test where we wanted people to do tasks with the message boards, we found that across the age brackets, most participants had not used message boards before and didn’t want to. Many simply refused to do the task. I asked them to do the tasks anyway. Guess what? The data wasn’t valuable. Message boards were not successful with these people because these people were not motivated to do the task.

By the time we got to the third location for the study, we’d adjusted our selection criteria. Though we still recruited people in a range of ages, we also recruited based on what they had done recently online, including shopping, banking, research, and various forms of communicating, including participating in message boards.

When we got to the message board tasks in the test, we asked participants who had experience with message boards to do those tasks. We could see how they interacted with the UI and they could give their expert opinion about what they liked and didn’t like about how the boards worked. We learned much more and gathered better data because these people were motivated to do the task rather than just going through the mechanical steps without understanding what they were doing or why.

If the security product team shouldn’t focus on demographics, what did you recommend they do to decide who to recruit?

We asked the client to think about what their reasons for doing the study. They said that they wanted to find out whether existing users were aware of all of the security statuses the client software presented. If users weren’t aware of the statuses, the client wanted to find out what they should do about that.

Next, we suggested they write a one sentence description of the participant they were looking for. This became something like, “We want to observe people who are security conscious and tech savvy and who buy online often.”

This made it much easier for us to recruit for this study because as you can see, there’s nothing about age, gender, or profession in that visualization. Instead, it was about what people did and their motivations for doing it.