Usability Labs: Our Take

by Tara Scanlon

Many clients ask us whether they should build a usability lab—a special set of rooms dedicated to usability testing, often with a one-way mirror and sophisticated AV equipment.

We tell them, well… it depends.

Though we do hundreds of usability tests every year, we don’t have a lab. We do most of our testing in conference rooms and offices ? or sometimes less than that. Last summer, we rented a tent in the middle of a field and successfully tested 91 users in it.

This doesn’t mean we disagree with people who place great value in usability labs. We just have a different perspective.

On the One Hand…

There’s little question that a usability lab is a good place to test products. There are clear advantages to having everything in one place and ready to go whenever testing is scheduled. Having the lab on-site also makes it easier for the whole team to attend testing sessions.

A usability lab can be a powerful political tool, visible evidence that the company is willing to spend money and effort to make its products usable.

This need to be visible can encourage developers to make sure upper management knows the lab is being used. At Lotus, for example, the lab is in a well-trafficked part of the building and an LCD panel outside the door announces what’s being tested today. We know one leader at another company who holds all his usability team meetings in the lab, just so his boss remembers that the lab is there.

Furthermore, in an era where almost no one has a real office, labs represent golden real estate. (They even have doors!) We visited one company where the usability lab is literally in a tower that stands above a sea of cubicles.

…On the Other Hand

Some advocates see labs as the only place to garner scientifically valid data. We’re not sure that’s true or necessary. Usability testing is not really a statistical exercise, and experience has shown us that the first six or eight users uncover the majority of serious usability problems—even without a fancy lab.

Even with a lab, if it’s busy or if someone doesn’t know how to use it, there’s often the perception that testing can’t be done. This can limit the amount of usability testing.

Not only that, but usability labs are expensive. With a limited budget, money set aside to build (or lease) a lab is money that can’t be used to do more testing; it’s a matter of setting priorities.

Usability testing, with or without a lab, always puts users into an unrealistic setting—by definition.

In a lab, we’ve found that users sometimes worry about who’s behind that one-way mirror and what those folks are doing. This fear may have merit: we know about some misbehavior behind the one-way mirror, such as playing with the camera controls instead of watching the test, or using the invisibility to catch up on sleep or to tell other observers what to think about the ongoing test.

By contrast, we’ve found that with our no-lab setup, once they begin the tests, users quickly ignore observers who are sitting in the same room with them. And these observers are forced to pay close attention to what’s happening (or at least not distract the users).

Life Without a Lab

We’ve never felt we needed a permanent usability lab. Our testing setup is simple and portable, and it adapts well to available space at our office or at clients’ conference rooms.

We seat the user (or users) at the computer or prototype with the facilitator sitting nearby. Observers from the development team sit where they can watch the participant work, and they all have cards or note paper to write down their observations and questions. The video camera, when we use it, records the screen, what the user says, and part of the user’s profile. We sometimes mount the camera tripod on a nearby desk to give us an unobstructed view.

People often ask us if we have trouble controlling observers who are in the same room as the test participants. This isn’t as much of a problem as you might think. Even the most rambunctious team members behave when they’re sharing the room with people who might use their software. But to make sure the users aren’t harassed, we also enforce strict rules for the observers.

Published here on July 1, 1999.