Using Paper Prototypes to Manage Risk

There are no rewards in life for being the first one with the wrong answer. Imagine spending years building a product, only to learn that it missed the needs of its intended market. History is littered with the carcasses of failed products and the companies that built them—product development is indeed a risky business.

We’re a consulting firm specializing in user interface design and usability issues, so our initial contact with new clients is usually when they realize their product is in trouble. Here’s a typical scenario: We got a call from the marketing manager for a commercial software product. The first release of the product had been around for a year, but sales had been only mediocre. Although customers liked the functionality, they complained that the software was difficult to use (the manager admitted that even people within the company had trouble using it!). The manager was worried that these problems were going to hurt sales of the next release, and that support costs would escalate. To manage these risks, the company needed to do something differently.

The company had spent considerable time and money to produce their first release, just to learn that they hadn’t quite gotten it right. In a sense, that first release was actually a “prototype” for the next release, a prototype that had taken many, many developer-years to build. Because their market had become increasingly competitive, the company couldn’t afford another so-so release. And they couldn’t wait years or even months for that key feedback—they needed it before their beta test, which was only a few weeks away.

That’s exactly what we helped them do. Using the techniques of paper prototyping and usability testing, we showed the team how to get quick feedback on their product before they had invested a lot of development effort. Total time required: about 6 days.

Day 1

We met with the entire development team. Like every other team we’ve ever worked with, these people already had a rich set of opinions (and sometimes, heated arguments!) about what was wrong with the interface and how to fix it. The trouble was, they had very little actual data on which problems were most critical to users or on which of the potential solutions (none of which had been implemented) were actually going to solve the problems. Without this data, there was a considerable risk that the team would implement the wrong solutions or even worse, remain ignorant of where the real problems lay.

The first thing we did with the team was to agree on a profile of their most important type of user and the most important things that those users did with the product. There was no time in the schedule to exhaustively test every aspect of the product, so we had to focus on the riskiest parts—the things that were going to make or break the next release of the product. In this case, the team members agreed that the biggest risk was that the product was so hard to use that it would make a negative impression on the technical buyer. As another example, in an insurance claims processing system, the biggest risk might be “Will users accept the way we’ve re-engineered the work flow?” If the team doesn’t know where the risks lie, that in itself is probably the biggest risk!

Days 2 and 3

We started looking for people who fit the user profile to bring in for the usability test sessions. Meanwhile, we showed the team how to construct a paper prototype of the product interface so we could get some data about those high-risk areas. Using common office supplies (markers, index cards, scissors, transparency film) the development team quickly sketched each element of the interface (screen, menu, error message) on a separate piece of paper.

We completed the prototype by the end of Day 3. It wasn’t very neat—it had hand-written text, crooked lines and last-minute corrections—but it was good enough to show what the screens would look like.

Days 4 and 5

We picked one developer to play the role of “computer,” simulating the behavior of the software by manipulating the pieces of paper. For example, if the user wanted to open a document in a Windows-based word processor, they would touch the word “File” on the menu bar. The “computer” would respond by putting down a piece of paper containing the File menu. The user would then touch the Open option, the computer would display the Open dialog box.

We spent 2 days usability testing the paper prototype. In each test session, we brought in two people who matched the profile of a typical user. We didn’t give them a demo or explain how to use the interface—instead, we asked them to do realistic work with the prototype. The computer wasn’t allowed to explain anything to users, but could only do what the software was designed to do.

We didn’t ask the users for their opinions of the interface; we watched them actually work with it (we’ve found that sometimes people say they love a product even though they can’t use it, or vice versa). The entire development team observed the sessions, taking notes on where the interface was giving the users problems.

The team was surprised by many of the issues they saw. In some cases, we found that developers had been arguing about something that the users didn’t even notice! At the same time, there were huge problems in the interface that no one had anticipated. Usability testing gave everyone on the team a sense of what the real issues were; the ones that would affect the success of the next release.

After each 2-hour test, the team discussed what they had seen and immediately made changes to the paper prototype. With a paper prototype, the entire team could collaborate on the design rather than funneling all the input into one or two programmers. Team members from all disciplines (engineering, marketing, technical communications, training and management) picked up the markers and scissors and contributed their expertise directly to the paper prototype.

Modifying a paper prototype is much less painful than for the developers than modifying an actual product. With a real product, because of the substantial amount of work they’ve put in, the team has an emotional investment in the status quo and will naturally tend to “defend” their design. Even when the team clearly understands the need for changes, it’s tough to throw away all that hard work. In contrast, because paper prototypes are so easy to create and modify, there is less invested effort to defend. As a result, development teams become more flexible and willing to try new ideas. This team spent about 2 hours making changes.

After lunch, we brought in 2 different users and had them do the same tasks, to see whether we’d made the product better or worse. Most of the changes proved to be improvements. Some changes were as simple as using a different word, moving a button from the top of the screen to the bottom or putting an example in a help file. Other changes were more substantial—we’ve seen teams completely redesign their products, literally overnight, after learning that their current design was headed for disaster.

Day 6

After 2 days of testing, the team had identified and addressed many of the biggest problems in the interface. On Day 6, we again met with the entire team to prioritize all the issues that remained in the product, using data—not opinions—that we had obtained from the usability tests. Given their tight schedule (they were a month away from beta testing), the team knew where they had to focus their efforts in order to make the next release a success. On the seventh day, the team rested.

By using paper prototyping and usability testing, the development team managed their risks by focusing on them earlier in the project, while there was still time to make changes. We had started the process by having the team pick those areas which they had perceived as most critical to the success of the next release—the areas of highest risk. Now, 6 days later, they had actual data on how to make the product better in those areas. Our question for the team was “OK, so now what’s your next greatest risk?” The team now understood how to use paper prototyping and usability testing to provide the data they needed for key design decisions that enabled them to build a better product.