Docs in the Real World

In two recent consulting projects, we worked with online documentation developers who wanted to understand the problems users encountered and how their documentation helped solve those problems. To find out, we went and observed users in their own work environments. Although the clients and their software differ significantly, we found similar issues.

Both development teams initially had specific things they wanted to know about how people used the documentation, such as whether the structure and organization made sense to users, and how often users went to the wrong place first.

By watching users in their own workplace, we found issues that we would never have seen in a usability lab. These observations revealed many larger issues surrounding the use of the documentation.

We Are Not Alone

For example, neither team expected to see the variety and volume of other resources that “compete” with the official documentation. These included such diverse sources as co-workers, including cubicle-mates and the internal help-desk, users’ own notes, web sites, training materials, online news, and team meetings.

It surprised both teams to learn how much users relied on these competing forms of information. They found that the most successful users had the most resources available and knew when to use each one. No user relied exclusively on the online documentation.

The teams realized that this competition wasn’t always a problem. For example, users at one company benefited by having a printed list of field codes for the mainframe; this information never changed and users often needed it quickly. (It surprised this team to learn how important the mainframe procedures were: the company was phasing out the mainframe, so the team hadn’t focused on supporting it.) In contrast, frequently updated sources such as printed price schedules quickly became obsolete and many users still had old ones.

Therefore, this team abandoned its original assumption that it had to document everything in one online source, and decided to look for the most effective way to convey each type of information. They realized they needed to coordinate with the training department on how to present the information, including instruction on using each of the available resources.

Trust-Breakers

We repeatedly observed users avoiding documentation because they’d learned not to trust it. We saw many reasons for this loss of trust, some of them seemingly trivial.

Every Error Counts

Even small inaccuracies damaged the confidence of some users. This was more likely with users who didn’t know how to work around the problem. For example, users were stopped in their tracks when the online procedures told them to press Enter—but neglected to tell them they had to press it twice.

Many of these users were not proficient with computers, so even this small error was enough to make them abandon the documentation and call the help desk. In contrast, errors in documented policy didn’t affect trust much, because the support reps usually had more proficiency with the policies—at least a general idea of what the policy was supposed to be—and could figure out what to tell the customer.

The Buck Stops Here

Users don’t care where the problem lies: they don’t recognize that online help involves different pieces from different sources. Although some problems were beyond the doc team’s control, such as a bookmarks that didn’t work or a bug in Microsoft’s help engine, they still affect the users’ perception of the documentation—and their willingness to use it.

Fixing Broken Trust

It’s easy to break trust, and difficult to restore it. Once they’re burned by the docs, users typically won’t look there again. Unfortunately, this behavior can persist even after developers fix a problem. For example, although one team corrected an error that appeared in Version 2.0 of the docs, users still didn’t try to use it in Version 4.0. No one had told them that the error was gone.

When trust takes a beating, it’s hard to know the effect. After a bad experience, users may be willing to use other parts of the doc—and sometimes are not. Our observations showed us that the development team needs a concerted, proactive marketing effort to restore broken trust.

Trust isn’t an all-or-nothing thing. It’s built up slowly by successful experiences, and quickly eroded by disappointments. One team had translated the documentation into English from its original German—but unintentionally left pockets of German. Users who encountered German immediately stopped looking for the answer (getting German usually meant they were in the wrong place). Every time they encountered German, users were less likely to rely on the rest of the documentation.

Trust and Communication

One important element of building trust is the way the documentation team handles feedback. One team had an internal feed-back line to let users report errors and request changes. But the team was understaffed, so users rarely got a response, even an acknowledgment. As a result, users became reluctant to report problems.

The team realized it needed to put a response system in place—even if users just got a quick generic thank-you—before it could expect users to report problems.

Speed Is Relative

Users sometimes avoided documentation if they felt that it took too long to use. Actual speed, the measurable time it takes to get the answer, often wasn’t a problem. Instead, perceived lack of speed, how long users think the process will take, prevented users from using the docs. And what is “acceptable” varies from user to user, even from situation to situation: with an irate customer on the line, seconds can seem like hours.

In one project, we saw an interesting Catch-22: support reps didn’t look up anything in the online documentation unless they already knew where to find it. But they never learned where the information was unless they spent time using the docs. This perception of slow access speed wasn’t caused by poor organization of the documentation, but by the sheer volume of information, which made the online documentation daunting to the uninitiated.

On the other hand, this team was pleased that users generally could find things, and that users looked in the wrong place far less often than they’d feared. So, instead of launching a reorganization, the team proposed some other changes. These included getting the management to give reps off-phone time to do research (difficult during busy season), having managers and help-desk staff encourage reps to look things up first, and devoting more training time to practice sessions with the online documentation.

Sell, Sell, Sell

The documentation can still fail, even if it’s as usable as possible. Just because you’ve built it doesn’t mean they will come. We saw a clear pattern: users generally didn’t know about new features after they’d been trained.

It amazed both teams when they discovered how many users didn’t know about some basic features. For example, one team learned that no one had trained users to press F1 to get help. (Users got very excited when we showed them this trick!)

The other team found that users never went to the tutorials or used the Back feature—which the team had spent lots of time developing—because the team had never told the users about them.

These findings initially discouraged the teams, until they saw it as a “marketing” problem: determining what the messages they wanted to convey to users about the documentation, and then finding the appropriate channels.

After seeking feedback from successful reps, one team learned that the most appropriate and effective marketing message was something like: “The online documentation is huge and overwhelming at first. But if you invest some effort in learning it, it will pay off big, making you more proficient in the job skills you’re rewarded for.”

To spread this message, the team decided that managers and the training department provided the best communication channels.

It Helped to Watch Users

Both teams learned that the documentation was only part of the picture of how people do their work.

Perfect documents don’t help users succeed unless the team accounts for all these other factors. Because they made these site visits, the teams came up with some changes that showed great promise for improving users’ success.