Evolution Trumps Usability Guidelines

Use a Search Box instead of a link to a Search page.

This is one guideline from the plethora of recently created usability guidelines to help designers produce more usable web sites. It makes sense. After all, there are more than 42 million web sites on the Internet. It should be simple to study these sites and put together a list of “do’s” and “don’ts” that, when followed, will produce easy-to-use sites.

Designing a web site, either usable or unusable, is hard work. There are many details that designers need to take into account, such as browser differences, content management, information architecture, and graphic design. Providing proven guidelines to developers can reduce their already overburdened workload, making one aspect of design that much simpler.

However, we are assuming the guidelines actually result in more usable sites. This is where things start to get murky.

For example, the guideline above suggests that access to your site’s Search functionality should be through a type-in box, not a link. We’ve seen various forms of this guideline in multiple places. The stated implication always is users are less likely to find Search if it is only a link to a separate search page and therefore users are less likely to succeed at accomplishing their goals.

What’s most interesting is that the guideline’s publishers never present any evidence that following it will actually improve the site. The best we’ve seen is one publisher who stated that on *their* site, when they changed the link to a type-in box, the use of Search increased 91%.

While 91% seems like a lot, if only 1.5% of the site’s total visitors originally used Search, a 91% increase would only bring it up to 2.9%—still unused by 97.1% of the folks visiting the site. In addition, that publisher states that they don’t know if people actually found the information they were seeking because of the change. Given this, is the guideline worth following?

Testing Guidelines

In the last 18 months, we’ve been testing guidelines against our growing collection of usability data. We can see which guidelines actually predict better results.

Testing a guideline is usually straightforward. We can translate a well-written guideline into a hypothesis and test it against our data. For example, we can look at e-commerce sites that have Search boxes on every page versus those sites that don’t and see if the presence of the Search box predicts that shoppers will buy more.

However, not all guidelines are written well. For example, one publisher of more than 200 e-commerce guidelines, had many guidelines we couldn’t test because they were not measurable, such as ”Make sure your checkout form fields are written clearly.“ When a guideline has a subjective standard (how does a designer know when something is ‘written clearly’?), we can’t form a measurable hypothesis to test.

Case study: Expected Placement of Design Elements

When we test guidelines, we find fascinating results. Recently, we looked at the standard placement of design elements for e-commerce sites. Our work was based on the guidelines put forward by Michael Bernard of the Software Usability Research Laboratory at Wichita State University in his wonderful paper: Examining User Expectations for the Location of Common E-Commerce Web Objects.

In this paper, he analyzed where users expected certain e-commerce elements, such as the shopping cart, Search, product categories, and login, to appear on the page. He found that his test users were very consistent in their expectations. From this, he naturally recommended that designers place these common elements in the locations that users were most likely to expect.

Using his guidelines, we constructed the following hypothesis: if sites place these common elements where users expect, we should see users purchasing more from the sites. With our hypothesis in hand, we went to our data.

The data we used for this analysis consists of more than 1,000 shopping expeditions on 13 sites with 44 users. Each user had made a list of products they needed to purchase, we gave them the money, and put them on sites that had the products they needed. Anytime users didn’t purchase their products, it was because of a problem with the design of the site.

When we looked at the 13 sites, we found that there was great variation on the placement of the common elements. Some sites placed these elements right where the users said they expected them, but others scattered them in every possible part of the screen.

Given this, we expected that sites matching the users’ expectations would have more sales. However, when we ran the analysis we found it wasn’t true. The sites that ignored the ‘expected placement’ of elements sold just as much product as those that matched it precisely.

Not only that, but users’ ratings of the sites didn’t matter either. According to the users, sites that didn’t place elements in the standard locations were just as easy, fun, and professional as those that did. In fact, the location of the elements had no effect on users’ rating of ‘The site met my expectations’.

Therefore, while users have a consistent expectation where designers will place the basic design elements, there’s no evidence to suggest that designers should place these elements there. So, the guidelines proposed in Michael’s paper are not likely to help designers.

Knowing Which Guidelines to Follow

By testing guidelines, we can identify those that will provide the best value. As we’ve been studying guidelines published by various folks, we’ve noticed something very interesting: web usability guidelines are very sensitive to the nature of the tasks and the subtle differences in the content of the site.

We see that a guideline’s effectiveness will change when the wording of the usability test’s tasks is changed slightly. For example, if we tell shoppers to buy a sweater, whether they are interested in buying one or not, we see different results from when we ask them to buy something they really need.

We’ve also noticed that the content dictates whether a guideline will pass the test. Many guidelines for Search, for example, depend on whether the content is uniquely identified (like books or proposed legislation) or not (like apparel or disease symptoms). (See Strategies for Categorizing Categories for more information about uniquely identified content.)

If the tasks we use or the nature of the content can affect whether a guideline passes the test, then that means that we won’t have many generalized guidelines which are always applicable. We need to conduct more research to understand what variables make each guideline effective, so that designers understand the context that makes the guidelines valuable.

Following Untested Guidelines Possibly Harmful

Sites still need designing while we’re waiting for the research. One option is to follow the guidelines as they stand. After all, smart people are producing these guidelines—they’ve got to be worth something.

Unfortunately, three outcomes result from testing guidelines. The guideline, when followed, either predicts success, doesn’t have any effect on success, or, (the worst case scenario) makes the site harder to use.

In a very recent study of some guidelines, we found that following the guidelines actually predicted failure in the clickstreams. For example, one published Search guideline is “Provide a clearly marked link to Advanced Search”. From this guideline, we hypothesized that users who found Advanced Search would succeed more often than those who didn’t.

However, in our data, we found that users who utilized advanced search rarely found their desired content. Use of advanced search functionality actually predicts failure of the task.

It turns out that following this guideline (making advanced search more visible) will potentially decrease the usability of the site. Those designs that didn’t provide advanced search actually did better in our testing.

This means that following untested guidelines is like drinking water from an unidentified source. It might quench your thirst, but it could also make you very ill.

A Practical Alternative to Guidelines: Evolution

If you shouldn’t follow untested guidelines, what should you do? Fortunately, evolution can help.

For most types of sites, there are existing models already out there. Someone who wants to produce a new site with pharmaceutical information, for example, can find dozens of existing pharmaceutical sites.

A small investment in studying how users interact with existing sites can reveal a lot about what works for your users on their tasks. You could easily develop an understanding of the ‘best practices’ and, from that, produce your own guidelines.

Because you will generate your guidelines by directly observing your users, these guidelines are far more likely to be of value than generalized guidelines produced from sites that have little or nothing to do with your work. Evolution has produced these sites and you can identify which have won the ‘survival of the fittest’ competition.

Even if you don’t have many comparable sites to look at, you can still use evolution to your advantage. Create your own mutations by trying out ideas and seeing how they work.

This is exactly how sites like eBay and Amazon have gotten to where they are today. They’ll isolate one of their many servers and change the design of a few pages, just for users of that server. They’ll compare the results with users of the other servers running the ‘existing site’. Because of their traffic volume, they can learn a lot in just a few hours.

You may need to leave things a little longer on your site, but the effect is still the same. You’ll quickly learn what works and what doesn’t, evolving your site into something that meets both the needs of your organization and your users.