Why On-Site Searching Stinks

In our most recent web-site studies, we watched users look for information within web sites. Our goal was to gather data about what makes a good link, but we did not tell users whether or not they should use the site’s search facilities. Users went to these search engines in almost half the tasks. Maybe they shouldn’t have.

Searching Reduces Success

Using an on-site search engine actually reduced the chances of success, and the difference was significant. Overall, users found the correct answer in 42% of the tests. When they used an on-site search engine (we did not study Internet search engines), their success rate was only 30%. In tasks where they used only links, however, users succeeded 53% of the time.

Garbage In/Garbage Out

A search engine’s results can only be as good as the inputs it receives. Our tests showed that users didn’t know what to type in a search edit box, or they didn’t know how to use the format the search engine expected.

Partial Matches

Users didn’t seem to understand that some search engines distinguish between partial and entire words. On the Car Talk site (cartalk.msn.com) they didn’t realize that a search for “tire” would give them results that contained the word entire or the phrase I’m tired.

Although the site gives users the option of searching for whole or partial words, no one changed the setting. Similarly, we saw that users were puzzled when plural and singular words produced different results. Prefix/suffix strippers might improve the quality of the search inputs.

Few users in our studies clicked to bring up search tips that were located on another page. We believe this shows that few users read search instructions, especially instructions that appear below the keyword box.

Misspelled Keywords

Lots of users misspell keywords. An analysis of a week’s worth of logs from Netscape’s DevEdge Online site showed that 3% of searches contained misspelled words—we saw 16 different misspellings for “JavaScript” alone. In most cases, the user was only off by a single letter:

jacascript javascipt javescript jvascript

javasrcipt javascritp javasctipt javascrcipt

javasript javescript javascrip javscript

javascfript javascripyt javasecipt javassript

Reasons for spelling errors include unfamiliarity with the English language or technical terms, difficulty in spelling, or simply hitting the wrong key. Users didn’t always know that typing errors would produce poor results and couldn’t tell that the “no matches found” message was caused by a typo.

A spell-checker would weed out some problems, especially if it suggested alternate spellings (Microsoft Word’s spell-checker is smart enough to suggest “videos” as a possibility when you type “vidoes.”)

Problems Interpreting Search Results

Correctly asking for the right information is only half the battle. Many users had problems even after they got the search results.

Unclear Relevancy

Users often had trouble determining why a search returned a particular item. Many search results offer no clue and can look like the example here from a search for “money market” on the Fidelity site (www.fidelity.com). Car Talk does better because it shows keywords in the context of a line, so users could weed out obviously irrelevant hits in the search results.

Some on-site engines generate links by looking at page title such as the result 6 from Fidelity, above, “funddata_toc.html.” In this case, the designer apparently did not use the HTML <title> tag. When designers create page titles, they may not be thinking about how those titles will look when they turn up in search results. Taking the time to create more descriptive page titles might help. This is one of the few search engine problems designers can control.

The Perils of Full-Text Searches

Most sites use a full-text search, which sometimes returned irrelevant and often amusing results. In a keyword search for “dinosaur” on the Smithsonian Magazine site, the first result led to an article about the steel industry!

We saw many other instances of this problem. Searching for “wild west” on the Discovery site returned a link to an article on sharks—though the article did indeed contain both those words.

A full-text search is a blunt instrument for chipping away at a large block of information in order to sculpt the desired result. An index is a more precise tool. No self-respecting human indexer would have referenced the steel industry article under “dinosaur.” Good indexing is a skill; humans do it better than machines. We anticipate that professional indexers may become more involved in web site design in the future. This may be difficult and expensive to implement, but the resulting user satisfaction might make it worthwhile.

Worse Than Nothing

Our data showed that today’s on-site search engines are worse than nothing—significantly worse. Searching is a difficult problem with no solution visible on the horizon. Until the technology is equal to the challenge, we suggest that designers seriously consider not including a search engine on their sites until the technology is equal to the challenge.

In the meantime, our testing data suggests that designers would have more success by focusing instead on creating effective links.