The CAA: A Wicked Good Design Technique

Here, in the nether-regions of the Greater Boston area, we have a linguistic habit of leaving the letter ‘R’ out of words like ‘car’ (pronounced caa) or ‘Harvard’ (pronounced haavaad). It’s what makes us special.

Around our office, CAA has another meaning: Category Agreement Analysis. This turns out to be a ‘wicked good’ (another New England-ism) technique to help designers arrive at a usable information architecture.

CAA, like its big brother, card sorting, helps determine what the top-level categories for a site should be. Also like card sorting, CAA tells us what category structure is most natural to users.

Card sorting, which involves writing the content on index cards and asking users to sort them into logical piles, with the piles becoming the architecture of the site, tells us how to organize content. CAA is similar, except there aren’t any cards.

Instead, the primary CAA instrument is a survey. To prepare the survey, you write down the most important content elements in a list. For example, if you are working on a technical support site, you might list all the different types of questions people want answered (possibly gathered from your call center logs). If you’re working on an intranet site, you might list content elements, such as the travel expense policy or critical sales information.

The survey consists of each item listed separately in the left column. In the middle column, you place a box for the user to fill in which you label “Main Category.” In the right column, you place another fill-in box labeled “Second-level Category (if necessary)”.

Then, to complete your data collection, all you need to do is distribute the survey to users. We find that you can get decent results from the survey with 100 users, but 300-400 is better. In addition to getting more users, varying your distribution techniques (such as using remote offices or handing out surveys in your retail stores) will help you get the perspectives of the broader community.

Once you’ve collected the data, analysis is quite simple. For each element in your content list, you count the different categories that users suggested, looking to see if you get 70% or better agreement on the top 2 or 3 terms.

For example, when we asked people where they’d expect to find Iguana Food on a pet supply site, 53% told us they’d expect the major category to be “Reptile”. (We were pleasantly surprised that 53% of Americans knew that iguanas were reptiles.) Another 20% suggested “Food”. The 73% agreement of these two terms told us that a category called “Reptile Food and Supplies” would do very well.

Contrast that with what people told us when we asked for the major categories for “Recordable Audio CDs” at a site selling computer supplies. The most agreement we got was 19% for “Accessories”. (Almost everything a computer supplies site sells, such as cartridges, mice, and cables, can be called “accessories”.) 16% said “Storage”, 13% said “CDs”, another 13% said “Media”, and 10% said “Audio”. There was no consensus on any of the categories.

When you have consensus on one or two category names, the users are telling you clearly what the category for that content should be. Our research shows that you can use that category with confidence because users will understand what is found when they click on it.

However, as in the case of our recordable audio CD’s, if you don’t have consensus, you need to gravitate to a different strategy, where you explain in the categories what users will find underneath. In essence, your top-level design has to teach users where they can find the specific content they are seeking.

If you’ve done card sorting before, CAA is different in two ways. First, a typical approach to card sorting is you list the categories and ask users to sort under them. Another popular alternative is that you have users name the categories after they’ve put cards in the piles.

Instead of presenting the user with a notion of structure and telling them the names of categories, CAA is more fluid. It uses a spontaneous word-association style, not restricting the users to designing a category structure. With CAA, you’re looking to measure the agreement on terms, not create the design of entire hierarchy.

Second, CAA uses far more participants. Card sorting, which is time intensive to set up and run, is really only practical with a dozen or so participants. CAA takes advantages of sampling a larger portion of the population.

Both techniques complement each other and you can use them together. When creating a new design, you might start with CAA to get a sense as to what you’re dealing with—do users have agreement on the top-level terms? Then, if they do, you can use card sorting to help work out the details of the hierarchy.

CAA is a good way to identify the key terms that users think of when dealing with the content on your site. You can quickly employ this easy technique without much setup or analysis cost. It’s a wicked good tool to put into your designer’s toolbox.