In this short read we will look at when card sorting is your best UX research friend and some ‘red flags’ to look for before committing to a navigation design based on card sorting results alone.
What is Card Sorting?
Firstly, what is a ‘card sort’? Basically, a card sort is when you ask some people to take a bunch of things or concepts (typically representing pages on a website) and place them into groups. In an ‘open’ card sort, the participant will name those groups themselves. In a ‘closed’ card sort, the categories are already provided and you’ll be asking people to put things into the category that makes most sense to them.
As a research method, card sorting is most often used when validating a new navigation design or when researching the effectiveness of an existing one. Card sorting can be done online or in person, using paper cards or a digital platform, and moderated (where you can ask the participant questions) or unmoderated (where a participant completes the task on their own).
Card sort data can show you where strong correlations exist between the items being sorted and categories, and where there is ambiguity. The insights are really useful in deciding on approaches to the information architecture of a site: things like using a poly vs mono-hierarchical navigation structure, presenting a single navigation or differentiating options for different users or contexts, and all sorts of other tricksy aspects of designing an effective navigation.
When card sorting is not a good match
Card sorting has become an industry standard approach to information architecture, but there are times when you shouldn’t trust the outcomes without additional research. Next we’ll look at some of the ‘red flags’ for a card sort outcome.
#1: Cards and content don’t match
When we present card sorts to participants we often take the names of the pages in the site as our cards (if it’s a research activity on an existing architecture). The first thing I want to know before starting a card sorting activity is if those pages contain content about more than one topic or concept. We see this a lot on corporate intranets; someone will put up a page that is called something like ‘Policies and Governance’ and fill it with a lot of information and links that are about many different aspects of the business from HR to IT. Users don’t look for pages, they look for information.
When we ask participants to sort cards into categories, they are going to create a concept of what that card represents. If their concepts don’t match the page content, then there is no real value in the outcome because it doesn’t tell you anything about whether a user will actually find what they expect to on the page. Remember, in a card sort you are showing people the card names and the categories only – they can’t see the page content.
When defining the cards to be used in a card sort activity, it can be a good idea to review the site content with a content audit so that you have a clear understanding of the content of each page before asking people to sort them. Look out for use of jargon, acronyms and metaphors in card labels too as they might be hard to place for participants.
#2: The concepts are ambiguous
Card sorting tells us a lot about relationships between concepts and detecting variance between groups of participants, but it this isn’t a realistic context for users’ behaviour. People never go online with the goal of seeing a lot of different concepts and putting them together in groups – not outside of a rainy Saturday puzzling session anyway. By sharing all of the ‘cards’ with a participant, you provide them with a level of insight that is unrealistic in a real life hunt for content.
When we show participants all of the topics alongside each other, we create a specific context for each of those cards. Grouping will be influenced by the other cards that are available. So we can learn how users might group things that they can see, but not how they might think of a concept in isolation.
Consider this example:
Card sort 1:
Card sort 2:
In this scenario, we are likely to see a high level of correlation between categories in each of the card sorts. In the first, we expect to see ‘Colours’ with orange, blue, green, and magenta, and ‘Shapes’ with square, circle, and triangle. In the second we can expect ‘Fruits’ with orange, strawberry, banana, lemon, and ‘Vehicles’ with bicycle, train, bus, helicopter*.
While it’s unlikely anyone will place ‘Orange’ into a category of shapes or vehicles, lets consider a third scenario:
Card sort 3:
Most card sorting, particularly remote unmoderated tests, will only allow a card to be placed into a single category. There’s very little scope for participants to indicate where they think there is ambiguity around the meaning of a concept. This is where we need to think twice about the use of card sorting as a sole source of insight for navigation design.
We can’t expect to standardise humans any more than we can standardise the weather. The point here is that card sorting can give us false confidence that we have made things ‘findable’ for everyone and mask the complexities of context, intention and even culture that can affect a user’s expectations.
#3: You’re asking the wrong question
A card sort will show you what your participants think goes together and maybe what they would call that. But sometimes what you actually want to know is if something is in the right place.
Many of you reading this will be enthusiastically shouting ‘Tree Testing’ at your screen at this point. And you’re right. Conducting a tree test, where you ask participants to find a particular page or piece of information by choosing from possible navigation options, is a very very good companion to card sorting. It allows us to work in the other direction and more closely replicate a user’s behaviour. So once you have some categories, it’s a nice activity to check that they are working.
But … If the challenge with card sorting is that participants know what is there, and their categories will be influenced by that, then the opposite is true for tree testing. Users can’t explore the adjacent categories in the way that many do when using a site in the wild, so there is an increased risk of false negatives, of thinking things are not working when they might be fine.
If you want to discover if a specific thing is where users would look for it, a tree test is a better fit than a card sort.
#4: You haven’t asked enough (or the right) people
One thing that I love about card sorting is that it is a highly versatile research methodology. I’ve used card sorts in usability testing sessions as prompts to open up discussions about discovery behaviours, I’ve used card sorting in workshops to facilitate debates about stakeholder viewpoints. Most often though, a card sort is used as a quantitative (statistical) remote and unmoderated activity to find out the ‘most likely’ or ‘best fit’ category for a particular page or content topic.
Using online tools like Optimal Workshop we can get powerful visual representations of the data gathered during a card sort in a clear and easy to interpret way, with indicators to show where there is strong alignment between responses that suggest the ‘right’ category for a card. These visualisations are based on calculations, typically percentage indicators, from the completed sessions. Before accepting these outcomes though, it’s important to consider the data in context.
For any kind of statistically derived finding, we need to ensure that we have asked enough people. In a sample of 10, the confidence in a finding is lower than if there is a strong alignment in a sample of 100. And it’s not just the numbers themselves, but who they represent. Maybe 100 members of the general public would align a topic with a ‘common sense’ category, but if you look at the data from a specialist set of participants you might find a different result.
If you are looking to use a card sorting activity to inform or validate a navigation structure it is important that you get a big enough data set to work from and that your findings can be generalised to the user group that your site is intended for.
Big green flag: Card sorting is considered alongside other insights
How should we define and refine Information Architecture then? If we can’t rely on card sorting or tree testing alone, then what is the right approach?
Well, as in much of life and UX, decisions are best taken when considered from many angles. Before committing to a navigational structure on the insights from a card sort alone, think about the other things you know about your users’ needs, such as user testing, user interviews, web statistics. Card sorting and Tree Testing are really valuable methods to add insights to that knowledge. Does all of your research agree? Are there areas where research findings are contradictory?
Caption: Card sort data outputs can show how many participants placed a card into each category, giving insight to help design navigation groupings.
There is no ‘one size fits all’ solution to site and content navigation, mostly we are looking to provide the best fit and mitigate issues that we know about for those whose needs aren’t being well met. An experienced Information Architect can help advise on ways to solve the problems and find the optimal solution for your product, service and users, whether that is adjusting the navigational structure or augmenting the user experience with cross links and other discovery support.
Here at Bunnyfoot, we understand that there are no cookie-cutter solutions to great UX design, and we will always look at each project with you to see what approach is the best fit for you, your content, and most or all your users.
—————-
*fun fact, a group of friends recently had a heated debate about whether a helicopter was a vehicle or not. Never invite an Information Architect to a dinner party 😀
Related Posts
