I recently had the pleasure of attending Rolf Molich’s “Five users will find 85% of the usability problems – and other myths about usability testing” talk hosted by UX Sheffield. This was part of a series of Rolf Molich talks, held in Oxford, Sheffield and London – all sponsored by Bunnyfoot.
Rolf Molich is one of the pioneers of usability having over 30 years’ experience in the field. His achievements include creating the method of heuristic evaluation with Jakob Nielsen, writing the best-selling Danish book User Friendly Computer Systems (selling roughly 30,000 copies, the book is available in English, with the title Usable Web Design) and founding the Danish usability consultancy DialogDesign.
Like all good usability professionals, Rolf is always keen to gather data and insight from his users. In this context his audience were his users, and throughout the talk the audience provided feedback on their mobile devices (via a website) on what they believed to be usability testing myths or not. Rolf then shared the live feedback before giving his justification on what were usability myths or not.
What Rolf Molich covered during his talk:
1. Usability professionals must set a good example
Usability professionals must set a good example in the work they produce, particularly when writing reports. Too often research findings get lost in unusable Usability Reports, i.e., the reports are too long, and containing too many usability issues. This leaves the reader confused with what issues they should address and/or what to tackle first.The onus is on the UX professional to report findings in a useful and usable way by focusing on the major issues and prioritising what needs to be tackled first. It also helps to use language that concisely conveys the severity of the problems contained in the report.
2. Data trumps opinion
Usability testing is a very important as a political instrument to show people in a very convincing way, that there might be problems with a product and how to fix them, rather than listening to the highest paid person’s opinion.
3. Five users will find 85% of the usability problems in a product
This theory was originally put forward over 15 years ago and is in fact a myth. In truth, with modern, complex products, five users are only enough for a single round of testing in an iterative cycle. It’s also important to remember that zero users give zero insights into the usability of your product – so some testing is always better than no testing. If you can, test early and test often. This allows for issues to be captured early in the project.
4. 25% of usability reports should be positive
Approximately 25% of usability reports should be positive to ensure the features that are working well are brought to the attention of all stakeholders. The danger of not reporting the positive aspects of a usability test are the product team may inadvertently remove features that are working well in their efforts to fix usability issues elsewhere.Rolf also says that highlighting the positive features makes the negative aspects of the report easier to digest for product teams – as “even developers have feelings!”.
5. Expert reviews are as reliable as user testing if carried out by an expert
Based on his CUE (Comparative Usability Evaluation) research findings, Rolf believes that expert reviews can be as reliable as user testing when carried out by an expert.This may be true in some cases, however, it really depends on your research objectives. Expert Reviews will certainly identify usability problems, however, User Testing with different user groups/personas will reveal issues and insights specific to that user-group and industry that may not be captured in expert reviews alone.
6. Moderator say as little as possible
The main goal of a usability test is to discover usability problems. This is true, however, it should also be the goal of the usability test to show skeptical stakeholders (Developers, Project Managers, Product Owners, etc.) that problems exist and there are ways to address them and improve the usability of the product. At Bunnyfoot we also find inviting stakeholders along to observe usability testing sessions a great way of getting buy in from them as it allows them to see first-hand the positives and negatives the participant is experiencing.
7. If you’ve found 60 serious problems, it’s OK to report them all in the report
This is a myth. As described earlier, it is the responsibility of the usability professional to report findings in a useful and usable way. 60 serious problems are too many for one report and it is up to the usability professional to filter the most serious problems and make them usable for the reader to understand.
8. Anybody can conduct a usability test
A little bit of a trick statement here, but yes, anybody can conduct a usability test. However, only a trained usability professional can conduct a quality usability test. Usability professionals are trained to plan, moderate and report findings effectively. This includes ensuring participants don’t deviate from key tasks and not influencing participants by their body language or by the way the participant is spoken to. Failure to do this effectively make skew the results of the user test.
Get CPUX certified
Rolf Molich has helped develop the CPUX (Certified Professional in User Experience) certification programme, an internationally recognised qualification that aims to standardise practice and language across the industry. Bunnyfoot is the leading provider of the CPUX program in the UK, limited spaces are available for the course this November.
If you’re interested in undertaking the CPUX (Certified Professional in User Experience) certification programme to improve your knowledge in this area, please get in touch.
Related Posts
