The power of UX benchmarking

Measuring the success of UX work for an e-commerce site with millions of hits a month can be relatively straightforward. If a change is implemented in the morning, and sales go up in the afternoon, it’s worked. (I simplify a little.) And with that many hits, A/B testing is a potentially powerful resource.

Producing quantitative evidence of UX success is not always so simple. For example, a recent UX review I led for Kings Court Trust concerned an online portal for our business partners. Sign-up rates, purchasing rates, amount spent, bounce rates, hit count – none of these measures applied. There were too few users for reliable A/B testing. So, to track success – or otherwise – I needed a different approach.

I relied on benchmarking. That means establishing a base level (a ‘benchmark’) and making an evaluation by comparing results to that benchmark. It’s how we judge whether sign-up rates, amount spent etc, have gone up or down, but for this project – in the absence of those measures – I used user satisfaction ratings.

I had to plan ahead. Before the project began (Feb 2015), I sent a survey to users, our business partners. A number of questions helped to formulate my UX review (to do with tasks, usage, etc), while others focused purely on benchmarking. I provided a number of statements (for example, ‘I can complete my tasks efficiently’, ‘It is easy to use Insight’) and asked users how much they agreed, from 1 to 7.

(Asking these questions couldn’t give me the evidence I needed to formulate solutions – for that, I undertook a number of usability tests with partners.)

9 months later, we launched a new version of the portal (Insight), which the development team had significantly overhauled based on my recommendations. 3 months after that, when users had got used to the new system, I sent a follow-up survey with the same benchmarking questions.

Benchmarking results

The results were as follows:

UX benchmarking results table

UX benchmarking results table, showing weighted averages for each survey out of 7

The average increase was 25.7%. Agreement with the key statement “Overall, I am satisfied with Insight”, increased 32.5%. In a number of cases the level of agreement increased by over 30%.

While there is still room for improvement, the percentage increases are significant. They very much indicate a successful project.

This was a hugely importance exercise, because:

  • It proved the effectiveness and importance of UX (particularly significant when you consider that I am the first UX specialist employed by Kings Court Trust and this was my first project)
  • For Kings Court Trust, it provided an indication of return-on-investment, with success justifying the cost and development resource
  • It provided quantitative evidence, as opposed to qualitative word-of-mouth feedback, that the users – our business partners – appreciated the changes, an important aspect of our commercial relationship with them
  • It will help to guide and justify future UX work
  • It identifies areas for further improvement
  • It motivates the team

Naturally, I made a big deal of these results within the company. And it was not just about me; our developers worked overtime to implement the changes, and a number of others were involved from marketing to project management. It was a fantastic team success, and these results vindicate our effort.

In this case, it’s worth noting that we did not decide in advance what constitutes success. Would it have been a successful project if mean satisfaction rates went up by 5%, 10%, 20%, 30%…? This can be difficult, but it’s a useful step that we’ve begun to implement on more recent projects. Though for this project, I don’t think we lost out by not doing so.

Design and UX have subjective elements, and this can cloud the need for objective measurements. Often, it simply doesn’t happen. But in my view, objectively measuring the success of UX work, in whatever way works for a particular project, is a critical part of the process. It justifies our work, proves our worth to employers and clients, and provides valuable metrics which can guide future projects. After all, we’re in the results business.

That means planning ahead and considering how the success of the project will be measured even before it begins.

Useful UX benchmarking resources:

Sainsbury’s, don’t make me think

Steve Krug’s “Don’t Make Me Think” was the first book I read on usability, back in 2010.

The central tenet – don’t make users think – concisely encapsulates UX.

‘Thinking’ is a barrier to users completing their tasks quickly.

Today, my local Sainsbury’s shop made me think. After I finished scanning my items at a self-service checkout and paying by card, I was confronted by the following screen.

UX at Sainsbury's: stop and think

UX at a Sainsbury’s self-service checkout: I was made to stop and think

My instinct was that something had gone wrong. Perhaps I’d entered my PIN incorrectly, or my card had been declined.

As it transpired, there was nothing amiss. I was free to remove my card, take my shopping and head home to cook some curry.

I had assumed (and I’m guessing I’m not the only one) that the big red encircled exclamation mark indicated an error. Put ‘error icon’ into Google images and you’ll get a bunch of red circles and a fair few exclamation marks.

By breaking convention, the screen had made me think, unnecessarily.

Alternative approach - better UX?

Alternative approach – a clearer indication of success

An alternative for Sainsbury’s would be to ensure the screen adheres to basic and long-established conventions about successful interactions. Specifically, the colour green and a big tick.

Wording that more clearly indicates system status – by specifying explicitly that payment has been successful – would be good, too.

Otherwise (I’ll add, just for balance), the often-maligned self-service experience was excellent.

Good UX is like a good football referee – you shouldn’t notice it’s there. Making users unnecessarily stop and think about their interactions slows them down, which can cause frustration and dissatisfaction.

The less we make users think, the better their experience.

Build services not websites: design principles from GOV.UK

I’m a little bit in love with GOV.UK.

It’s the behemoth online home of the UK government, bringing the nation’s digital services under one roof. An ambitious project which centralises and standardises the web presence for hundreds of disparate agencies.

GOV.UK has a unique set of design challenges: design for the whole country; for any and all users and needs; for the disabled; for the digitally non-proficient. It encompasses all manner of services, from car tax renewals to grant of probate guidance, from passport applications to registration of a lasting power of attorney.

Understandably, it has suffered significant growing pains since its launch in 2012.

Staggering, then, that this award-winning service succeeds on so many levels.

User-focused design principles

The body responsible for GOV.UK’s design is the Government Digital Service. Its design principles and design manual are robust and consistently-applied across the service.

Unlike many other design guidelines available online (e.g. these) – the starting-point of the GDS design principles is content and user needs, rather than a visual language.

They include:

Start with needs (of users, not the government). If you don’t know what the user needs are, you won’t build the right thing. Do research, analyse data, talk to users. Don’t make assumptions

This is principle number one and it’s user experience through-and-through.

The principles also highlight the critical need for simplicity. Here’s number four:

Do the hard work to make it simple. Making something look simple is easy. Making something simple to use is much harder — but that’s what we should be doing. Don’t take “It’s always been that way” for an answer.

Users – who are seeking to perform critical tasks, as opposed to simply browsing a website, are at the heart of the principles:

Build digital services, not websites. A service is something that helps people to do something. Our job is to uncover user needs and meet those needs.

To clumsily sum up in my own words:

  • It’s all about the needs of users
  • Content is king
  • Make it useful, readable and accessible – “prettiness” is not an objective
  • Less is more

Seeing GOV.UK’s user-focused principles in action

As a rule, GOV.UK contains nothing superfluous. Here’s a page relating to applying for a grant of probate, an area of particular interest to me:

Gov uk design - grant of probate

An uncluttered, needs-focused and content-driven page at GOV.UK.

Contrast the clear, needs-focused design above with a page from the DirectGov service which GOV.UK replaced:

Old design for Directgov

The cluttered, hard-to-read and unfocussed approach of’s predecessor, Directgov.

Ultimately, the reason GOV.UK succeeds is the GDS’s unflinching goal to satisfy users’ needs as simply as possible.

Huge credit is also due to the content-writers and editors, condensing often complex and technical subjects into clear, accessible advice. For example, the grant of probate page, above. The GDS’s Writing for GOV.UK guide shows how it’s done.

GOV.UK isn’t perfect, but attaining such consistent usability is a hugely impressive achievement, and one – precisely because of its simplicity – that’s easy to ignore or under-appreciate.

In the UK, successful, ambitious, long-term government projects are rare things, especially in the digital domain. GOV.UK is something we can cherish and learn from.

There is a wealth of publicly-available resources from GDS. Recommended reading includes:

Household heuristics – why a flashing light is good UX

Instinctively, we all want software and products that just work. Confusion, frustration and general faffing about are signs of bad user experience. We feel joyful about things that are simple or intuitive, but when we’re stopped us in our tracks by poor design, things get ugly – fast.

We all intuitively have a sense of user experience, and we’ll all have encountered usability issues – positive or negative – countless times in our lives, without necessarily ascribing them to this thing called “UX”.

A long-standing and well known list of usability guidelines is Jakob Nielsen’s “10 usability heuristics for user interface design“. Around in its current for since 1995, it goes to show just how universal are the basic principles of good usability and how they persist through upheavals in technology.

Nielsen’s background is in software and his heuristics were developed with digital user interfaces in mind. However, most of them apply equally to physical products.
For example, item one:

Visibility of system status
The system should always keep users informed about what is going on, through appropriate feedback within reasonable time.

For my birthday a few weeks ago my wife – perhaps concerned at my increasingly scruff-bag appearance – bought me a beard trimmer.

A already had a beard trimmer but it was getting on a bit. I was glad to upgrade to the latest in trimming technology.

There were features of my old trimmer that I now realise I took for granted: a flashing orange light when the battery was running low; a flashing green light when the battery was charging; and a steady green light when the battery was charged.

Not so in Trimmer: The Sequel. The instructions said to charge before use, but there was no indication of whether it was doing so. A light, a vibration, a beep: anything would have done. My instinct was that I’d made a mistake, but no: it all checked out, and the next morning it had charged. And when the battery runs down, I won’t know until the trimming stops: hopefully not leaving me with demi-trimmed facial hair.

Feedback – expressed by Nielsen as “visibility of system status” – is something so basic and so ubiquitous that we take it for granted, and we miss it when it’s not there. I’m frustrated at the designers of the beard trimmer: did this issue not come up in their meetings? Did they do any user testing? Had they ever read Nielsen’s heuristics? These things aren’t minor; they have a big impact on our relationship with a product.

Looking around a house, interface feedback is everywhere. From the battery indicator on our phones, to the standby light on the telly, to the LED display on a modern electric oven, to the light on the kettle, to the beep after a completed dishwasher cycle, to the vibration of a linked set of baby monitors.

Do all your household appliances provide the best possible feedback? Had you even thought about it? Like most of us, I only tend to notice these things when they’re absent.

Software or product designers would do well to read over those ten ever-so-simple, 30-year-old heuristics from Nielsen now and again. That’s not to say they can never be broken (rules are there to be broken), and they’re no substitute for user testing or a shortcut to success. But life – and my new beard trimmer – would be that little bit more satisfying if they were followed more often.

A few fabulous UX lessons from NUX4

Yesterday was the fourth Northern User Experience conference in Manchester. A hugely impressive event, particularly considering it’s organised by an army of volunteers. 600 UX practitioners (and some people considering moving in to the industry) were in attendance, which probably meant that no actual UX work was being done in the north of England.

I took something away from every one of the seven talks; here are just a few highlights.

Fall in love with problems

Tomer Sharon is a User Experience Researcher at Google in New York. His keynote speech was terrific. Three mantras resonated with me:

1. Do the right thing, then do things right.

(Basically, get super clear on the problem you’re solving first, before attempting to build anything.)

2. Fall in love with problems, then with solutions.

(A similar idea – make sure the problem is perfectly articulated and, until it is, don’t get distracted with solutions. Put the problem you’re solving at the heart of everything you do.)

3. Observe people, don’t just listen to what they say.

(User research! Critical to the process. Surveys and focus groups alone won’t cut the mustard.)


NUX4, Manchester, November 2015


Communicating the UX process to clients

Jenny Grinblo from Future Workshops was another experienced and entertaining speaker.

Jenny stressed the power of bringing user voices into the boardroom. That might mean video clips or audio clips from user tests – something I can attest is an important and powerful part of the process. What I didn’t realise is that seeing or hearing users’ feelings direct stirs chemicals in the brain. Which chemicals, I can’t recall, but anyway, there’s science behind the idea that a video is worth a thousand reports.

Often UX practitioners’ clients are more comfortable with ‘how’, not with ‘what’ or ‘why’. So they might focus on minor changes to an unsuccessful product, rather than being immediately willing to delve more deeply into the root problems. It’s up to us to help guide them to the underlying problems.

Storyboarding is an effective way of communicating user needs to businesses. That means telling a story of the ‘what’ and the ‘why’, eventually reaching the ‘superhero’ moment of the solution, i.e. how the product is solving the problem. Six months later when ideas are still flying around, the initial storyboard helps ground discussions back to solving the problem that was agreed at the initial stages.

Ruthless prioritisation is key. In other words, not only identifying the tasks and requirements of different user groups but also making the utmost effort to prioritise them, and compare the priorities of different groups.

Communicating the UX process to businesses can be difficult. Asking difficult questions (e.g. ‘how will this product make money?’), explaining the decision-making process and – as a final resort – scaring the client in some way (perhaps pointing out what will happen if the wrong decisions are made), can help.

It’s ok to feel uncertain (phew!)

I took away some interesting insight about user workshops, something on the horizon for me in the coming weeks and months.

The principle of encouraging divergent (no holds barred) thinking before converging on specifics appeals to me as a way of encouraging interaction. As does grouping similar user groups together for different workshops, for clarity on the different users’ needs. I’m also keen to use an inordinate amount of post-it notes… always good to see photos of other UX-ers relying on humble notes and pens.

Workshops aren’t the whole story (they’re no replacement for user testing), but it was great to hear more about how effective they can be, and the best way to manage them.

It was also reassuring to hear that it’s ok to feel uncertain and overwhelmed by the process at times. But as UX practitioners, if we trust in the process, it will all come good.

Kindness is designing for the worst

The final keynote was an unexpectedly touching and affecting talk by Sara Wachter-Boettcher, a UX consultant based in Philadelphia.

Sara referred to personal experiences to stress that unnecessary and poorly-worded questions, and inappropriate interfaces, can not only make a user feel excluded but also rejected and isolated. She referred to a period-tracking app that assumed all sex was happy and consensual and an unnecessary question on a form which asked about sexual assault.

The moral of the story is: always design for ‘stress cases’. In other words, consider all your users, particularly the worst off, and make sure the content and design is appropriate for them. That’s the UX of kindness.