The problems you find in usability testing, often appear totally obvious with hindsight. But until you see them, they are not obvious at all.

A real life usability test

I was watching a TV show about the successful ditching of US Airways Flight 1549 in the Hudson River. Like any usability geek, I managed to relate this incident back to the principles of user centred design.

On that day, a real-life usability test was taking place. A man called Jeff Skiles was the test participant. The item being tested was a manual that tells you what to do when your Airbus A320 suffers double engine failure.

A heroic landing

Like me, you were probably awestruck at the skill and professionalism of US Airways pilot Chesley Sullenberger and his co-pilot Jeff Skiles. They successfully ditched a passenger jet into the Hudson River, saving the lives of every passenger and crew member on board. Not bad for a day’s work eh?

The work of the co-pilot

Thankfully, passenger jets don’t suffer double engine failure very often. So when they do, there’s a manual with step-by-step instructions of what to do. It’s the co-pilot’s job (or whichever pilot isn’t flying the plane) to read through the manual.

In the case of Flight 1549, all attempts to restart the engines failed. So the Captain took the brave decision to ditch the plane into the Hudson River. The manual that co-pilot Jeff Skiles was using included instructions for such an eventuality.

Step-by-step instructions

The person (or people) writing this important manual, took a seemingly wise decision to provide it in step-by-step instructions. This made the instructions simple and easy to understand in the time of a crisis. Skiles went through the instructions, preparing the plane for a water landing. However, there was a small but important problem with this approach that would soon become apparent.

The ditch button

The flaw in the manual arose as the plane got close to the water. When ditching on water, there is a button on the Airbus A320 that seals the underside of the plane. This prevents it taking on water and sinking.

This button should be pressed just before impact. In a step-by-step instruction manual, the instruction to do so is on the last page.

By the time co-pilot Jeff Skiles was in a position to read such an instruction, the plane was a few metres above the water. He was probably busy providing all sorts of readings to Captain Sullenberger, so was unlikely to still be reading the manual.

Skiles didn’t press the button, so the Airbus A320 took on water after impact and began to sink.  Thankfully everyone was able to escape the plane and survived.

The benefit of hindsight

With the benefit of hindsight, we can see that a co-pilot is unlikely to be reading a manual, seconds before the plane he’s in hits the water. But until someone uses it to ditch a plane, such an insight doesn’t come to light. The manual’s reader needs to be pre-warned about the button, so they don’t miss the instruction altogether.

So what does this have to do with websites?

If you aren’t testing your site with real people, you are missing the opportunity to the find problems that are impossible to predict. Thankfully the risk of missing these insights will rarely be as serious as this example.

Websites don’t pass or fail usability tests

Everyone on Flight 1549 survived. So you could argue that the manual passed the test. But this isn’t a helpful way of looking at usability testing. It’s not something a website passes or fails.

If you approach testing as a phase that your site needs to pass, you are unlikely to be open-minded enough to get the full benefit of it.

Instead the aim is to find improvements that can be made. For example, re-ordering the flow of information, so that your users get the appropriate information at the point when they need it.

Enjoy this post?

If you enjoyed this post then you might also like some of these:

• Focus group usability testing
• Avoiding the R word
• Usability tests and the effect of learning
• Usability test tasks to avoid

You can also share this article using these buttons.

Why the increased focus on e-commerce?

It’s cheaper for customers

We’re likely to be buying on the internet more often, because we’ll be trying to find a better deal. When our disposable income is reduced, we’re more likely to shop around. These days that means buying and researching prices on the internet.

Those of us who are shopping online already, are likely to do it more and for a wider range of purchases. Others who aren’t yet shopping online may soon find it hard to resist.

The price comparison website Confused.com recently launched a new TV advertising campaign based exclusively on  the usability of its new website. The campaign appears to be aimed at people who want to get the best deal, but are resistant to using a website to get it. Confused.com are trying to entice reluctant internet shoppers on to their website and usability is their selling point.

It’s cheaper for businesses

According to the Society of Information Technology Management,  the average web interaction costs a UK organisation £0.27. Over the phone it’s £3.76 and face-to-face is £9.34 (borrowed info from Gerry McGovern). So the cost-saving benefit of e-commerce is pretty obvious really.

When budgets are being slashed, it’s a good idea to use it wisely. The web is the cheapest route-to-market for most companies. So we can expect to see some of them pushing their web offering strongly.

Getting more from your e-commerce site

You can increase the volume of business you do online in 2 ways.

• Increase the number of people who visit your site.
• Reduce the number of people who leave without buying anything

You can increase the number of people coming to your site by spending  more money on advertising, pay-per-click campaigns and search engine optimisation. But you could also get more out of your current visitors, by fixing the problems that make them leave without buying anything.

Most of the people who visit an e-commerce website will leave without making a purchase. So even a small reduction in the number off these drop-outs can result in an impressive increase in sales. These improvements can often be achieved with relatively simple changes.

An extreme example is described in Jared Spool’s article, the $300 Million Button. Spool explains how an e-commerce website saw a massive increase in sales just by letting people buy without creating an account.

As amazing as this story is, you don’t need to look far on the web to find a retailer repeating the same mistake. Perhaps you’re reading this and realising that your own site is one of them.

Breaking the redesign cycle

You don’t need to redesign your website in order to make significant improvements to it. A good thing that might come-out of the credit crunch is that more web managers realise, large-scale redesign projects are a waste of time and money.

A responsive company does not improve itself by throwing everything away and starting again every few years. Instead it tweaks and adjusts itself by improving only the things that need to be changed. It’s just a matter of finding the things you need to improve.

Finding the areas to improve

The 2 techniques I most often use to help my clients improve their websites are:

• Expert usability reviews
• Usability testing

It’s often a good idea to use them both, but not at the same time. You start with an expert usability review and then follow up with usability testing after you’ve made some improvements. This allows you to get more out of both techniques.

The expert usability review

Many e-commerce websites make similar mistakes, so an experience usability person can often spot them. By asking a usability specialist to review your site, you can find issues that are likely to cause problems for your visitors. They should be able to recommend positive changes for you to consider.

A classic example is the number of websites that force registration (or even appear to do so) before making a sale, as described in Jared Spool’s article.

Usability testing

Usability testing is a very good way to find the problems that are limiting your site’s potential to sell products. This involves watching people using your website unassisted.  You can run your own usability testing or get a professional to do it for you. There are benefits in both approaches and you can also do a bit of a mixture of both.

What’s more important is that you take enough of an interest to watch the tests yourself and that you test regularly. It’s better to test your site with a handful of people every few months, than do all-singing-all-dancing studies every year or so.

Such grand studies can leave you with more information than you know what to do with. They can also be fairly demoralising, rather than inspiring a culture of constant improvement.

Measuring success

Before you begin to improve your website, it’s a good idea to find some measurements that you’d like to see improve. It’s all very well trying to identify usability issues. But the changes you make should lead to a measurable improvement in the performance of the site. Surprisingly, this is a step that many businesses overlook.

Continuous improvement

Your website can be improved. This is a fact regardless of which site it is. Improving websites is easier and more effective manage when done continuously. If you manage an e-commerce website and are worried about the financial climate. Then I recommend you start improving your website now and never stop.

Did you find this useful?

If you found this article useful or even if you want to tell me I’m wrong, please leave a comment using the form below. You might also be interested in the following posts:

• Focus group usability testing
• Usability test tasks to avoid
• The benefits and limitations of expert reviews

One technique that is becoming popular with some is focus group usability testing. I recommend that you avoid this technique completely. I’ll try to explain why.

What is focus group usability testing?

Instead of one-on-one sessions, the facilitator asks 6-8 people to carry out tasks simultaneously using individual computers. After they have attempted the task, the facilitator then asks the participants collectively if they experienced any problems. The problems they report become the findings of the study.

The idea is that you can pick up on themes and find common problems. You can involve lots of participants in a short period of time. It also means that you are using less of the facilitator’s time. So it’s cheaper.

Quality of findings

Defenders of focus group usability testing will argue that the number of findings returned from this method is comparable with one-on-one testing. However the number of findings is not an adequate measure of the success of usability techniques. The issue with testing simultaneously in focus groups is that you can not rely on the accuracy of your findings.

The problem with focus groups

Focus groups are arguably useful for many types of research. Gathering data on user behaviour is not one of them.

Humans are poor at describing their own behaviour. This is because they don’t often remember their precise actions. Social pressure also inhibits their ability to tell the truth.

This social pressure will be multiplied when they are trying to describe their actions to 8 people they have never met. The people most likely to inaccurately describe the problems they experience, are the ones with the biggest ego. These are the same people who will be doing all of the talking at your ‘test’.

Observing the user experience

The key to usability testing is observation. Anyone who has observed a number of usability tests will know that participants’ recollections of their actions often have little resemblance to reality. You need to watch them if you want to know what they did.

The causes of important usability issues often go completely unnoticed by the test participant. The issues are picked up because the participant is being observed.

I can think of only 2 ways to properly observe a room full of people who are simultaneously using your website. The first is to match every participant with an observer. This doesn’t work as a budget technique because it’s very expensive. It’s also pretty complicated to collect the results and will probably be very intrusive. The other option is to use the Lernaean Hydra as your test facilitator.

Cut costs not corners

There are many ways that you can make the most of a limited usability budget. The key is to cut out the expensive bits without affecting the reliability of your findings. A skilled usability professional will be able to suggest approaches to meet your budget without returning unreliable results.

Some examples include:

• Using fewer participants
• Recruiting your own participants
• Ditching a written report in favour of a one day workshop
• Using a meeting room instead of a lab

Of course, the biggest cost-cutter is to learn how to run your own tests. Truly helpful usability consultants will encourage you to do this and help you learn. I’ll write a more detailed post on this topic soon.

What do you think?

So what are your thoughts about group tests? Please leave a comment and let me know your opinion.

Avoiding the R word makes it easier to improve your website.

Usability should be an iterative process

There is no such thing as the perfect design. You can test your site, fix the problems that you find, and then test it again. You’ll find more areas for improvement, they’ll just be different ones and hopefully less severe. The best way to constantly improve your website is to keep doing this in cycles (alongside other usability activities of course).

A redesign is a large task

Much of your website probably works OK. In a redesign project you end up changing all of it. A redesign is a massive undertaking for most sites. Much of the work involves changing parts of your website that actually work fine. In fact, by changing it so drastically you risk resolving some problems but introducing new ones elsewhere. Why not focus on the opportunities for improvement in small phases instead?

The distraction of the homepage

If you’ve ever been involved in a redesign project you’ll know how much time is wasted arguing about the minute details of the homepage.

The homepage is a symbol the whole project for many people. So around 60-80% of the time spent discussing the project can centre on the homepage. A few months after launch, much of the stuff you were arguing about can be changed without many people noticing.

Redesigns are too high profile

If you’re ‘introducing a new feature’, most of the company will leave you alone to get on with it. When the R word is used, everyone wants a piece of the action.

The company management will of course want to change something and you’ll have to waste time persuading them to leave it alone. You will of course fail. When the organisation is large, then there will be a lot of these important people sticking their nose in. This can result in butchery of the very things you’re trying to improve.

Not many problems will require a redesign to fix.

Many of the usability problems on your site can be improved without a redesign. Don’t be scared to make big changes to your website over a period of time. The things that appear to be gaping inconsistencies to you will often go unnoticed by your users. Those who notice will seldom care.

The threat of deadlines

Website redesigns usually have deadlines. Deadlines are a good motivator, but they can lead to corners being cut. Many of the requirements you started with can be dropped in order to meet the deadline you set.

When you release iterative improvements, those deadlines are easier to reach. You can also pick up dropped requirements in the next phase. When you’re doing a redesign you may have to wait years to do the same.

How did you get in this situation in the first place?

If you feel you need to redesign the site, then ask yourself how you managed to get yourself into this situation. Perhaps it’s because you haven’t improved the site significantly since the last redesign? So what are you going to do this time to stop it happening again?

Even if you do redesign it this time, consider planning this to be your last full-scale redesign.

Learning from the big boys

When was the last time the BBC redesigned its website? How about Amazon? These organisations change their websites, sometimes significantly. But they no longer seem to redesign them. Instead they evolve over time.

By doing so, they constantly refine the site without having to start again from scratch.

Treat your website like a garden

The approach I favour is to constantly improve a website instead of redesigning it every few years. It’s a lot more efficient. Think of your website as a garden. You don’t uproot the whole thing every 3 years because you’re bored with it. Instead you identify areas for improvement and it develops and improves over time.

By doing this, you leave the stuff that’s working OK alone and concentrate on the areas that can be improved. As a result your website will improve more rapidly.

The importance of vocabulary

It’s important to use the same words as your users on your website. They often use different words from you and probably recognise a fraction of the terms and abbreviations you use on a daily basis.

This is not only important for writing passages of text but also for way finding on your site. Good links and headers use the same words that users are thinking when they need to use them. Trigger words and Customer Carewords are based on this fact. “Words drive action” as Gerry McGovern puts it.

Trigger words

In 2004 Jared Spool wrote about trigger words. He argued that users were more likely to find what they are looking for when a site used exactly the same words they use to describe their goals. These words trigger a response because they match the words in users’ heads.

Finding your site’s trigger words

Trigger words can be found by talking to users. One suggestion Jared made was to find trigger words during usability testing. You can identify them by asking target users to describe their needs and goals before they have seen the website. You can then compare the words they use to the equivalents used on the website.

You can also use your search logs to find trigger words. Spool found that in tests, people were more likely to use site search when the website did not use the trigger words. When they used site search they typed the same trigger words they had mentioned before they had seen the site.

Customer Carewords

Gerry McGovern discusses Customer Carewords in his book Killer Web Content. (It’s a good book but the formatting of the text makes it difficult to read in places.) Customer Carewords is Gerry McGovern’s methodology for finding two things:

• The things that are most important to a website’s users.
• The words they use to describe those things.

The methodology for doing this is fairly simple. Create a long list of potential carewords and ask lots of your target users to choose the ten things that are most important to them. OK it’s slightly more involved than this. But I need to condense an entire chapter into one sentence. The list you provide them with contains lots of alternative terms for the same things. So you could have a hundred or so items in total.

In your results you find that a small number of potential carewords are chosen far more frequently than the other words. McGovern argues that 5% of your website delivers 25% of it’s value. This 5% surfaces in the Customer Carewords methodology and he calls it the ‘Long Neck’.

So what’s the difference?

The first difference between the two concepts is that trigger words do not help you prioritise, they just give you the correct words to use. However this isn’t what I’m trying to compare. I’m sure Jared Spool understands task priorities and uses other techniques to find them. I want to compare carewords to the trigger words.

User-generated versus user-selected

For me the trigger words are slightly stronger than the carewords. This is because they are generated by the user. They are found by inspecting search logs or through open discussion during usability tests. The words are a by-product of a conversation or a site search. No suggestions are made and the user is unaffected by the fact that you are noting the words they use. They don’t know it’s happening.

Customer Carewords however are selected from a list of pre-defined terms. I think that this can have an impact on the words chosen. McGovern himself admits as much in his book. In one study he carried out on a travel site, ‘special offers’ received twice as high a score as ‘deals’. So by his methodology ‘special offers’ was the strongest careword. However he found that there were 76,238 internet searches made monthly in the U.S. for ‘vacation deals’ or ‘travel deals’. In the same period there were just 46 searches for ‘vacation special offers’ or ‘travel special offers’.

It seems to me that if you place importance on the words by specifically asking for them, you influence the results. In a discussion, the respondents from McGovern’s study may have innocently said ‘deals’ numerous times. But when asked to choose a word, they chose ‘special offers’. So ‘deals’ is the trigger word and ‘special offers’ are the carewords.

So why did this happen? McGovern’s response is “the words people search with may not always be those they like to see when they visit a website“. This is based on the actual respondents’ justifications for their actions. My opinion is that their own justification can be biased by ego.

Interestingly McGovern leaves this question unanswered. He doesn’t say which option was the one he took. Perhaps he chose the trigger word?

Customer Care Trigger Words

I think you can merge both exercises and just prioritise the words that users provide spontaneously. So you override carewords that are contradicted by trigger words.

Carrying out Gerry McGovern’s Customer Carewords methodology will help you find out what’s important on your website. You can then cross-check your customer carewords against their trigger word equivelants. Check your search logs and note the words that usability testing participants use by interviewing them before they see the website. If the trigger words don’t match the carewords then use the trigger words.

Doing so will let you know what information your users prioritise. You’ll also be able to use their vocabulary when providing it.

Simulating reality

Usability tests attempt to discover what people really do when they use websites. Essentially we’re trying to simulate situations that occur in real life. Good test facilitators try to preserve this reality as much as they can. However in reality, participants do not spend an hour or so carrying out a handful of different tasks on a website.

Learning effect

As the participant attempts each task, they (hopefully) learn a little more about the site. So in subsequent tasks you don’t get to observe how truly new users will perform the task. This is why you should be careful about the tasks you include in your study. In reality people might attempt only one task on your site, so this learning effect can be artificial.

Randomising tasks

One option is to randomise the tasks. So no participant does the tasks in the same order. This balances the effect of learning across the tasks. If you’re timing tasks, using eye tracking or are specifically interested in success metrics, then this is the way to go.

Prioritising tasks

If you have a clear idea of the key tasks for your site and they have an obvious priority order then you can order them by priority. So your top task gets to go first. The rest follow in priority order. However you have to accept that your completion levels are a little flawed.

Prioritise then randomise

Some sites have a single task that is significantly more important than all of the others. Take for instance an airline. Booking a flight is easily the top task. There are lots of other tasks but none of them come close in terms of importance. In cases like this you can start every participant with your key task. After this the following tasks are randomised.

So you get the benefit of having no learning effect on your number one task. On all the others the learning effect is distributed across the tasks.

Free browse

The free browse task is when the facilitator just asks the participant to just spend a few minutes looking around the site. People often make it the first task in the test. But it’s a waste of valuable lab time.

It’s totally artificial. In a usability test you’re trying to simulate reality. You want to see how people are actually using your site out there in the real world. Nobody is going to just browse around your site without purpose. It’s tempting just to get their impressions of the site but it’s of no real value. If you start off with a free browse task you’re basically giving the user the opportunity to teach themselves how to use your site.

It’s a waste of your first task. There is naturally going to be some learning that takes place during a usability test. So your first task is the most real. It is the one that the participant attempts fresh. This is the moment when you get to see how people use your site when they’ve never used it before. If you use this opportunity to let participants have a quick browse around then you’re throwing away the most useful part of the session.

Homepage review

The homepage review task is when the facilitator walks the participant through each part of the homepage and ask for their impressions.

It too is artificial. Like the free browse task, the homepage review does not simulate reality. You can get a bit of useful information from it, but at the cost of making the rest of your tasks unreal.

You help the participant decide where to click. When people attempt to find information on a website they don’t look at every link on the homepage. They often follow the first appropriate looking link.

If you’ve previously walked them through every aspect of the site’s homepage you are giving them an insight that they will not get in reality. Instead of following the first link that looks appropriate, they can use the knowledge from the homepage review to decide which of all the options is the best. They are then performing a scenario that bares little resemblance to the experience real users will have.

Eyetracking the homepage

Eyetracking is a bit of a touchy subject in usability at present. It is only really helpful in very specific studies but usability companies are finding that it wins business. But that’s the topic for about a dozen other posts.

The results of just eyetracking participants looking at the homepage are totally useless without giving them a task to do. I’ll explain this with an example.

If I were to ask you to look at a door without any other instruction you would probably stare blankly at it. If you knew I was eyetracking, you might feel obliged to look around different bits of the door.

However if I were to ask you to open the door then you would most likely look at the door handle. If I were to ask you to remove the door, you’d probably look at the hinges. You see the areas you look at will depend on the task you are trying to complete.

It’s a totally useless exercise to just show a participant a page and then track their eyes while they do it. Nobody will ever visit your site to just stare blankly at the page. You might end up with a pretty picture to pass around the office. But that’s all it is, a pretty picture.

The results of eyetracking a page will differ widely depending on the task that the participants are asked to complete.

Try to simulate reality

Usability testing is about trying to simulate what’s happening in the real world and benefit from the observations. The tasks above are vanity tasks that remove an element of reality from the session. You will get more realistic results if you avoid using them.