Simulating reality

Usability tests attempt to discover what people really do when they use websites. Essentially we’re trying to simulate situations that occur in real life. Good test facilitators try to preserve this reality as much as they can. However in reality, participants do not spend an hour or so carrying out a handful of different tasks on a website.

Learning effect

As the participant attempts each task, they (hopefully) learn a little more about the site. So in subsequent tasks you don’t get to observe how truly new users will perform the task. This is why you should be careful about the tasks you include in your study. In reality people might attempt only one task on your site, so this learning effect can be artificial.

Randomising tasks

One option is to randomise the tasks. So no participant does the tasks in the same order. This balances the effect of learning across the tasks. If you’re timing tasks, using eye tracking or are specifically interested in success metrics, then this is the way to go.

Prioritising tasks

If you have a clear idea of the key tasks for your site and they have an obvious priority order then you can order them by priority. So your top task gets to go first. The rest follow in priority order. However you have to accept that your completion levels are a little flawed.

Prioritise then randomise

Some sites have a single task that is significantly more important than all of the others. Take for instance an airline. Booking a flight is easily the top task. There are lots of other tasks but none of them come close in terms of importance. In cases like this you can start every participant with your key task. After this the following tasks are randomised.

So you get the benefit of having no learning effect on your number one task. On all the others the learning effect is distributed across the tasks.

Free browse

The free browse task is when the facilitator just asks the participant to just spend a few minutes looking around the site. People often make it the first task in the test. But it’s a waste of valuable lab time.

It’s totally artificial. In a usability test you’re trying to simulate reality. You want to see how people are actually using your site out there in the real world. Nobody is going to just browse around your site without purpose. It’s tempting just to get their impressions of the site but it’s of no real value. If you start off with a free browse task you’re basically giving the user the opportunity to teach themselves how to use your site.

It’s a waste of your first task. There is naturally going to be some learning that takes place during a usability test. So your first task is the most real. It is the one that the participant attempts fresh. This is the moment when you get to see how people use your site when they’ve never used it before. If you use this opportunity to let participants have a quick browse around then you’re throwing away the most useful part of the session.

Homepage review

The homepage review task is when the facilitator walks the participant through each part of the homepage and ask for their impressions.

It too is artificial. Like the free browse task, the homepage review does not simulate reality. You can get a bit of useful information from it, but at the cost of making the rest of your tasks unreal.

You help the participant decide where to click. When people attempt to find information on a website they don’t look at every link on the homepage. They often follow the first appropriate looking link.

If you’ve previously walked them through every aspect of the site’s homepage you are giving them an insight that they will not get in reality. Instead of following the first link that looks appropriate, they can use the knowledge from the homepage review to decide which of all the options is the best. They are then performing a scenario that bares little resemblance to the experience real users will have.

Eyetracking the homepage

Eyetracking is a bit of a touchy subject in usability at present. It is only really helpful in very specific studies but usability companies are finding that it wins business. But that’s the topic for about a dozen other posts.

The results of just eyetracking participants looking at the homepage are totally useless without giving them a task to do. I’ll explain this with an example.

If I were to ask you to look at a door without any other instruction you would probably stare blankly at it. If you knew I was eyetracking, you might feel obliged to look around different bits of the door.

However if I were to ask you to open the door then you would most likely look at the door handle. If I were to ask you to remove the door, you’d probably look at the hinges. You see the areas you look at will depend on the task you are trying to complete.

It’s a totally useless exercise to just show a participant a page and then track their eyes while they do it. Nobody will ever visit your site to just stare blankly at the page. You might end up with a pretty picture to pass around the office. But that’s all it is, a pretty picture.

The results of eyetracking a page will differ widely depending on the task that the participants are asked to complete.

Try to simulate reality

Usability testing is about trying to simulate what’s happening in the real world and benefit from the observations. The tasks above are vanity tasks that remove an element of reality from the session. You will get more realistic results if you avoid using them.