Usability tests and the effect of learning

Posted October 6th, 2008 by David Hamill

During usability tests participants will learn how to use the website during the session. This can have an affect on the accuracy of your results.

Simulating reality

Usability tests attempt to discover what people really do when they use websites. Essentially we’re trying to simulate situations that occur in real life. Good test facilitators try to preserve this reality as much as they can. However in reality, participants do not spend an hour or so carrying out a handful of different tasks on a website.

Learning effect

As the participant attempts each task, they (hopefully) learn a little more about the site. So in subsequent tasks you don’t get to observe how truly new users will perform the task. This is why you should be careful about the tasks you include in your study. In reality people might attempt only one task on your site, so this learning effect can be artificial.

Randomising tasks

One option is to randomise the tasks. So no participant does the tasks in the same order. This balances the effect of learning across the tasks. If you’re timing tasks, using eye tracking or are specifically interested in success metrics, then this is the way to go.

Prioritising tasks

If you have a clear idea of the key tasks for your site and they have an obvious priority order then you can order them by priority. So your top task gets to go first. The rest follow in priority order. However you have to accept that your completion levels are a little flawed.

Prioritise then randomise

Some sites have a single task that is significantly more important than all of the others. Take for instance an airline. Booking a flight is easily the top task. There are lots of other tasks but none of them come close in terms of importance. In cases like this you can start every participant with your key task. After this the following tasks are randomised.

So you get the benefit of having no learning effect on your number one task. On all the others the learning effect is distributed across the tasks.

Tags: tasks, Usability testing