UsabilityNet logo 1 UsabilityNet logo 2

Diagnostic evaluation

Summary

User based evaluation of a working system, where the primary objective is to identify usability problems.

Benefits

  • Major usability problems are identified.
  • An understanding is gained of why the user has difficulties with the system.
  • Approximate measures can be obtained for the users' effectiveness, efficiency and satisfaction.

Method

Planning

  • It is important that the users, tasks and environment used for the test are representative of the intended context of use.
  • Select the most important tasks and user group(s) to be tested (e.g. the most frequent or the most critical).
  • Select users who are representative of the user group(s). 3-5 users are sufficient to identify the main issues. 8 or more users of each type are required for reliable measures. For complex systems such as an e-commerce web site, larger numbers may be require to explore all aspects of the system.
  • Consider using user-defined tasks, where users are asked to define their own goals prior to the evaluation session.
  • Produce task scenarios and input data and write instructions for the user (tell the user what to achieve, not how to do it).
  • Plan sessions allowing time for giving instructions, running the test, answering a questionnaire, and a post-test interview.
  • Invite developers to observe the sessions if possible. An alternative is to videotape the sessions, and show developers edited clips of the main issues.
  • Two administrators are normally required to share the activities of instructing and interviewing the user, operating video equipment (if used), noting problems, and speaking to any observers.
  • If possible use one room for testing, linked by video to another room for observation.
  • If usability measures are required, observe the user without making any comments.
  • If measures are not required, prompt the user to explain their interpretation of the contents of each screen and their reason for making choices.

Running sessions

  • Welcome the user, and give the task instructions.
  • Do not give any hints or assistance unless the user is unable to complete the task.
  • Observe the interaction and note any problems encountered.
  • If required time each task.
  • Either ask the user to think aloud, or prompt the user to explain their interpretation of the contents of each screen and their reason for making choices.
  • At the end of the session, ask the user to complete a satisfaction questionnaire.
  • Interview the user to confirm they are representative of the intended user group, to gain general opinions, and to ask about specific problems encountered.
  • Assess the results of the task for accuracy and completeness.

Output

  • Produce a list of usability problems, categorised by importance (use sticky notes to sort the problems), and an overview of the types of problems encountered.
  • Arrange a meeting with the project manager and developer to discuss whether and how each problem can be fixed.
  • If measures have been taken, summarise the results of the satisfaction questionnaire, task time and effectiveness (accuracy and completeness) measures.
  • If a full report is required, the offsiteCommon Industry Format provides a good structure.

More information

More information on usability testing:

STC offsiteresources for usability testing and usability labs.

offsiteINUSE Handbook.

offsiteMUSiC Performance Measurement Method provides detailed instructions for measuring effectiveness, efficiency and satisfaction.

Variations

The degree of formality of the session depends on the relative importance of understanding usability problems and obtaining usability measures. To obtain accurate performance measures the more formal version of the procedure should be used (where the context of evaluation carefully matches the intended context of use, and there is no interaction with the user during testing). At early stages and with complex systems (such as web sites) most benefit is obtained by participatory evaluation, obtaining a detailed understanding of how the user is thinking. Later in development the user may just be asked to think aloud in order to get a more realistic assessment of their behaviour.

The user based evaluation can be complemented by expert or heuristic evaluation.

Next steps

Once improvements have been made, evaluate the new version of the system, or if it is the final evaluation consider using performance testing.

Case studies

Background reading

Dumas, JS, and Redish, Janice, A. (1999) Practical Guide to Usability Testing, Intellect Books.

Rubin, Jeffrey (1994) Handbook of Usability Testing. John Wiley and Sons, New York, NY


Home | Professional Groups | Tools & Methods | Usability Practitioner | Usability for Managers | EU Project Support | About this Site
©UsabilityNet 2006. Reproduction permitted provided the source is acknowledged.