UsabilityNet logo 1 UsabilityNet logo 2

Evaluate prototype

Summary

Participative user based evaluation of a paper or machine prototype to identify usability problems, where the user is probed to explain their expectations and problems.

Benefits

  • Potential usability problems can be detected at an early stage before development is complete.
  • A deeper understanding of the users' expectations and impressions of the system.

Method

Planning

  • Select the most important tasks and user group(s) to be tested (e.g. the most frequent or the most critical).
  • Select users who are representative of the user group(s). 3-5 users are sufficient to identify the main issues.
  • Consider using user-defined tasks, where users are asked to define their own goals prior to the evaluation session.
  • Produce task scenarios and input data and write instructions for the user (tell the user what to achieve, not how to do it).
  • Plan sessions allowing time for giving instructions, running the test, and a post-test interview.
  • Invite developers to observe the sessions if possible. An alternative is to videotape the sessions, and show developers edited clips of the main issues.
  • For a paper prototype a designer is needed to play the role of "computer".

Running sessions

  • Welcome the user, and give the task instructions.
  • For a paper prototype, as the user selects options on each screen, the designer explains what happens, and either points to the next screen or presents the next screen to the user.
  • Do not give any hints or assistance unless the user is unable to complete the task.
  • Observe the interaction and note any problems encountered.
  • The user may be prompted for their impressions of a page design, what they think different elements may do, and what they expect the result of their next action to be. The user may also be asked to suggest how individual elements could be improved.
  • Interview the user to gain general opinions, and to ask about specific problems encountered.

Output

  • Produce a list of usability problems, categorised by importance (use sticky notes to sort the problems), and an overview of the types of problems encountered.
  • Arrange a meeting with the designers to discuss whether and how each problem can be fixed.

Variations

Storyboards or wizard of oz prototypes can be evaluated.

The degree of formality of the session depends on the nature of the prototype and the stage of development. At early stages and with complex systems (such as web sites) most benefit is obtained by obtaining a detailed understanding of how the user is thinking. Later in development the user may just be asked to think aloud in order to get a more realistic assessment of their behaviour, or performance testing may be used.

The user based evaluation can be complemented by expert or heuristic evaluation.

Next steps

Once improvements have been made, evaluate the new version of the system, or if it is the final evaluation consider using performance testing.

Case studies

Background reading

Dumas, JS, and Redish, Janice, A. (1999) Practical Guide to Usability Testing, Intellect Books.

Rubin, Jeffrey (1994) Handbook of Usability Testing. John Wiley and Sons, New York, NY


Home | Professional Groups | Tools & Methods | Usability Practitioner | Usability for Managers | EU Project Support | About this Site
©UsabilityNet 2006. Reproduction permitted provided the source is acknowledged.