Participative user based evaluation of a paper or machine
prototype to identify usability problems, where the user is
probed to explain their expectations and problems.
- Potential usability problems can be detected at an early
stage before development is complete.
- A deeper understanding of the users' expectations and
impressions of the system.
- Select the most important tasks and user group(s) to be
tested (e.g. the most frequent or the most critical).
- Select users who are representative of the user group(s).
3-5 users are sufficient to identify the main issues.
- Consider using user-defined tasks, where users are asked
to define their own goals prior to the evaluation session.
- Produce task scenarios
and input data and write instructions for the user (tell
the user what to achieve, not how to do it).
- Plan sessions allowing time for giving instructions, running
the test, and a post-test interview.
- Invite developers to observe the sessions if possible.
An alternative is to videotape the sessions, and show developers
edited clips of the main issues.
- For a paper prototype a designer is needed to play the
role of "computer".
- Welcome the user, and give the task instructions.
- For a paper prototype, as the user selects options on
each screen, the designer explains what happens, and either
points to the next screen or presents the next screen to
- Do not give any hints or assistance unless the user is
unable to complete the task.
- Observe the interaction and note any problems encountered.
- The user may be prompted for their impressions of a page
design, what they think different elements may do, and what
they expect the result of their next action to be. The user
may also be asked to suggest how individual elements could
- Interview the user to gain general opinions, and to ask
about specific problems encountered.
- Produce a list of usability problems, categorised by importance
(use sticky notes to sort
the problems), and an overview of the types of problems
- Arrange a meeting with the designers to discuss whether
and how each problem can be fixed.
wizard of oz prototypes
can be evaluated.
The degree of formality of the session depends on the nature
of the prototype and the stage of development. At early stages
and with complex systems (such as web sites) most benefit
is obtained by obtaining a detailed understanding of how the
user is thinking. Later in development the user may just be
asked to think aloud in order to get a more realistic assessment
of their behaviour, or performance
testing may be used.
The user based evaluation can be complemented by expert
or heuristic evaluation.
Once improvements have been made, evaluate the new version
of the system, or if it is the final evaluation consider using
Dumas, JS, and Redish, Janice, A. (1999) Practical Guide
to Usability Testing, Intellect Books.
Rubin, Jeffrey (1994) Handbook of Usability Testing. John
Wiley and Sons, New York, NY