Performance testing
Summary
Performance testing is a rigorous usability evaluation of
a working system under realistic conditions to identify usability
problems and to compare measures such as success rate, task
time and user satisfaction with requirements.
Benefits
- Major usability problems are identified that may not be
revealed by less formal testing, including problems related
to the specific skills and expectations of the users.
- Measures can be obtained for the users' effectiveness,
efficiency and satisfaction.
Method
Planning
- It is important that the users, tasks and environment
used for the test are representative of the intended context
of use.
- Select the most important tasks and user groups to be
tested (e.g. the most frequent or the most critical).
- Select users who are representative of each user group.
3-5 users are sufficient to identify problems. 8 or more
users of each type are required for reliable measures.
- Produce a task scenario and
input data and write instructions for the user (tell the
user what to achieve, not how to do it).
- Plan sessions allowing time for giving instructions, running
the test, answering a questionnaire, and a post-test interview.
- Invite developers to observe the sessions if possible.
An alternative is to videotape the sessions, and show developers
edited clips of the usability problems.
- Two administrators are normally required to share the
activities of instructing and interviewing the user, operating
video equipment (if used), noting problems, and speaking
to any observers.
- If possible use one room for testing, linked by video
to another room for observation.
- If usability measures are required, observe the user without
making any comments.
- If measures are not required, prompt the user to explain
their interpretation of the contents of each screen and
their reason for making choices.
Running sessions
- Welcome the user, and give the task instructions.
- Do not give any hints or assistance unless the user is
unable to complete the task.
- Observe the interaction and note any problems encountered.
- Time each task.
- At the end of the session, ask the user to complete a
satisfaction
questionnaire.
- Interview the user to confirm they are representative
of the intended user group, to gain general opinions, and
to ask about specific problems encountered.
- Assess the results of the task for accuracy and completeness.
Output
- Produce a list of usability problems, categorised by importance
(use post-it-notes to sort the
problems), and an overview of the types of problems encountered.
- Arrange a meeting with the project manager and developer
to discuss whether and how each problem can be fixed.
- If measures have been taken, summarise the results of
the satisfaction questionnaire, task time and effectiveness
(accuracy and completeness) measures.
- If a full report is required, the
 Common
Industry Format provides a good structure.
More information
There is a detailed example
of a usability report using the Common Industry Format.
More information on usability testing can be found in the
INUSE
Handbook. The MUSiC
Performance Measurement Method provides detailed instructions
for measuring effectiveness, efficiency and satisfaction
STC resources
for usability testing and usability labs.
Variations
Performance testing is used to obtain measures to establish
or compare with usability requirements. Earlier in development
less formal methods such as participatory evaluation or diagnostic
evaluation are more appropriate.
Next steps
Collect
feedback from users after release to inform any redesign.
Case studies
Background reading
Dumas, JS, and Redish, Janice, A. (1999) Practical Guide
to Usability Testing, Intellect Books.
Rubin, Jeffrey (1994) Handbook of Usability Testing. John
Wiley and Sons, New York, NY
ISO
9241-11 Guidance on usability |
