Page contents

Overview
Background
Purpose of the trial
How the users, tasks and measures were selected
Summary of CIF results
Benefits to Ericsson from using the CIF
Benefits to Electrolux from using the CIF
Discussion

Overview

In order to support the procurement process, the common industry format (CIF) for usability testing has been used to obtain usability metrics at Ericsson and Electrolux in Sweden. First, the current Ericsson system was tested using the CIF to obtain a baseline value for usability, which resulted in an usability requirements document. The document required information about the way in which the supplier had developed the product and detailed information about user performance and user satisfaction metrics. Secondly, the likely supplier of a new system called WEBRES developed by Electrolux was tested and the results compared to the requirements document.

Ericsson was investigating the purchase of a new interface to the existing travel management system - RES. The current text-based interface no longer supported users task demands and requirements effectively and it was not found easy to use. RES was used by about 25% of Ericsson employees although the goal was to increase this percentage significantly. It was a complete system that supported business travel management both nationally and internationally.

Ericsson did not wish to make significant changes to the underlying system. They looked to a number of potential suppliers who were able to provide a new interface, which would enhance the efficient and effective use of the existing system. At this time, the prime candidate supplier was Electrolux. They used essentially the same travel management system as Ericsson but had developed in-house interface solution - WEBRES. This interface was a possible replacement to the current RES interface.

At this time, Ericsson had not considered any usability issues during the purchase process. There was no objective data about user performance/subjective assessment using RES and there was no usability requirements document. The PRUE project was able to support the procurement process by conducting summative usability testing on RES using the CIF format. The objective was to obtain user performance and satisfaction metrics of the RES interface through formal user testing. The results from testing led to a usability requirements document against which potential suppliers could be assessed. Subsequently, similar testing using the CIF was conducted on Electrolux's WEBRES. The test results were then compared and contrasted to the usability requirements document prepared by Ericsson.

PRUE has been of great value to Ericsson. Without PRUE, usability issues would not have been included in this procurement process. PRUE has resulted in a usability requirements document for a new interface to RES, which was regarded as a new and positive input to the Ericsson decision process when selecting a supplier organization. The usability requirements were regarded as a good compliment to the existing functional and technical specifications. Further, there were benefits of increased understanding of user performance using the current system (poor performance is a large cost in time to Ericsson), an objective understanding of what users thought of the current system and what they require from the new.

From the supplier perspective, Electrolux have been able to understand the performance/ satisfaction related Ericsson requirements for WEBRES. The CIF for usability testing enabled them to assess how WEBRES compared to the Ericsson requirements, and it has provided a clear indication of what improvements need to be made to the interface design.

Background

At the start of the PRUE project, Ericsson was investigating the purchase of a new interface to the existing travel management system - RES. The current text-based interface no longer supported users task demands and requirements effectively and it was not found easy to use. The interface lacked overview of the travel claim and did not support the wide variation and complexity of travel carried out today. There are usability problems, such as use of many codes and abbreviations, plus poor readability.

RES was used by about 25% of Ericsson employees although the goal was to increase this percentage significantly. It was a complete system that supported business travel management both nationally and internationally. Further, managers use RES to approve and attest business travel and expenses, financial controllers use it to review travel claims and RES is connected to other financial systems to ensure that staff are reimbursed quickly and efficiently for travel expenses. It also provides extensive statistical travel data for the Ericsson Enterprise

As Ericsson did not wish to make significant changes to the underlying system, they looked to a number of potential suppliers who were able to provide a new interface, which would enhance the efficient and effective use of the existing system. At this time, the prime candidate supplier was Electrolux. Electrolux used essentially the same travel management system as Ericsson but developed in-house interface solution - WEBRES. The interface was an Internet solution and a possible replacement to the current RES interface.

At this time, Ericsson had not considered any usability issues during the purchase process. There was no objective data about user performance/subjective assessment using RES and there was no usability requirements document. Ericsson agreed to participate in the PRUE trial application to assess the benefits of using the CIF in this procurement process and also to assess the benefits of using summative usability testing in general within the organisation during procurement.

As the prime supplier organisation at this time, Electrolux agreed to participate in the trial application. Users had not formally tested WEBRES for usability.

Within the context of this process, PRUE has carried out five main activities:

A stakeholder meeting to present the importance of usability and specify the intended context of use of the Ericsson travel management system.

A formal user evaluation of the existing system - RES - according to CIF to provide measures of usability as baseline requirements.

A usability requirements document (not in the public domain) to be used by Ericsson when assessing possible alternatives to RES. The document required information about the way in which the supplier had developed the product and required the supplier to provide detailed information about user performance and user satisfaction metrics.

A formal user evaluation of the Electrolux travel management system - WEBRES to establish whether the Ericsson requirements had been met.

An initial benefit analysis of using the CIF to the consumer organisation and the supplier organisation.

Purpose of the trial

The main objective of the trial application was to evaluate whether one possible successor to the existing travel management system at Ericsson, met specified usability requirements. This was demonstrated by applying the common industry format for usability testing during this procurement process to the current Ericsson system (RES) first and then the Electrolux system (WEBRES).

First, user performance and satisfaction metrics of the current RES interface were obtained through formal user testing following the CIF for usability testing. The test results were fed into a usability requirements document in the form of usability goals and against which alternative systems/interfaces could be judged.

Subsequently, using very similar formal user testing following the CIF, user performance and satisfaction metrics of the WEBRES user interface. The test results were compared to the Ericsson usability requirements document. This provided Ericsson with a further decision basis upon which to assess the Electrolux product.

How the users, tasks and measures were selected

Usability metrics

The metrics and measures defined in the CIF were used; the effectiveness parameter - Task completion (%), the efficiency measure - task time (min), number of errors and number of assists.

User satisfaction was rated using the standardised questionnaire - SUMI for the RES test and WAMMI for the WEBRES test. Results of SUMI and WAMMI can be correlated to each other to make comparison of results possible.

Participants

Typical users from the Ericsson Enterprise carried out the testing. A pilot test was carried out at the beginning of the test sessions. Test participants were selected according to two criteria. Firstly, they were Ericsson employees. Secondly, they were familiar with the travel management process and completed travel management accounting procedures up to five times per year. One key group was 'Consultants', who use the travel management system occasionally. Another group, secretaries who help others create their travel claims reports, was included. They were experienced and efficient using the system.

Selection of users for the tests was done via a large e-mail mail shot to Ericsson employees. Test participants familiar with RES were selected according the defined user profile (8 for RES and 7 for WEBRES). The users tested fulfilled the user profile, although some claimed more than 5 business trips per year. Travel varies depending on project work.

The tests were conducted in Swedish, the tasks were in Swedish and users were Swedish. RES and WEBRES use Swedish primarily, although parts of them are in English. The corporate language is English.

Tasks

After some interviews with relatively experienced business travellers, selected tasks for the study were the most frequent travel management tasks at Ericsson Enterprise. Tasks were typical and not complex. Business trips over multi-destinations and long time periods were not selected. The tasks are summarised below:

Task 1: Create a travel application (not applicable to WEBRES)

Task 2: Create a foreign travel report - return travel to Vienna.

Task 3: Create a travel report for domestic travel with train and rental car.

Task 4: Create a travel report for various own car-related expenses.

Users were provided with 'simulated' travel vouchers/receipts for each of the tasks. As users had not actually experienced the business trip, they were given some time to understand the information before starting on the task.

Procedure

The design of the test followed a logical sequence of events for each user and across tests. One person ran the test and one observed. Other persons related to the project or the development team at Ericsson and then Electrolux observed the testing. The testing procedure was as follows:

• User were greeted and offered refreshment.
• Users were given information about the goals and objectives of the test.
• Users were asked a set of predefined background questions prior to the testing
• Users were then given a series of clear instructions specific for the test.
• Users were asked to complete the four tasks in order quickly and efficiently.
• Users were asked to complete a SUMI/WAMMI questionnaire direct after completing the last task.
• Users were asked a set of predefined background questions after the testing
• Users thanked and for their participation were given cinema tickets worth 160 SEK.

As users were familiar with the product and the process of travel management there were no nondisclosure agreements, form completion, warm-ups or pre-task training.

Summary of CIF results

All users had problems in performing the tasks and they made many mistakes. Some of the mistakes were minor, some were easily corrected with/without assistance and some were so severe that the user did not succeed at all. In the interviews preceding the tests many users indicated that they normally have problems with their travel reports after business trips.

Table 1: Summary of usability metrics by user.

The tables above show no major differences in user performance between the two tested systems except for task time and satisfaction. Error rates were unexpectedly high in both systems. The number of assists required when testing the current system was also unexpectedly high. The web-based WEBRES was slower in two cases out of three but was still perceived as better by the users as indicated by the satisfaction measurements.

User satisfaction was rated using the standardised questionnaire - SUMI for the RES test and WAMMI for the WEBRES test. Results of SUMI and WAMMI can be correlated to each other to make comparison of results possible.

Table 2: User satisfaction results – a comparison.

While caution should be used when comparing the subjective assessment results, it is clear that RES performed poorly as it was not liked by users despite the fact that they used RES relatively regularly.

WEBRES on the other hand performed quite well on the WAMMI scale with a high controllability value. Effort would need to be taken to improve attractiveness and efficiency values. This relatively high subjective value was interesting as user performance was actually slightly lower than RES.

Neither systems were acceptable as the global scores of 20 and 45 were less than the average industry average values of 50 for the standardised SUMI and WAMMI questionnaires. In terms of requirements, Ericsson would like a replacement system/interface to be at lest as acceptable as the industry average, which means having a score of at least 50.

Table 3: Summary the overall results by metrics.

The overall results presented in the table above indicate that WEBRES performs slightly better than RES (high overall task completion, few errors). However, mean total task time was longer. There was no significant difference for completion rate efficiency. Users preferred the WEBRES interface.

In conclusion, while users preferred WEBRES they did not perform as well as expected when using it. The results indicate that the design of the WEBRES interface needs to be improved so that it supports typical tasks more efficiently.

Benefits to Ericsson Enterprise from using the CIF

Ericsson assesses that the work done within the PRUE project so far has proven to be of great value.

Initial testing lead to the creation of a usability requirements document for the new travel management system. This was regarded as a new and positive input to the Ericsson decision process when negotiating with Electrolux over the new interface to the current system. The usability requirements are regarded to be good compliment to the functional and technical specifications.

The usability requirements document will support Ericsson during purchase negotiations with Electrolux, or any other supplier of travel management systems.

Testing using the CIF has led to increased understanding of user performance and subjective assessment in current RES system. This has been considered in internal financial terms as poor performance results in a substantial cost to Ericsson.

Testing has given Ericsson an clearer understanding of what users think of the current system and what they require from the new.

Ericsson is considering products from other supplier organizations than the one tested, and the usability requirements document will be one factor supporting the assessment of these alternatives.

There is an interest from Ericsson to continue using the CIF as a basis summative testing during procurement and requirements work both within this specific project and in general across the organisation. Ericsson will support the proposed dissemination activity plan.

Benefits to Electrolux from using the CIF

Electrolux assesses that the PRUE project has led to the following benefits :

Testing of RES using the CIF and the creation of a usability requirements document for the new travel management system has helped Electrolux understand Ericsson usability requirements and the importance of user performance and subjective assessment.

Testing WEBRES using the CIF has enabled Electrolux's to compare the performance of their product against the usability requirements document.

Testing has given Electrolux an clearer understanding of user performance and user satisfaction of WEBRES. It has provided an indication as to how efficiently WEBRES supports the basic and typical tasks. Testing has given them a basis upon which to improve the WEBRES interface.

Discussion

Summative usability testing using the CIF has advantages to both consumer and supplier organisations during procurement. It one of the most effective ways to enrich the consumer's requirements document with objective user performance and satisfaction metrics (based on the existing system/product used).

Consumers will find that it provides an important input when evaluating potential competitive products from a number of supplier organisations during the procurement of a new system/product.

By using the CIF test structure, suppliers are able to demonstrate that their product complies to the usability metrics defined in the requirements document. It will also provide them with information as to how to improve the product/system from the user/usability perspective. It will also encourage suppliers of products and systems to apply principles of user-centred design during the development of their product/system.

Last updated 12-Mar-02