Usability requirements: how to specify, test and report usability |
Case
study (UK):
Prezzybox/in2netlogic: Ecommerce Web Site
Page contentsOverview OverviewThis study was concerned with an online shopping website called Prezzybox. The site, developed by in2netlogic, offers a wide range of gift ideas for purchasers to select and have delivered to a friend or relative or to themselves. The website was evaluated by the PRUE partner - the Research School of Ergonomics and Human Factors (RSEHF) at Loughborough University (which incorporates the former HUSAT). The main objective of the user test was to obtain user performance and satisfaction metrics for Prezzybox along with user comments to highlight any problems with the site. Testing was carried out with 12 users who were all experienced with the Internet and interested in buying from the Prezzybox site. Users were asked to make a real purchase from the site - this being their payment for taking part. Thus the evaluation was designed to test the success of users in completing an online purchase - a crucial success factor for any online shopping site. The result of the evaluation was a CIF report, which documented user performance and satisfaction with the site. This showed that while 10 out of the 12 users were able to make a purchase, 2 failed to do so. If the consumer organisation could also capture those two users, their sales would increase by 20%! These performance results were also supported with user satisfaction ratings. The levels of satisfaction recorded were 'satisfactory' showing that there is room to improve the 'user experience' when using the site. Evaluator comments and recommendations for change to the site were also included, highlighting features that could be changed to help improve user success in making a purchase. The benefits of the CIF to the consumer organisation (Prezzybox) are:
The benefits of the CIF to the supplier organisation (in2netlogic) are:
Currently the Prezzybox site is being refined. If possible a second round of usability evaluation will be performed so that the consumer and supplier organisations can receive concrete evidence that the site has been improved. It is also hoped to show that this usability activity has enhanced the site from the user's point of view. In summary, the evaluation approach of the CIF is recommended for other online shopping providers in order to test what proportion of users actually make a purchase from their site. It also sets a baseline against which new versions and upgrades to the site can be compared, while also highlighting the problems that need to be fixed if the site is to achieve a greater number of online sales. Background - The evaluation of a shopping websiteThe Web is now populated with many online shopping sites but evidence seems to show that many users fail to make a purchase from them. In the United states, a study carried out in 1999 by Jared Spool who tested a series of shopping websites showed that for even the most successful site only 42% of users were able to make a purchase. For the least successful site, only 28% were able to successfully buy online. Recent data from another leading human factors researcher, Jakob Nielsen, has shown a success rate for online shopping of between 50% and 60%. These figures, which are at best moderate, clearly will have a major effect on reducing profit margins in the e-commerce sector and will also restrict the wider use of online shopping in the future. But how can a design team know whether their online shopping site will be successful in the marketplace? An expert evaluation by Human Factors personnel, or an informal study by users may highlight problems, but how can the online shopping service provider know objectively measure how successful their site will be? The only way is to ask a sample of users to try and purchase an item from the site 'for real' and for an evaluator to observe whether they are successful. This was the approach adopted by the PRUE partner, RSEHF - Research School in Ergonomics and Human Factors (formerly HUSAT) at Loughborough University. The Institute were asked by the in2netlogic web site development company help them evaluate an online present buying website that they had produced for Prezzybox, a midlands-based mail-order company supplying novel and high quality gifts to the public via an online shopping site. The CIF (Common Industry Format) offers a useful framework for organising and reporting on such a study. The EC supported PRUE project was set up to test the CIF in a range of different IT development situations. Thus the evaluation of the Prezzybox website was selected as RSEHF's case study within the PRUE project. Purpose of the trialAlthough the Prezzybox website had been running for some time, no formal testing of it with users had been carried out. Thus the main business objective of the user trial was to find out how easily users could find suitable items on the site to buy and whether they could successfully make a purchase. The benefits of the CIF to the consumer organisation (Prezzybox) are:
The benefits of the CIF to the supplier organisation (in2netlogic) are:
Trial plan, tasks and user selectionA PRUE trial plan was designed to test the success of users in completing an online purchase - a crucial success factor for any online shopping site. Users were asked to make a real purchase from the site - this being their payment for taking part. The measurements to be carried out within the trial were essentially to record user performance in finding and buying a gift for themselves or for a friend and also satisfaction metrics with the whole process of using Prezzybox. User comments and evaluator observations were also recorded to highlight any problems with the site. Testing was carried out with 12 users who were all experienced with the Internet and interested in buying from the Prezzybox site. The user trials were carried out in RSEHF's offices via a laptop with modem telephone connection to the Internet to simulate shopping from home. Users were asked to make a real purchase from the site - this being their payment for taking part. Thus the test was designed to test the success of users in completing an online purchase - a crucial success factor for any online shopping site. Summary of CIF resultsThe result of the evaluation was a CIF report, which documented user performance and satisfaction with the site. User performanceThe main performance metric was whether users could make a purchase. While 10 out of the 12 users were successful in making a purchase, 2 (16%) were not, which indicates a potential loss of sales. The reasons for the failure to buy were that one user was concerned about revealing personal information when completing details on the site, while the other was not successful in finding anything they really wished to buy. Arguably these findings would not have transpired with a traditional formative evaluation. Thus there is scope to improve the site, e.g. by explaining to the end users that any personal details entered (name, address, email, preferences, etc.) will not be passed on to a third party. Similarly, an improved indication of the range of search facilities available could help users find something they wish to purchase more effectively. The other metrics were:
There was also one minor assist each for two users. One user had looked through many items to purchase but wanted to know if there was any kind of search facility available. They were informed that they could look at the Gift Wizard. They investigated this facility but eventually selected an item identified previously. Thus the assist may have affected the task time but, arguably, not the final success in making a purchase. Another user was making a purchase and wanted confirmation that they did not need to register first. They did not expect to but wanted confirmation. Again the assist may have speeded up the process slightly. Without the confirmation the user would have spent a little longer looking at the screen instructions which informed them that registration was not necessary. The average time of 45 minutes is not a major issue and reflects more people's interest in finding an item on the Prezzybox site rather than difficulty in overcoming problems. (Within the total task times for each user, there was very little unproductive time spent in overcoming problems in using the system.) However any improvement in the navigation facilities, screen layout and download times may show an improvement in the performance scores as people are able to view items on the site more efficiently. User acceptanceA Software Acceptability Questionnaire (SAQ) developed at Loughborough was applied which produced mean user ratings for a range of factors on a scale of 1 to 7. The scores were all reasonably high, i.e. Usefulness: 4.2, Clarity: 4.5, Efficiency: 4.6, Supportiveness: 4.5 and Satisfaction: 3.8, while leaving scope for improvement. The scores, which are mostly around the mid-point of the scale (4.0), show that people feel that the website is satisfactory, which is borne out by the fact that in general users enjoyed using the site. The usefulness score (4.2) reflects some peoples' comments that the site did not have all the kinds of items that they would have expected from a present or gift site. Some users wanted to be surer of the quality while others wanted the site to be more focussed. The score for clarity (4.5) shows that users had no significant problems in using the site, but could not scan the layout of the goods on screen as easily as they would have liked. Similarly, under efficiency (4.6), users could move around easily but most commented that the site could be quicker to display pages. This could have been a bigger issue if users themselves were paying the telephone bill. Again the score for support and help (4.5) is fairly high, reflecting the reassuring security statement and well guided purchase process. The satisfaction score is slightly below the midpoint (3.8). User commentsThe performance and acceptability scores were supplemented with user and evaluator comments, and recommendations for change, to the site. Users commented that Prezzybox was very interesting and enjoyable to use but they experienced problems with some aspects of the site: namely screen layout, item ordering, and needing a simpler means of specifying price range. Some users also failed to notice the 'Gift Wizard', a special feature of the site, provided to help the users select items to meet the characteristics of a particular person. Providing a clearer button for it could enhance user interaction with the site. Future plansFollowing dissemination of the report, the consumer and supplier organisations have met to discuss improvements to Prezzybox which will be implemented over time. If possible, a follow-up test will be carried out to show, objectively, if the site has improved from the user's point of view. Hopefully this approach will set the pattern of for testing e-commerce sites in the future to help them meet user needs and be easy to operate. Experiences from using the CIFUsing the CIF has been an interesting experience. It has provided a useful framework for not only planning a user trial but on specifying the objective of the trial itself and developing a suitable plan to achieve it. The CIF recommends some basic metrics, which must be taken but also provides sufficient flexibility to apply other metrics more suitable to the application itself. The shopping study highlighted the interesting issue about whether not using a function could be classed as a type of 'error'. For example, within the Prezzybox website, the Gift Wizard is a special feature designed to help users find gifts to match the interests and personality of the person receiving the present. However this feature was not very prominent and most users did not realise that it was present. By missing out on this feature, they may not have used the site as efficiently as possible, so may be less inclined to return to it in future. The CIF does allow the report writer to add comments and suggestions for improving the system being tested in an Appendix, which clients are normally very interested in, even for summative testing, so perhaps there should be a specific section in the report itself to record this type of information. Feedback on the CIF from the consumer and supplier organisationsA feedback form was distributed to both in2netlogic and Prezzybox. It is interesting to compare the ratings that each of them gave to the report as a whole and to each individual section.
Interestingly the consumer organisation seemed to place slightly more value on the introductory parts of the report and the subjective ratings than the supplier. The supplier and the consumer were moderately interested in the method used to perform the study and the user comments, evaluator observations and suggestions for change. In terms of the main benefits in receiving the CIF, and taking part on the project, both organisations felt that most useful aspect was an impartial and objective analysis of current operation, and generation of new ideas. Neither group identified any problems with the CIF, although the consumer organisation had hoped for more comments on all features of the Prezzybox site. Although some feedback on most features was provided, these depended on which features were used during the trials. The supplier stated that weighted comments would have be very useful - i.e. how important was each suggestion or observation. Both organisations stated that the CIF would definitely lead to changes to the system. Some minor changes were made immediately, but it was also stated that all development is done through a tight specification routine to maximise quality and this takes time. However the improvements are expected to be carried out in the first quarter of 2002. The consumer organisation felt that the CIF improved the knowledge of usability although this was less true of the supplier organisation. As the supplier and consumer organisations already work closely together, it was not expected that the CIF would have a great impact on supplier/consumer communication. DiscussionAlthough the trials elicited many user comments about how to improve Prezzybox, everyone who took part seemed to thoroughly enjoyed the experience of browsing the site and choosing a gift to purchase. Nearly all stated that they would visit the site again. The generation of the PRUE evaluation report provided a useful focus for discussion between RSEHF, the consumer and supplier organisations and laid the path for producing an improved shopping site. The idea of performing a real trial, where users actually purchased an item (as payment for taking part) enabled the group to see how well the site performed in a real situation. This was illustrated by the performance measures although these did not reflect non-use of key features such as the Gift Wizard. The report provided a good format for reporting all features and will serve as a good record for performing the repeat trials. Although suggestions for change can be included in the report in a separate section, perhaps more thought needs to be given as to how summative results can be related more directly to design recommendations. In a developing design situation, this approach could show how design changes lead to improved user performance and satisfaction ratings. Conclusions - what other shopping sites can learnThe summative results reported in the CIF show the most important and stringent test of an online (web) shopping site i.e. can users make a purchase from it? Without such a test, an online shopping service cannot be sure how effective their site is, and whether they may be losing many sales as a result of poor design. Similarly, without producing a baseline for use of the site, the provider will have no basis for comparison when they produce a revised site. The CIF reports the trials in a clear and comprehensive way, allowing trials to be repeated in a faithful way and thus to produce valid results for comparison. There is scope for the CIF to be improved i.e. by integrating formative data more closely with the summative results, so that the consumer and supplier organisations can determine which changes should be of greatest priority in order to enhance user performance i.e. to achieve a great number of e-shopping sales.
Last updated 12-Mar-02 |