ࡱ> 5701234? jbjb %`}}l8\2`* (444#L%(=??????, k'''k/44g///'44=/'=/6/#7ab.Haaa/a/ IST Take Up Action IST-1999-20692 PRUE Providing Reports of Usability Evaluation Final Report Guide to specifying and evaluating usability as part of a contract Nigel Bevan (Serco Usability Services, editor) Nigel Claridge (IconmediaLab) Maria Athousaki (SIEM), Martin Maguire (Loughborough University) Tiziana Catarci (Universita' di Roma "La Sapienza") Giacinto Matarazzo (Fondazione Ugo Bordoni) Gianluigi Raiss (AIPA, Rome) Version 1.0  SAVEDATE \@ "d-MMM-yy" \* MERGEFORMAT 11-Mar-02 Abstract This document provides a guide for how to specify and test usability requirements as part of a contractual relationship between a supplier and acquirer. It includes case studies of four trials carried out in Europe by the EU-funded PRUE project. This report is available online at www.usability.serco.com/prue Keywords: usability, evaluation, procurement This report was edited by. Nigel Bevan Serco Usability Services 22 Hand Court London WC1V 6JF, UK Tel: + 44 20 74 21 64 74 Fax: + 44 20 74 21 64 77 nbevan@usability.serco.com 2002 PRUE Consortium Extracts may be reproduced provided the source is acknowledged Contents  TOC \o "1-2" 1. Executive Summary  PAGEREF _Toc414417994 \h 5 2. Guide for Suppliers  PAGEREF _Toc414417995 \h 6 2.1 Why does usability matter?  PAGEREF _Toc414417996 \h 6 2.2 Usability as part of a contract with the customer  PAGEREF _Toc414417997 \h 6 3. Guide for Customers and Procurers  PAGEREF _Toc414417998 \h 7 3.1 Why does usability matter?  PAGEREF _Toc414417999 \h 7 3.2 Usability as part of a contract with the supplier  PAGEREF _Toc414418000 \h 7 3.3 Usability as part of acceptance testing  PAGEREF _Toc414418001 \h 8 4. Specifying and testing usability requirements  PAGEREF _Toc414418002 \h 9 4.1 How to specify usability requirements  PAGEREF _Toc414418003 \h 9 4.2 How to test usability requirements  PAGEREF _Toc414418004 \h 10 5. What is the Common Industry Format?  PAGEREF _Toc414418005 \h 12 6. Overview of case studies  PAGEREF _Toc414418006 \h 13 7. Italian Ministry of Justice/SEMA: Legal Web Site  PAGEREF _Toc414418007 \h 14 7.1 Overview  PAGEREF _Toc414418008 \h 14 7.2 Background  PAGEREF _Toc414418009 \h 16 7.3 Context of use  PAGEREF _Toc414418010 \h 17 7.4 Evaluation of existing system  PAGEREF _Toc414418011 \h 17 7.5 Evaluation results  PAGEREF _Toc414418012 \h 18 7.6 Satisfaction results  PAGEREF _Toc414418013 \h 19 7.7 Evaluation conclusions  PAGEREF _Toc414418014 \h 19 7.8 Conclusions  PAGEREF _Toc414418015 \h 19 8. Ericsson/ Electrolux: Travel Management System  PAGEREF _Toc414418016 \h 21 8.1 Overview  PAGEREF _Toc414418017 \h 21 8.2 Background  PAGEREF _Toc414418018 \h 22 8.3 Purpose of the trial  PAGEREF _Toc414418019 \h 23 8.4 How the users, tasks and measures were selected  PAGEREF _Toc414418020 \h 23 8.5 Summary of CIF results  PAGEREF _Toc414418021 \h 24 8.6 Benefits to Ericsson Enterprise from using the CIF  PAGEREF _Toc414418022 \h 27 8.7 Benefits to Electrolux from using the CIF  PAGEREF _Toc414418023 \h 27 8.8 Discussion  PAGEREF _Toc414418024 \h 27 9. Prezzybox/in2netlogic: Ecommerce Web Site  PAGEREF _Toc414418025 \h 29 9.1 Overview  PAGEREF _Toc414418026 \h 29 9.2 Background - The evaluation of a shopping website  PAGEREF _Toc414418027 \h 30 9.3 Purpose of the trial  PAGEREF _Toc414418028 \h 30 9.4 Trial plan, tasks and user selection  PAGEREF _Toc414418029 \h 31 9.5 Summary of CIF results  PAGEREF _Toc414418030 \h 31 9.6 Experiences from using the CIF  PAGEREF _Toc414418031 \h 33 9.7 Feedback on the CIF from the consumer and supplier organisations  PAGEREF _Toc414418032 \h 33 9.8 Discussion  PAGEREF _Toc414418033 \h 35 9.9 Conclusions what other shopping sites can learn  PAGEREF _Toc414418034 \h 35 10. Robissa Travel/ExcelSoft: Travel Agent Software  PAGEREF _Toc414418035 \h 36 10.1 Overview  PAGEREF _Toc414418036 \h 36 10.2 Case study  PAGEREF _Toc414418037 \h 37 10.3 Benefits for the participants  PAGEREF _Toc414418038 \h 40 10.4 Conclusions  PAGEREF _Toc414418039 \h 41 11. Usability of the CIF standard  PAGEREF _Toc414418040 \h 42 11.1 Correct implementation of the CIF template  PAGEREF _Toc414418041 \h 42 11.2 The validity and appropriateness of the measures used  PAGEREF _Toc414418042 \h 42 11.3 Any differences in how the template has been implemented  PAGEREF _Toc414418043 \h 43 11.4 Satisfaction with the CIF standard  PAGEREF _Toc414418044 \h 45 12. Trial conclusions  PAGEREF _Toc414418045 \h 47 13. References  PAGEREF _Toc414418046 \h 48  Executive Summary This document provides a guide for how to specify and test usability requirements as part of a contractual relationship between a supplier and acquirer. It includes case studies of four trials carried out in Europe by the EU-funded PRUE project. Incorporating usability requirements in the procurement process can reduce the risk of failure when implementing a newly acquired system and increase ease of use and thus productivity and/or profitability. Lack of user performance requirements was a fundamental reason for the expensive costs and delays incurred when new passport issuing software developed by Siemens was installed in the UK [12]. Two studies have shown that the user success rate in purchasing from current ecommerce web sites is in the range of 25-60% [9,12]. Small improvements in user performance could lead to substantial increases in revenue. Defining requirements for user performance and satisfaction is not difficult to do, and involves three related activities: analysing the context of use, defining task scenarios, and specifying testable requirements for effectiveness, efficiency and satisfaction for each scenario. Evaluating usability requirements needs a carefully designed usability test with at least 8 representative users carrying out realistic tasks. A new Common Industry Format for documenting usability results has been developed by a US-based group of companies coordinated by NIST [1,4]. The format has been approved as an American standard (ANSI/NCITS-354-2001), and is intended to be submitted to ISO. The EU-funded PRUE project [11] has demonstrated the value of using the Common Industry Format in four case studies applied to different situations: public and private contracts for development of a web site, and acquisition of a travel management system and travel agency software. Serco worked with the Italian Ministry of Justice to introduce usability requirements into in the acquisition of a new legal information web site, and SUA worked with Ericsson to introduce usability requirements in the procurement of an office software product. Loughborough University and SIEM used the CIF to evaluate the effectiveness of an existing system to assess its acceptability and the need for improvements as part of a contractual relationship with a supplier. Loughborough assessed an online shopping website, and SIEM assessed travel agency software. Guide for Suppliers Why does usability matter? Customers have increasing expectations of usability, so when developing a product it is important to include usability with other requirements and ensure that the usability requirements have been met. Incorporating user-centred methods into the development process can reduce development time, increase sales and reduce support costs. Reduced development time Focussing on user and organisational needs will reduce the development time by: reducing the late changes otherwise needed to produce a product that meets user needs reducing the cost of future redesign of the architecture to make future versions of the product more usable Increased sales marketing the product as easier to use than the competition provides an increased competitive edge repeat sales will be made to more satisfied customers the usability of the product will be given higher ratings in the trade press Support savings reduced costs of producing training materials reduced help line support See the case studies (Section 6) for examples of the benefits that can be obtained by suppliers. Usability as part of a contract with the customer The customer may wish to include usability requirements in the contract. Issues to consider when negotiating the content of the contract include: To be meaningful the requirements should specify task scenarios which are realistic for the customers users. It is difficult to assess the cost and risk of developing a system to meet specific performance and satisfaction criteria, unless these are derived from an existing comparable system (see  HYPERLINK "http://www.usabilitynet.org/methods/requirements/existing.asp" http://www.usabilitynet.org/methods/requirements/existing.asp). Requirements that the new system must be at least as usable as the existing system provide reasonable assurance for the customer without being unduly demanding for the supplier. Guide for Customers and Procurers Why does usability matter? Inadequate usability can lead to significant additional costs associated with poor productivity and increased support overheads. It can also significantly increase the risk that implementation of a new system may fail. It is therefore important to include usability requirements in procurement specifications, and to test usability prior to purchase. The potential benefits of increased usability are: Usage savings reduced task time and increased productivity fewer user errors that have to be corrected later fewer user errors leading to increased quality of service less training, support and documentation is required reduced staff turnover as a result of higher satisfaction and motivation Support savings reduced costs of producing training materials reduced time providing training reduced time spent by other staff providing assistance when users encounter difficulties reduced help line support Improve the quality of life less stress from frustrating software users are more satisfied Health and safety legislation The European Directive on Display Screen Equipment (implemented in the national legislation of EU countries) requires that software is suitable for the task and easy to use. See the case studies (Section 6) for examples of the benefits that can be obtained by consumers. Usability as part of a contract with the supplier A potential supplier may be concerned at the increased risks and costs associated with developing a system to meet specific usability requirements. A conservative strategy is to require that the new system must be at least as usable as an existing system (see  HYPERLINK "http://www.usabilitynet.org/methods/requirements/existing.asp" http://www.usabilitynet.org/methods/requirements/existing.asp). This protects against the risk of the costs and overheads associated with reduced usability. If the existing system has known problems, some improvement may be required. If a satisfaction questionnaire such as SUMI or WAMMI is used that has population norms, it is possible to require that user satisfaction is at least as great as the industry average. To be meaningful the requirements should specify task scenarios which are a realistic representation of the expected usage. Usability as part of acceptance testing Ideally testing should involve at least 8 users to assess whether predefined requirements have been met. However, even if no specific requirements have been defined, usability testing will still identify any areas where users have difficulties with the tasks. If the main objective is to identify problems rather than obtain measures, it is more effective to test more tasks with fewer users (4 or 5 per task). In both cases it is important to use representative users carrying out realistic tasks. Specifying and testing usability requirements How to specify usability requirements The Common Industry Format provides a good structure for requirements for effectiveness, efficiency and satisfaction. A template for requirements can be found at www.usability.serco.com/prue. Defining usability requirements involves three related activities: analysing the context of use, defining task scenarios that can be tested, and specifying requirements for effectiveness, efficiency and satisfaction for each scenario. Analyse context of use The first stage is to collect and agree detailed information about: Who are the intended users and what are their tasks? (Why will they use the system? What is their experience and expertise?) What are the technical and environmental constraints? (What types of hardware will be used in what organisational, technical and physical environments?) Arrange a half-day meeting. Invite stakeholders who have knowledge about the intended users and usage. This may include: project manager user representative(s) developer(s) training support The first two are key areas. You will also need a facilitator and a person to record the information provided during the meeting. To obtain information on the context of use, a detailed checklist will be needed. Discuss and fill in each item on the context checklist. Try to obtain consensus where there is uncertainty or disagreement. If information is missing, agree how this can be obtained. Avoid prolonged discussion of minor issues. Define scenarios of use Next identify scenarios of use (use cases) to provide examples of usage as an input to design, and to provide a basis for subsequent usability testing. Scenarios specify how users carry out their tasks in a specified context. To maintain design flexibility, they should not specify what product features are used. Try to generate scenarios to cover a wide range of situations, not just the most common ones or those of most interest to the design team. Try to include problem situations that will test the system concept, not just straightforward scenarios. Specify requirements for effectiveness, efficiency and satisfaction For each chosen task and user type, estimate: the acceptable task time and the optimum target how to score effectiveness by agreeing what errors the user might make the effectiveness target the satisfaction target. If there is an existing or competitor system, it can be evaluated by the selected users and tasks with the objective that usability for the new system should be at least as good as for the old system. For more information see the methods in the Requirements column in the table of methods at  HYPERLINK "http://www.usabilitynet.org/methods" http://www.usabilitynet.org/methods How to test usability requirements To obtain valid results, at least 8 representative users need to be tested with realistic tasks. Usability testing needs to be carefully planned and carried out. In addition to assessing whether usability requirements have been met, major usability problems can be identified, including problems related to the specific skills and expectations of the users. Planning It is important that the users, tasks and environment used for the test are representative of the intended context of use. Select the most important tasks and user groups to be tested. Select users who are representative of each user group. 8 or more users of each type are required for reliable measures. Produce a task scenario and input data and write instructions for the user (tell the user what to achieve, not how to do it). Plan sessions allowing time for giving instructions, running the test, answering a questionnaire, and a post-test interview. Invite developers to observe the sessions. If developers cannot be present, videotape the sessions, and show developers edited clips. Two administrators are normally required: one to interact with the user, and one to note problems and to speak to any observers. If possible use one room for testing, linked by video to another room for observation. Observe the user without making any comments. Running sessions Welcome the user, and give the task instructions. Do not give any hints or assistance unless the user is unable to complete the task. Observe the interaction and note any problems encountered. Time each task. At the end of the session, ask the user to complete a satisfaction questionnaire such as SUMI. Interview the user to confirm they are representative of the intended user group, to gain general opinions, and to ask about specific problems encountered. Assess the results of the task for accuracy and completeness. Reporting Summarise the results of the satisfaction questionnaire, task time and effectiveness (accuracy and completeness) measures. Document the results in the Common Industry Format. Produce a list of usability problems, categorised by importance (use post-it-notes to sort the problems), and an overview of the types of problems encountered. The summative measures will help place the problems in order of importance. Arrange a meeting with the project manager and developer to discuss whether and how each problem can be fixed. For more information see  HYPERLINK "http://www.usabilitynet.org/methods" http://www.usabilitynet.org/methods/testmeasure/testing.asp What is the Common Industry Format? The Common Industry Format (CIF) for usability test reports specifies the format for reporting the results of a summative usability evaluation. The most common type of usability evaluation is formative, i.e. designed to identify usability problems that can be fixed. A summative evaluation produces usability metrics that describe how usable a product is when used in a particular context of use [3,7]. The CIF report format and metrics are consistent with the ISO 9241-11 [5] definition of usability: The extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use. The type of information and level of detail that is required in a CIF report is intended to ensure that: Good practice in usability evaluation had been adhered to. There is sufficient information for a usability specialist to judge the validity of the results (for example whether the evaluation context adequately reproduces the intended context of use). If the test was replicated on the basis of the information given in the CIF, it should produce essentially the same results. Specific effectiveness and efficiency metrics must be used, including the unassisted completion rate and the mean time on task. Satisfaction must also be measured. It was envisaged that a supplier would provide a CIF report to enable a corporate purchaser to take account of usability when making a purchase decision. A purchaser could compare CIF reports for alternative products (particularly if a common set of tasks had been used). The purchaser might specify in advance to a supplier the required values of the usability measures (for example based on the values for an existing product). The original motivation for the CIF came from usability staff in purchasing companies who were frustrated at purchase decisions made exclusively on the basis of functionality. These companies experienced large uncontrolled overhead costs from supporting difficult to use software [4]. The CIF format was agreed by a working group of usability experts from purchasing and supplying companies, including companies such as IBM, Microsoft, Hewlett-Packard, Boeing, US West and Kodak.. It was based on collating good practice from the different companies, and aligning this with ISO 9241-11. Overview of case studies The main purpose of the PRUE project was to evaluate the business value of introducing the CIF into the purchasing and procurement process for different tpes of systems in a variety of contractual environments and countries. One trial is for the originally envisaged role in the acquisition of corporate software (Ericsson/Electrolux). Another extends this to acquisition of a custom-designed web site (Ministry/SEMA). These trials both focus on evaluating whether a new web-based system meets requirements established by prior evaluation of the existing text-based system. The two other trials assessed the usefulness of the CIF in negotiating for improvements in a custom-designed system, by evaluating the usability of an existing product to establish requirements for an improved version. The in2netlogic/prezzybox. com trial was in the UK where usability is well-established as an issue, and Robissa Travel/ExcelSoft was in Greece where usability is less well recognised. Italian Ministry of Justice + SEMA Ericsson Telecom + Electrolux in2netlogic + prezzybox. com Robissa Travel + ExcelSoft Product Web site: legal infoTravel expense reportingWeb site: ecommerce Travel management UsersLegal staff and publicOffice staffPublicTravel agent staffExisting systemTraditional dial upText basedExisting web siteExisting productNew systemWeb basedWeb basedImproved versionImproved versionContractual relationshipThe Ministry will use CIF results as part of acceptance criteriaEricsson will use CIF results as part of acceptance criteriaBasis for negotiation for improvementsBasis for negotiation for improvementsConsumer organisation benefitsConvenient and accurate legal informationIncreased productivity and quality of travel informationIncreased business Increased office efficiency  PRUE was also evaluating the usability of the CIF itself. How easy to use and appropriate would the CIF be for use in these trials by the usability specialists from each of the partners? See Section 11. For information on case studies in the USA, see the IUSR web site: http://www.nist.gov/iusr. Italian Ministry of Justice/SEMA: Legal Web Site Overview PRUE has helped the Italian Ministry of Justice introduce usability requirements into the procurement of a new enhanced version of the existing dial-up ITALGIURE-FIND legal information service. SEMA has a contract with the Ministry to produce a new system accessible from internet browsers and mobile phones. No information was previously available on the usability of the existing system, and there were no usability requirements in the contract (although the contract required ISO 13407 [6] to be used). PRUE planned three activities to introduce usability to the procurement process: A stakeholder meeting to identify the importance of usability and the intended context of use An evaluation of the existing system to provide measures of usability as baseline requirements An evaluation of a prototype of the new system to establish whether the requirements have been met The stakeholder meeting was attended by senior representatives of the Ministry, SEMA and other stakeholders. Two user groups were identified: expert users with legal knowledge need functionality to support fast and efficient searches, consistent with the natural logic of juridical research. The system will also be accessible to new users without any legal knowledge who should be able to carry out simple searches without training. In the evaluation of the existing system, five experts and four non-experts attempted to complete different sets of typical tasks. Expert users successfully accomplished 63% of tasks in comparison with 52% of non-expert users. The evaluation also provided the opportunity for SEMA designers to observe real usage of the system. An important objective is for the new system to at least equal and if possible improve on these success rates. If the measured success rates and user satisfaction scores are not acceptable, the Ministry could negotiate with SEMA to provide an improved system. The trial has demonstrated several benefits of introducing summative usability testing into the procurement process: It provides a concrete benchmark for user performance and satisfaction, thus reducing the risk that the new system is more difficult to use (and therefore less successful) than the existing system. It highlights usability problems with the existing system that need to be addressed in the design of the new system. It provides specific goals for usability and gives developers the opportunity to became familiar with typical user task scenarios. It provides the framework for the more detailed usability work required by ISO 13407. Summative evaluation results reported in the CIF format have thus been very useful in helping the Ministry understand the needs of different user groups, and in establishing usability requirements for the new system. The user testing has also clear benefits in helping the supplier (SEMA) better understand Ministry of Justice requirements. Background PRUE has helped the Italian Ministry of Justice introduce usability requirements into the procurement of a new enhanced version of the existing dial-up ITALGIURE-FIND legal information service. SEMA has a contract with the Ministry to produce a new system accessible from internet browsers and mobile phones. The evaluation results from the existing system have provided a basis for establishing the requirements for the new system. Italgiure is the information retrieval software of the Corte di Cassazione. It returns sentences, case law, rules, summaries, etc. coming from the five databases of the Court: Database of the Senior Council of Magistrates: sentences of the Council Databases of the jurisprudence: sentences of Supreme Court, case law, civil jurisprudence, penal jurisprudence, etc. Normative laws databases: national laws, regional laws, circular letters, Ministry rules, etc. Doctrine databases: doctrine and juridical debates, juridical papers and journals, etc. Other databases Two versions are currently in use: ITALGIURE-FIND uses a command line interface, while EasyFind 5.0 is a windows-based form-filling interface for the Italgiure system. The new version of ITALGIURE-FIND will replace the existing services. The main enhancements are: Extend the number and size of files Widen availability by making access possible from internet browsers and mobile phones Improve the search interface, with versions for expert users and general users. PRUE planned three activities: A stakeholder meeting to identify the importance of usability and the intended context of use An evaluation of the existing system to provide measures of usability as baseline requirements An evaluation of a prototype of the new system to establish whether the requirements have been met Context of use The stakeholder and context meetings were attended by senior representatives of the Ministry, SEMA and other stakeholders. It was concluded that expert users with legal knowledge need functionality to support fast and efficient searches, consistent with the natural logic of juridical research. The system will also be accessible to new users without any legal knowledge who should be able to carry out simple searches without training. To carry out more complex searches, these users may need access to online training about the legal system and the use of efficient search strategies. Longer term objectives include packaging the legal information in a format more intelligible to the non-legal user, and incorporating additional databases. The environment in which the product is used is mainly an office and professional environment. Access will normally be by a web browser, but mobile phone access will also be supported. The new system should enable existing users to complete their searches at least as quickly and as accurately as before, with at least as much satisfaction The type of user work that is supported by the product is mainly the retrieval of legal information. Typical tasks are: a) expert users: whether the supreme court has decided a question, and if so how many times and what laws were applied b) general users: find a specific legal text by reference number, find judgments that relate to a defined legal problem. Potential user groups include people concerned with industrial relations, taxes and wages, damages claims, property disputes, marriage and divorce problems, patronage (insurance and pensions), university professors regarding university legislation. Evaluation of existing system The new system New ITALGIURE-FIND currently only has a temporary developer interface, so is not yet ready for evaluation. Design of the new user interface is just starting. The main objective of the evaluation was to establish a baseline for development of the new interface and to identify existing usability problems. To develop an easy to use interface developers needs to have a good understanding of how users perform their tasks at the moment, and any difficulties they have. In order to fill this gap an evaluation focused on the current interface of ITALGIURE-FIND named EasyFind 5.0 was carried out. Two different databases were considered in the evaluation: Civil jurisprudence, coming from the general databases of jurisprudence; the Normative Laws Databases. The user population is composed of: internal expert employees; internal inexperienced employees; different kinds of subscribers such as lawyers, juridical professional and practitioners, researchers, citizens. No groups with special needs are supported by the product. 9 persons divided in two categories (5 expert and 4 non expert ITALGIURE-FIND users) were tested. Tasks Simple tasks for non expert (n=10) and more difficult tasks for experts (n=14) were used: expert tasks are typically related with decisions of the Supreme Court together with the detail of the laws that were applied and the frequency pursuance; non expert tasks are typically related with finding specific legal text by specifying its reference number or retrieving judgments concerning a certain legal problem. The selected tasks were defined by the training manager, based on her experience in teaching the EasyFind interface in the training course. The tasks were selected as representative of general tasks for both expert and non expert users. Procedure Users were informed that the usability of EasyFind 5.0 was going to be tested to check whether the system met the needs of the users. They were told that it was not a test of their abilities. No pre-task training was used. Users were then given 20 minutes to complete all tasks. When each task finished the correctness was checked and time completion was recorded. If the user required assistance the task was considered as failed. At the end of the session users completed the SUMI questionnaire. After the experimental session a debriefing was conducted in order to collect information about how to improve the new interface. Evaluation results Results in terms of effectiveness, efficiency, and satisfaction were obtained. In particular, effectiveness was measured as "number of tasks successfully accomplished"; efficiency in terms of "time to complete successfully all the task", satisfaction as SUMI scores. The main reason for conducting the test was to get a baseline for the evaluation of the new system by comparing the users' performances and satisfaction whenever carrying on the same tasks in the old and new system. Results can be summarized as follows: Effectiveness: non expert users successfully accomplished an average of 52% of tasks in comparison with 63% of tasks by expert users; Satisfaction: non expert users judged the interface as satisfactory as expert users (SUMI score is 55 for the two categories of users). Satisfaction results Italgiure survey A SUMI survey of 7 existing internal Italgiure users showed a very consistent positive attitude to Italgiure. Users liked it very much and found it efficient, though they did not find it easy to learn, in particular they did not feel that the software helped them overcome problems they had in using it. Easyfind survey In contrast, although a SUMI survey of 8 existing external Easyfind users showed that they quite liked it, they had a consistently negative opinion of the efficiency, helpfulness, controllability and learnability. Easyfind evaluation In the Easyfind evaluation, the opinions of the expert and inexpert internal users were similar and generally quite positive despite the difficulty in completing all tasks. Evaluation conclusions Italgiuri The survey results confirmed that existing internal expert users are very satisfied with the Italgiure command line interface. Easyfind The success rates for expert and inexpert tasks of 63% and 52% were not high, and there was dissatisfaction with some aspects of the interface. Results show that the new interface has to be both: At least as good as Italgiure Find for internal expert users; Better than Easy Find 5.0 for both internal non expert users and general public. This is the most important feedback for designers of the new interface (New Italgiure Find) as these two search interfaces (Italgiure Find and EasyFind 5.0) are very different in terms of required skills and working experience. Starting from the test results the baseline objectives for the new system will be provided within next December. Such objectives (both for expert and non expert) will be the baseline in order to compare usability characteristics of the New Italgiure Find interface with the existing interfaces (Italgiure Find and EasyFind 5.0). A planning for future work has been defined as well. Conclusions Summative evaluation results reported in the CIF format have been very useful in helping the Ministry understand the needs of different user groups, and in establishing usability requirements for the new system. Ministry of Justice benefits The work done within the PRUE project so far has proven to be of great value for Ministry of Justice. The initial tests lead to development of a usability requirements document for the new interface. It has been seen as a new and positive input to the Ministry of Justice decision process when selecting a supplier of a new system. The usability requirements are regarded to be good compliment to the functional and technical specifications. The benefits are the increased understanding of user performance in current system, understanding of what user think of the current system and what they require from the new. Ministry of Justice is considering products from other supplier organizations than the one tested, and the usability requirements document will be one factor supporting the assessment of these alternatives. Supplier organization benefits The user testing of the supplier organizations product (SEMA) has also clear benefits. They have been able to understand Ministry of Justice requirements, assess their product and improve the interface. Ericsson/ Electrolux: Travel Management System Overview In order to support the procurement process, the common industry format (CIF) for usability testing has been used to obtain usability metrics at Ericsson and Electrolux in Sweden. First, the current Ericsson system was tested using the CIF to obtain a baseline value for usability, which resulted in an usability requirements document. The document required information about the way in which the supplier had developed the product and detailed information about user performance and user satisfaction metrics. Secondly, the likely supplier of a new system called WEBRES developed by Electrolux was tested and the results compared to the requirements document. Ericsson was investigating the purchase of a new interface to the existing travel management system - RES. The current text-based interface no longer supported users task demands and requirements effectively and it was not found easy to use. RES was used by about 25% of Ericsson employees although the goal was to increase this percentage significantly. It was a complete system that supported business travel management both nationally and internationally. Ericsson did not wish to make significant changes to the underlying system. They looked to a number of potential suppliers who were able to provide a new interface, which would enhance the efficient and effective use of the existing system. At this time, the prime candidate supplier was Electrolux. They used essentially the same travel management system as Ericsson but had developed in-house interface solution - WEBRES. This interface was a possible replacement to the current RES interface. At this time, Ericsson had not considered any usability issues during the purchase process. There was no objective data about user performance/subjective assessment using RES and there was no usability requirements document. The PRUE project was able to support the procurement process by conducting summative usability testing on RES using the CIF format. The objective was to obtain user performance and satisfaction metrics of the RES interface through formal user testing. The results from testing led to a usability requirements document against which potential suppliers could be assessed. Subsequently, similar testing using the CIF was conducted on Electroluxs WEBRES. The test results were then compared and contrasted to the usability requirements document prepared by Ericsson. PRUE has been of great value to Ericsson. Without PRUE, usability issues would not have been included in this procurement process. PRUE has resulted in a usability requirements document for a new interface to RES, which was regarded as a new and positive input to the Ericsson decision process when selecting a supplier organization. The usability requirements were regarded as a good compliment to the existing functional and technical specifications. Further, there were benefits of increased understanding of user performance using the current system (poor performance is a large cost in time to Ericsson), an objective understanding of what users thought of the current system and what they require from the new. From the supplier perspective, Electrolux have been able to understand the performance/ satisfaction related Ericsson requirements for WEBRES. The CIF for usability testing enabled them to assess how WEBRES compared to the Ericsson requirements, and it has provided a clear indication of what improvements need to be made to the interface design. Background At the start of the PRUE project, Ericsson was investigating the purchase of a new interface to the existing travel management system - RES. The current text-based interface no longer supported users task demands and requirements effectively and it was not found easy to use. The interface lacked overview of the travel claim and did not support the wide variation and complexity of travel carried out today. There are usability problems, such as use of many codes and abbreviations, plus poor readability. RES was used by about 25% of Ericsson employees although the goal was to increase this percentage significantly. It was a complete system that supported business travel management both nationally and internationally. Further, managers use RES to approve and attest business travel and expenses, financial controllers use it to review travel claims and RES is connected to other financial systems to ensure that staff are reimbursed quickly and efficiently for travel expenses. It also provides extensive statistical travel data for the Ericsson Enterprise As Ericsson did not wish to make significant changes to the underlying system, they looked to a number of potential suppliers who were able to provide a new interface, which would enhance the efficient and effective use of the existing system. At this time, the prime candidate supplier was Electrolux. Electrolux used essentially the same travel management system as Ericsson but developed in-house interface solution - WEBRES. The interface was an Internet solution and a possible replacement to the current RES interface. At this time, Ericsson had not considered any usability issues during the purchase process. There was no objective data about user performance/subjective assessment using RES and there was no usability requirements document. Ericsson agreed to participate in the PRUE trial application to assess the benefits of using the CIF in this procurement process and also to assess the benefits of using summative usability testing in general within the organisation during procurement. As the prime supplier organisation at this time, Electrolux agreed to participate in the trial application. Users had not formally tested WEBRES for usability. Within the context of this process, PRUE has carried out five main activities: A stakeholder meeting to present the importance of usability and specify the intended context of use of the Ericsson travel management system. A formal user evaluation of the existing system RES - according to CIF to provide measures of usability as baseline requirements. A usability requirements document (not in the public domain) to be used by Ericsson when assessing possible alternatives to RES. The document required information about the way in which the supplier had developed the product and required the supplier to provide detailed information about user performance and user satisfaction metrics. A formal user evaluation of the Electrolux travel management system WEBRES to establish whether the Ericsson requirements had been met. An initial benefit analysis of using the CIF to the consumer organisation and the supplier organisation. Purpose of the trial The main objective of the trial application was to evaluate whether one possible successor to the existing travel management system at Ericsson, met specified usability requirements. This was demonstrated by applying the common industry format for usability testing during this procurement process to the current Ericsson system (RES) first and then the Electrolux system (WEBRES). First, user performance and satisfaction metrics of the current RES interface were obtained through formal user testing following the CIF for usability testing. The test results were fed into a usability requirements document in the form of usability goals and against which alternative systems/interfaces could be judged. Subsequently, using very similar formal user testing following the CIF, user performance and satisfaction metrics of the WEBRES user interface. The test results were compared to the Ericsson usability requirements document. This provided Ericsson with a further decision basis upon which to assess the Electrolux product. How the users, tasks and measures were selected Usability metrics The metrics and measures defined in the CIF were used; the effectiveness parameter - Task completion (%), the efficiency measure task time (min), number of errors and number of assists. User satisfaction was rated using the standardised questionnaire SUMI for the RES test and WAMMI for the WEBRES test. Results of SUMI and WAMMI can be correlated to each other to make comparison of results possible. Participants Typical users from the Ericsson Enterprise carried out the testing. A pilot test was carried out at the beginning of the test sessions. Test participants were selected according to two criteria. Firstly, they were Ericsson employees. Secondly, they were familiar with the travel management process and completed travel management accounting procedures up to five times per year. One key group was Consultants, who use the travel management system occasionally. Another group, secretaries who help others create their travel claims reports, was included. They were experienced and efficient using the system. Selection of users for the tests was done via a large e-mail mail shot to Ericsson employees. Test participants familiar with RES were selected according the defined user profile (8 for RES and 7 for WEBRES). The users tested fulfilled the user profile, although some claimed more than 5 business trips per year. Travel varies depending on project work. The tests were conducted in Swedish, the tasks were in Swedish and users were Swedish. RES and WEBRES use Swedish primarily, although parts of them are in English. The corporate language is English. Tasks After some interviews with relatively experienced business travellers, selected tasks for the study were the most frequent travel management tasks at Ericsson Enterprise. Tasks were typical and not complex. Business trips over multi-destinations and long time periods were not selected. The tasks are summarised below: Task 1: Create a travel application (not applicable to WEBRES) Task 2: Create a foreign travel report return travel to Vienna. Task 3: Create a travel report for domestic travel with train and rental car. Task 4: Create a travel report for various own car-related expenses. Users were provided with simulated travel vouchers/receipts for each of the tasks. As users had not actually experienced the business trip, they were given some time to understand the information before starting on the task. Procedure The design of the test followed a logical sequence of events for each user and across tests. One person ran the test and one observed. Other persons related to the project or the development team at Ericsson and then Electrolux observed the testing. The testing procedure was as follows: User were greeted and offered refreshment. Users were given information about the goals and objectives of the test. Users were asked a set of predefined background questions prior to the testing Users were then given a series of clear instructions specific for the test. Users were asked to complete the four tasks in order quickly and efficiently. Users were asked to complete a SUMI/WAMMI questionnaire direct after completing the last task. Users were asked a set of predefined background questions after the testing Users thanked and for their participation were given cinema tickets worth 160 SEK. As users were familiar with the product and the process of travel management there were no non-disclosure agreements, form completion, warm-ups or pre-task training. Summary of CIF results All users had problems in performing the tasks and they made many mistakes. Some of the mistakes were minor, some were easily corrected with/without assistance and some were so severe that the user did not succeed at all. In the interviews preceding the tests many users indicated that they normally have problems with their travel reports after business trips. RESWEBRESTask #Unassisted Task Effectiveness [(%)Complete]Task Time (min) medianErrorsAssistsUnassisted Task Effectiveness [(%)Complete]Task Time (min) medianErrorsAssists1803,34114----25710,472015022,001313637,4580677,50714582,0630675,2550Table 1: Summary of usability metrics by user. The tables above show no major differences in user performance between the two tested systems except for task time and satisfaction. Error rates were unexpectedly high in both systems. The number of assists required when testing the current system was also unexpectedly high. The web-based WEBRES was slower in two cases out of three but was still perceived as better by the users as indicated by the satisfaction measurements. User satisfaction was rated using the standardised questionnaire SUMI for the RES test and WAMMI for the WEBRES test. Results of SUMI and WAMMI can be correlated to each other to make comparison of results possible. RESWEBRESMedianMedianAffect1730AttractivenessControl2860ControllabilityEfficiency2234EfficiencyHelpfulness2125HelpfulnessLearnability2923LearnabilityGlobal2045Global UsabilityThe overall score for RES on the SUMI satisfaction questionnaire was 20. The value of 50 is the industry average SUMI score. The overall score for WEBRES on the WAMMI satisfaction questionnaire was 45. The target value of 50 is the industry average WAMMI score.Table 2: User satisfaction results a comparison. While caution should be used when comparing the subjective assessment results, it is clear that RES performed poorly as it was not liked by users despite the fact that they used RES relatively regularly. WEBRES on the other hand performed quite well on the WAMMI scale with a high controllability value. Effort would need to be taken to improve attractiveness and efficiency values. This relatively high subjective value was interesting as user performance was actually slightly lower than RES. Neither systems were acceptable as the global scores of 20 and 45 were less than the average industry average values of 50 for the standardised SUMI and WAMMI questionnaires. In terms of requirements, Ericsson would like a replacement system/interface to be at lest as acceptable as the industry average, which means having a score of at least 50. . RESWERRESMean task completion rate for all tasks 55% 56%Mean number of total errors per user 5 3.57Mean total task time 28 min 34 minutesCompletion rate efficiency (completion rate per minute)2.9 3.0Global score on the SUMI/WAMMI user satisfaction questionnaire 20 (SUMI) 45 (WAMMI)Table 3: Summary the overall results by metrics. The overall results presented in the table above indicate that WEBRES performs slightly better than RES (high overall task completion, few errors). However, mean total task time was longer. There was no significant difference for completion rate efficiency. Users preferred the WEBRES interface. In conclusion, while users preferred WEBRES they did not perform as well as expected when using it. The results indicate that the design of the WEBRES interface needs to be improved so that it supports typical tasks more efficiently. Benefits to Ericsson Enterprise from using the CIF Ericsson assesses that the work done within the PRUE project so far has proven to be of great value. Initial testing lead to the creation of a usability requirements document for the new travel management system. This was regarded as a new and positive input to the Ericsson decision process when negotiating with Electrolux over the new interface to the current system. The usability requirements are regarded to be good compliment to the functional and technical specifications. The usability requirements document will support Ericsson during purchase negotiations with Electrolux, or any other supplier of travel management systems. Testing using the CIF has led to increased understanding of user performance and subjective assessment in current RES system. This has been considered in internal financial terms as poor performance results in a substantial cost to Ericsson. Testing has given Ericsson an clearer understanding of what users think of the current system and what they require from the new. Ericsson is considering products from other supplier organizations than the one tested, and the usability requirements document will be one factor supporting the assessment of these alternatives. There is an interest from Ericsson to continue using the CIF as a basis summative testing during procurement and requirements work both within this specific project and in general across the organisation. Ericsson will support the proposed dissemination activity plan. Benefits to Electrolux from using the CIF Electrolux assesses that the PRUE project has led to the following benefits : Testing of RES using the CIF and the creation of a usability requirements document for the new travel management system has helped Electrolux understand Ericsson usability requirements and the importance of user performance and subjective assessment. Testing WEBRES using the CIF has enabled Electroluxs to compare the performance of their product against the usability requirements document. Testing has given Electrolux an clearer understanding of user performance and user satisfaction of WEBRES. It has provided an indication as to how efficiently WEBRES supports the basic and typical tasks. Testing has given them a basis upon which to improve the WEBRES interface. Discussion Summative usability testing using the CIF has advantages to both consumer and supplier organisations during procurement. It one of the most effective ways to enrich the consumers requirements document with objective user performance and satisfaction metrics (based on the existing system/product used). Consumers will find that it provides an important input when evaluating potential competitive products from a number of supplier organisations during the procurement of a new system/product. By using the CIF test structure, suppliers are able to demonstrate that their product complies to the usability metrics defined in the requirements document. It will also provide them with information as to how to improve the product/system from the user/usability perspective. It will also encourage suppliers of products and systems to apply principles of user-centred design during the development of their product/system. Prezzybox/in2netlogic: Ecommerce Web Site Overview This study was concerned with an online shopping website called Prezzybox. The site, developed by in2netlogic, offers a wide range of gift ideas for purchasers to select and have delivered to a friend or relative or to themselves. The website was evaluated by the PRUE partner - the Research School of Ergonomics and Human Factors (RSEHF) at Loughborough University (which incorporates the former HUSAT). The main objective of the user test was to obtain user performance and satisfaction metrics for Prezzybox along with user comments to highlight any problems with the site. Testing was carried out with 12 users who were all experienced with the Internet and interested in buying from the Prezzybox site. Users were asked to make a real purchase from the site - this being their payment for taking part. Thus the evaluation was designed to test the success of users in completing an online purchase - a crucial success factor for any online shopping site. The result of the evaluation was a CIF report, which documented user performance and satisfaction with the site. This showed that while 10 out of the 12 users were able to make a purchase, 2 failed to do so. If the consumer organisation could also capture those two users, their sales would increase by 20%! These performance results were also supported with user satisfaction ratings. The levels of satisfaction recorded were 'satisfactory' showing that there is room to improve the 'user experience' when using the site. Evaluator comments and recommendations for change to the site were also included, highlighting features that could be changed to help improve user success in making a purchase. The benefits of the CIF to the consumer organisation (Prezzybox) are: To find out how successful consumers will be in making a purchase from their site i.e. what percentage can actually make a purchase? To provide a benchmark for user performance and attitude, which can be used for comparison with the shopping site when it is revised. To obtain insights into any problems that users face when using the site (to complement the summative results) and to receive suggestions for improving the site. The benefits of the CIF to the supplier organisation (in2netlogic) are: To obtain objective feedback on the success of the design they produced. To identify the most important usability issues that will enable the shopping site to support more successful purchases and therefore improve the profitability of the site for the consumer organisation. A new contract to improve the site, based on the test results and the comments and suggestions for improving the site. Currently the Prezzybox site is being refined. If possible a second round of usability evaluation will be performed so that the consumer and supplier organisations can receive concrete evidence that the site has been improved. It is also hoped to show that this usability activity has enhanced the site from the user's point of view. In summary, the evaluation approach of the CIF is recommended for other online shopping providers in order to test what proportion of users actually make a purchase from their site. It also sets a baseline against which new versions and upgrades to the site can be compared, while also highlighting the problems that need to be fixed if the site is to achieve a greater number of online sales. Background - The evaluation of a shopping website The Web is now populated with many online shopping sites but evidence seems to show that many users fail to make a purchase from them. In the United states, a study carried out in 1999 by Jared Spool who tested a series of shopping websites showed that for even the most successful site only 42% of users were able to make a purchase. For the least successful site, only 28% were able to successfully buy online. Recent data from another leading human factors researcher, Jakob Nielsen, has shown a success rate for online shopping of between 50% and 60%. These figures, which are at best moderate, clearly will have a major effect on reducing profit margins in the e-commerce sector and will also restrict the wider use of online shopping in the future. But how can a design team know whether their online shopping site will be successful in the marketplace? An expert evaluation by Human Factors personnel, or an informal study by users may highlight problems, but how can the online shopping service provider know objectively measure how successful their site will be? The only way is to ask a sample of users to try and purchase an item from the site 'for real' and for an evaluator to observe whether they are successful. This was the approach adopted by the PRUE partner, RESHF - Research School in Ergonomics and Human Factors (formerly HUSAT) at Loughborough University. The Institute were asked by the in2netlogic web site development company help them evaluate an online present buying website that they had produced for Prezzybox, a midlands-based mail-order company supplying novel and high quality gifts to the public via an online shopping site. The CIF (Common Industry Format) offers a useful framework for organising and reporting on such a study. The EC supported PRUE project was set up to test the CIF in a range of different IT development situations. Thus the evaluation of the Prezzybox website was selected as RSEHF's case study within the PRUE project. Purpose of the trial Although the Prezzybox website had been running for some time, no formal testing of it with users had been carried out. Thus the main business objective of the user trial was to find out how easily users could find suitable items on the site to buy and whether they could successfully make a purchase. The benefits of the CIF to the consumer organisation (Prezzybox) are: To find out how successful users will be in making a purchase from their site i.e. what percentage can actually make a purchase? To provide a benchmark for user performance and attitude, which can be used for comparison with the shopping site when it is revised. To obtain insights into the problems that users face when using the site (to complement the summative results) and to receive suggestions for improving the site. The benefits of the CIF to the supplier organisation (in2netlogic) are: To obtain objective feedback on the success of the design they produced. To identify the most important usability issues that will enable the shopping site to support more successful purchases and therefore improve the profitability of the site for the consumer organisation. A new contract to improve the site, based on the test results and the comments and suggestions for improving the site. Trial plan, tasks and user selection A PRUE trial plan was designed to test the success of users in completing an online purchase - a crucial success factor for any online shopping site. Users were asked to make a real purchase from the site - this being their payment for taking part. The measurements to be carried out within the trial were essentially to record user performance in finding and buying a gift for themselves or for a friend and also satisfaction metrics with the whole process of using Prezzybox. User comments and evaluator observations were also recorded to highlight any problems with the site. Testing was carried out with 12 users who were all experienced with the Internet and interested in buying from the Prezzybox site. The user trials were carried out in RSEHF's offices via a laptop with modem telephone connection to the Internet to simulate shopping from home. Users were asked to make a real purchase from the site - this being their payment for taking part. Thus the test was designed to test the success of users in completing an online purchase - a crucial success factor for any online shopping site. Summary of CIF results The result of the evaluation was a CIF report, which documented user performance and satisfaction with the site. User performance The main performance metric was whether users could make a purchase. While 10 out of the 12 users were successful in making a purchase, 2 (16%) were not, which indicates a potential loss of sales. The reasons for the failure to buy were that one user was concerned about revealing personal information when completing details on the site, while the other was not successful in finding anything they really wished to buy. Arguably these findings would not have transpired with a traditional formative evaluation. Thus there is scope to improve the site, e.g. by explaining to the end users that any personal details entered (name, address, email, preferences, etc.) will not be passed on to a third party. Similarly, an improved indication of the range of search facilities available could help users find something they wish to purchase more effectively. The other metrics were: Mean total task time (45 minutes - which reflected strong user interest in the site, Completion rate efficiency for correctly completed tasks (2.7%), and 'Mean number of total errors' per user (0.9). There was also one minor assist each for two users. One user had looked through many items to purchase but wanted to know if there was any kind of search facility available. They were informed that they could look at the Gift Wizard. They investigated this facility but eventually selected an item identified previously. Thus the assist may have affected the task time but, arguably, not the final success in making a purchase. Another user was making a purchase and wanted confirmation that they did not need to register first. They did not expect to but wanted confirmation. Again the assist may have speeded up the process slightly. Without the confirmation the user would have spent a little longer looking at the screen instructions which informed them that registration was not necessary. The average time of 45 minutes is not a major issue and reflects more people's interest in finding an item on the Prezzybox site rather than difficulty in overcoming problems. (Within the total task times for each user, there was very little unproductive time spent in overcoming problems in using the system.) However any improvement in the navigation facilities, screen layout and download times may show an improvement in the performance scores as people are able to view items on the site more efficiently. User acceptance A Software Acceptability Questionnaire (SAQ) developed at Loughborough was applied which produced mean user ratings for a range of factors on a scale of 1 to 7. The scores were all reasonably high, i.e. Usefulness: 4.2, Clarity: 4.5, Efficiency: 4.6, Supportiveness: 4.5 and Satisfaction: 3.8, while leaving scope for improvement. The scores, which are mostly around the mid-point of the scale (4.0), show that people feel that the website is satisfactory, which is borne out by the fact that in general users enjoyed using the site. The usefulness score (4.2) reflects some peoples' comments that the site did not have all the kinds of items that they would have expected from a present or gift site. Some users wanted to be surer of the quality while others wanted the site to be more focussed. The score for clarity (4.5) shows that users had no significant problems in using the site, but could not scan the layout of the goods on screen as easily as they would have liked. Similarly, under efficiency (4.6), users could move around easily but most commented that the site could be quicker to display pages. This could have been a bigger issue if users themselves were paying the telephone bill. Again the score for support and help (4.5) is fairly high, reflecting the reassuring security statement and well guided purchase process. The satisfaction score is slightly below the midpoint (3.8). User comments The performance and acceptability scores were supplemented with user and evaluator comments, and recommendations for change, to the site. Users commented that Prezzybox was very interesting and enjoyable to use but they experienced problems with some aspects of the site: namely screen layout, item ordering, and needing a simpler means of specifying price range. Some users also failed to notice the 'Gift Wizard', a special feature of the site, provided to help the users select items to meet the characteristics of a particular person. Providing a clearer button for it could enhance user interaction with the site. Future plans Following dissemination of the report, the consumer and supplier organisations have met to discuss improvements to Prezzybox which will be implemented over time. If possible, a follow-up test will be carried out to show, objectively, if the site has improved from the user's point of view. Hopefully this approach will set the pattern of for testing e-commerce sites in the future to help them meet user needs and be easy to operate. Experiences from using the CIF Using the CIF has been an interesting experience. It has provided a useful framework for not only planning a user trial but on specifying the objective of the trial itself and developing a suitable plan to achieve it. The CIF recommends some basic metrics, which must be taken but also provides sufficient flexibility to apply other metrics more suitable to the application itself. The shopping study highlighted the interesting issue about whether not using a function could be classed as a type of 'error'. For example, within the Prezzybox website, the Gift Wizard is a special feature designed to help users find gifts to match the interests and personality of the person receiving the present. However this feature was not very prominent and most users did not realise that it was present. By missing out on this feature, they may not have used the site as efficiently as possible, so may be less inclined to return to it in future. The CIF does allow the report writer to add comments and suggestions for improving the system being tested in an Appendix, which clients are normally very interested in, even for summative testing, so perhaps there should be a specific section in the report itself to record this type of information. Feedback on the CIF from the consumer and supplier organisations A feedback form was distributed to both in2netlogic and Prezzybox. It is interesting to compare the ratings that each of them gave to the report as a whole and to each individual section. Report sectionConsumer rating of usefulness (1= very useful, 7=not useful)Supplier rating of usefulness (1= very useful, 7=not useful)Whole report34Executive summary35Introduction34Method33Performance results (task completion, task time, errors, assist, etc.)54Subjective ratings24User comments, evaluator observations suggestions for change.22 Appendix Evaluation materials57 Interestingly the consumer organisation seemed to place slightly more value on the introductory parts of the report and the subjective ratings than the supplier. The supplier and the consumer were moderately interested in the method used to perform the study and the user comments, evaluator observations and suggestions for change. In terms of the main benefits in receiving the CIF, and taking part on the project, both organisations felt that most useful aspect was an impartial and objective analysis of current operation, and generation of new ideas. Neither group identified any problems with the CIF, although the consumer organisation had hoped for more comments on all features of the Prezzybox site. Although some feedback on most features was provided, these depended on which features were used during the trials. The supplier stated that weighted comments would have be very useful i.e. how important was each suggestion or observation. Both organisations stated that the CIF would definitely lead to changes to the system. Some minor changes were made immediately, but it was also stated that all development is done through a tight specification routine to maximise quality and this takes time. However the improvements are expected to be carried out in the first quarter of 2002. The consumer organisation felt that the CIF improved the knowledge of usability although this was less true of the supplier organisation. As the supplier and consumer organisations already work closely together, it was not expected that the CIF would have a great impact on supplier/consumer communication. Discussion Although the trials elicited many user comments about how to improve Prezzybox, everyone who took part seemed to thoroughly enjoyed the experience of browsing the site and choosing a gift to purchase. Nearly all stated that they would visit the site again. The generation of the PRUE evaluation report provided a useful focus for discussion between RSEHF, the consumer and supplier organisations and laid the path for producing an improved shopping site. The idea of performing a real trial, where users actually purchased an item (as payment for taking part) enabled the group to see how well the site performed in a real situation. This was illustrated by the performance measures although these did not reflect non-use of key features such as the Gift Wizard. The report provided a good format for reporting all features and will serve as a good record for performing the repeat trials. Although suggestions for change can be included in the report in a separate section, perhaps more thought needs to be given as to how summative results can be related more directly to design recommendations. In a developing design situation, this approach could show how design changes lead to improved user performance and satisfaction ratings. Conclusions what other shopping sites can learn The summative results reported in the CIF show the most important and stringent test of an online (web) shopping site i.e. can users make a purchase from it? Without such a test, an online shopping service cannot be sure how effective their site is, and whether they may be losing many sales as a result of poor design. Similarly, without producing a baseline for use of the site, the provider will have no basis for comparison when they produce a revised site. The CIF reports the trials in a clear and comprehensive way, allowing trials to be repeated in a faithful way and thus to produce valid results for comparison. There is scope for the CIF to be improved i.e. by integrating formative data more closely with the summative results, so that the consumer and supplier organisations can determine which changes should be of greatest priority in order to enhance user performance i.e. to achieve a great number of e-shopping sales. Robissa Travel/ExcelSoft: Travel Agent Software Overview In the context of the PRUE Project, SIEM, in cooperation with a software development company and a representative client (a travel agency), planned and carried out a usability test for a software application (Global Travel) that supports the management tasks of a travel agency. All the steps of the evaluation procedure, as well as the final results were recorded and reported using the CIF format. The procedure that was followed included the following steps: Step 1. Definition of the product to be tested Step 2. Definition of the context of use Step 3. Specification of the usability requirements Step 4. Specification of the context of the evaluation Step 5. Design of the evaluation Step 6. Performance of the user tests and collection of data Step 7: Report and analysis of the collected data The benefits for both participating organizations (i.e., the supplier and the consumer) were considerable: The supplier, through a standardised and valid test procedure obtained a concrete and objective measure of the usability of the tested product in order to demonstrate its quality but also spot potential problems. Additionally, the evaluation data (usability problems, new user requirements, ideas, etc.) that were collected will be useful as input for the design of a future version of the product. The consumer was able to judge the usability of the supplier's product, through the evaluation process as well as the extent to which the specific product caters for his/her particular needs. Another expected benefit was that the consumer's opinion and needs, highlighted during the whole process, will be taken into account for the design of the next version of the product. Additionally, the process helped the consumer identify his / her real needs, state them to the developers in an organised and understandable way, and produce a document for common reference. Finally, since, on the one had, the supplier will be able to take into account the consumer's needs by using a well-documented and objective procedure, and, on the other hand, the consumer will have an objective process (the CIF report) for judging the extent to which this was accomplished, it is clear that there is a unique opportunity for the creation of a close and mutually beneficial relationship between them. The result of such a relationship can be a loyal customer for the supplier and software products of higher quality, productivity and usability for the consumer. In conclusion, the study proved that the CIF format can provide real added-value to a software project, because it is a structured and well-defined process. Both the consumer and the supplier considered it as an efficient, effective and worthwhile activity that had positive results for both. The extra resources (time and effort) required are well justified and spent, and both organizations are positive to using again the CIF format in the future. Of course it has to be noted that in order for the whole process to be resource-effective but also valid and productive, an organization that has high expertise in usability engineering is required, since both the wording and the process of the CIF format require a theoretical background, but also a considerable amount of concrete previous experience, in the field. Case study In the context of the PRUE Project, SIEM, in cooperation with a software development company and a representative client (a travel agency), planned and carried out a usability test for a software application (Global Travel) that supports the back office management tasks of a travel agency. All the steps of the evaluation procedure, as well as the final results were recorded and reported using the CIF format. In this page, the procedure that was followed is described. Step 1: Definition of the product to be tested After interviews with potential suppliers (software development companies), the final supplier and product to be tested were selected. As mentioned above, the product was Global Travel, a computer application for carrying out management tasks of a travel agency. The version tested was 2.01. Step 2: Definition of the context of use The next step was to thoroughly study the product to be tested and, in cooperation with the supplier, define the context of use. This part required specifying which would be the intended user groups, the skills, mental or physical capabilities users should have and the user tasks and goals that could be achieved with the product. The primary user group of Global Travel is employees of a travel agency. The application requires that the users have some familiarity with the use of personal computers and the Windows operating system and that they have a basic knowledge of a travel agencys tasks. As already mentioned, the main tasks supported by the tested software application were the management tasks of a travel agency. More specifically, Global Travel is a commercial product, in the Greek language, which supports all different kinds of reservations, such as hotels, tickets, organized tours and transportation. Additionally, it includes several facilities, such as client and supplier profiles, as well as ledger, statistics and other accounting services. The environment in which the users are expected to use the product is a "typical" office environment, e.g., a travel agency. The product's minimum requirements include a personal computer with a Pentium processor, a keyboard and a mouse, running an MS Windows operating system. Step 3: Specification of the usability requirements This part of our study included the definition of relevant usability requirements. These were the following: Effectiveness Whether users can complete their tasks correctly and completely. Efficiency Whether tasks are completed in an acceptable length of time. Satisfaction The users subjective opinion when using the product. Step 4: Specification of the context of the evaluation Having already specified the context of use, it was not very difficult to specify the context of evaluation. The criteria for selecting participants were the following: Familiarity with the use of personal computers and the MS Windows operating system. Basic knowledge of a travel agencys tasks. In terms of frequency of use, this could be described as a minimum of one month's experience in performing such tasks. Knowledge of the Greek language. The travel agency that supplied the representative users was "ROBISSA Travel". This was because many employees of this agency had only recently started using the Global Travel product. Thus, they had the "ideal" exposure time to the product. In this step, the task scenarios that participants would perform were also defined. The most important functions of the software application that should be tested were identified through analysis of the product and interviews with the supplier and the potential users. The task scenarios that were selected include: Reserving an airplane ticket for an existing client. Creating a new client profile and making a hotel reservation for this client. Entering payment information, concerning the previously created client. To ensure that the test conditions would be identical to the "real" context of use the environment selected for carrying out the test was the travel agencys central department. Step 5: Design of the evaluation After interviewing several potential participants eight of them were finally selected, seven employees and a manager. A room at the central department of the travel agency was selected for the evaluation. Furthermore, the computer selected was a Pentium (133MHz, 32 MB RAM), with a standard mouse and keyboard and a 15 color monitor at 800x600 resolution. The evaluation was designed as follows: First the participants would have to use the software to perform the task scenarios. Their actions would be recorded by using the MS Office '97 Camcorder application. As an additional means of recording the test and collecting relevant data a digital camera would be used. A number of metrics were selected for analyzing the data collected from this part of the evaluation: Completion Rate The percentage of participants who completely and correctly achieve each task goal. Mean Goal Achievement The extent to which each task is completely and correctly achieved. Number of errors Tasks with components that required more than one attempt form this metric. Task time The mean task time required to complete each task. Completion Rate Efficiency The quotient of completion rate and mean task time. It specifies the percentage of users who were successful for every unit of time. Goal Achievement Efficiency The quotient of mean goal achievement and task time. It specifies the percentage of each task completely and correctly achieved for every unit of time. For the Mean Goal Achievement metric the best score was defined to be 100%. For each subtask that would not completely or correctly performed, if it was crucial, a 20% would be deducted, else a 10%. The maximum task time was already defined for each task, when the task scenarios were designed. In addition to the above metrics, in order to measure the participants satisfaction with the system the IBM Usability Satisfaction Questionnaires (translated in Greek) would be used. More specifically, after the completion of each task scenario, the user would have to fill in an "After-Scenario Questionnaire (ASQ)", which included questions about the users opinion concerning the ease of task completion, the time to complete the task and the adequacy of support. After completing all the scenarios the user would have to fill in the "Computer System Usability Questionnaire (CSUQ)" to asses the users opinion concerning characteristics of the overall system such as ease of use, simplicity, effectiveness, information and user interface quality. Step 6: Performance of the user tests and collection of data At first the test procedure was introduced and explained to the users, as well as the overall objectives of the evaluation. Furthermore, the users were informed about the fact that recordings would take place and reassured that the test results would be kept confidential. After the initial introduction, the task list was given to the user and the evaluation procedure started. After performing each task scenario the corresponding ASQ questionnaire was filled in and then the user moved on to the next task scenario. Finally, after completing all the task scenarios, the CSUQ questionnaire was completed. Finally, the participant was debriefed and offered an educational software product of SIEM as a reward for his/her participation. During the test an observer was always present, so as to register all information needed for the measurements and help the participant feel more comfortable. The observers role was not to help the participants, since they should only have access to the type of assistance that would be available in real use conditions. Step 7: Report and analysis of the collected data The data collected during the test were analyzed and the results were reported in appropriate format (e.g., tables and graphs). The final test results were the following: the mean completion rate for all tasks was 100%; the mean goal achievement for all participants was 94.5%; the mean number of errors was 0.46; the mean task time was 4.6 minutes; the completion rate efficiency was 21.8%; the goal achievement efficiency was 20.6%. The overall score of the "Computer System Usability Questionnaire (CSUQ)" was 4.35 (on a scale from 1 to 7, where 1 is the best rating) while the average score of the "After-Scenario Questionnaire (ASQ)" was 3.5 (on a scale from 1 to 7, where 1 is the best rating). Benefits for the participants The supplier, through a standardised and valid test procedure obtained a concrete and objective measure of the usability of the tested product in order to demonstrate its quality but also spot potential problems. Additionally, the evaluation data (usability problems, new user requirements, ideas, etc.) that were collected will be useful as input for the design of a future version of the product. The consumer was able to judge the usability of the supplier's product, through the evaluation process as well as the extent to which the specific product caters for his/her particular needs. Another expected benefit was that the consumer's opinion and needs, highlighted during the whole process, will be taken into account for the design of the next version of the product. Additionally, the process helped the consumer identify his / her real needs, state them to the developers in an organized and understandable way, and produce a document for common reference. Finally, since, on the one had, the supplier will be able to take into account the consumer's needs by using a well-documented and objective procedure, and, on the other hand, the consumer will have an objective process (the CIF report) for judging the extent to which this was accomplished, it is clear that there is a unique opportunity for the creation of a close and mutually beneficial relationship between them. The result of such a relationship can be a loyal customer for the supplier and software products of higher quality, productivity and usability for the consumer. Conclusions This study aimed at exploring the usefulness and value of the CIF report format. Real practice showed that it can provide real added-value, because it is a structured and well-defined process. Both the consumer and the supplier considered it as an efficient, effective and worth performing activity that had positive results for both. The extra resources (time and effort) required are well justified and spent, and both organizations are positive to using again the CIF format in the future. Of course it has to be noted that in order for the whole process to be resource-effective but also valid and productive an organization that has high expertise in usability engineering is required, since both the wording and the process of the CIF format require a theoretical background, but also a considerable amount of concrete previous experience, in the field. Usability of the CIF standard PRUE evaluated the usability of the CIF itself. How easy to use and appropriate would the CIF be for use in these trials by the usability specialists from each of the partners? Correct implementation of the CIF template The CIF format instructions were followed correctly by each organisation, except for some minor interpretations of: the Title page (see 11.3.1 below) assists (see 11.3.3 below) The close conformance could be expected of the main partners who are experienced usability professionals. It was reassuring that the report by a consumer organisation with limited usability experience also followed all the rules, as although the evaluation was carried out in conjunction with an experienced partner, the report was drafted without additional assistance. The validity and appropriateness of the measures used Effectiveness measures Two types of effectiveness measures defined in the CIF standard were used in the trials: Task Completion Rate: percentage of tasks completed Mean goal achievement: extent to which the task was correctly and completely achieved (allowing for partial achievement) Task completion rate is a commonly-reported measure. The CIF also suggests use of goal achievement as this more directly assesses the extent to which the task results meet the user or business need. The findings were quite salutatory, with mean goal achievements in the tasks used in the four trials of 55-56%, 52-63%, 80% and 87-100%. Efficiency measures All partners reported task time and effectiveness/task time (but only task time was interpreted, see 11.3.4). Satisfaction measures Three partners used standardised questionnaires (SUMI, WAMMI, CSUQ and ASQ) and one used an in-house questionnaire. All these options are allowed by the CIF. Only SUMI and WAMMI are standardised against a large sample of data. In both cases a score of 50 represents the industry average. the original travel expenses systems was extremely poorly rated with a SUMI score of 20 (in the bottom 1% of systems) while the new system scored 45 on WAMMI (below the industry average of 50) SUMI scores for the legal information system ranged from 41 for external users to 55 for internal users The scores from any of the questionnaires can be used to set benchmarks for comparison with values obtained for revised or alternative products. Other measures Two partners reported the number of assists (see 11.3.3). The other partners did not give assists. Two partners also reported the number of errors. This is information is optional as it is of most interest to the developers. Any differences in how the template has been implemented Title page The CIF standard requires that the following information is included on the Title page: Who prepared the report. Customer Company Name. Customer Company contact person. Contact name(s) for questions and/or clarifications. Supplier phone number. Supplier email address. Supplier mailing or postal address. The list above seems to assume that testing was by the supplier organisation, and it was not exactly followed in all PRUE CIFs. A more logical requirement would be: Consumer Company Name and contact details Supplier Company Name and contact details Who prepared the report. Contact name(s) and details for questions and/or clarifications. Executive summary The CIF standard recommends including a Tabular summary of performance results. One of the reports included a detailed table. A more appropriate level of detail to recommend for the executive summary would be a Tabular summary of the means of the performance results. Assists The CIF standard states If it is necessary to provide participants with assists, efficiency and effectiveness metrics shall be provided for both unassisted and assisted conditions, and the number and type of assists shall be included as part of the test results. A summative evaluation should as closely as possible reproduce the intended context of use, so the only assistance available should be any that would actually be available in real use (typically documentation or a telephone help desk). For summative evaluation it is simplest to terminate a task if the user is unable to continue without assistance (as done on one trial). However it is common practice in usability evaluations to assist a user if they are otherwise unable to continue, in order to get feedback on subsequent parts of the task. If this information is required as part of a summative evaluation trial the assist can be seen as terminating the trial for the purpose of effectiveness and efficiency measures (as done in one trial). Another reason for giving assists is to guide users around bugs in prototype software (as done in one trial). Finally assists may be given to guide users when they have difficulty (as done in two trails). One CIF report suggests that the two assists given had no significant impact on the results, and another made an allowance in the (already very poor) effectiveness scores. This is an area where users of the CIF standard need more specific guidance. Efficiency Efficiency is determined by task time, and measured by the ratio of effectiveness/task time that provides a productivity measure of percentage of goal achievement per minute. This is useful when making comparisons: for example an original system had a mean effectiveness of 55% and task time of 28 mins, while the new system had a mean effectiveness of 56% and task time of 35 mins. Combining goal achievement with time gives efficiency measures of 2.9%/min and 1.9%/min respectively. This is a useful comparison to make when overall productivity is important, but may not always be relevant. For example: is it useful to trade off the time it takes to fill in an expense return against the accuracy? No partners chose to interpret this measure (and three partners incorrectly used the units of % rather than %/min). Some of the group developing the CIF also had difficulty with this measure, which suggests that it needs to be much more clearly explained in any future revision of the CIF. Graphical presentation The CIF standard recommends inclusion of Graphical Presentation of Performance Results. (This is incorrectly shown as a requirement in the checklist in Annex A of the CIF standard.) One CIF report, following the checklist requirement, provided graphics showing the differences between individual users. It would be more appropriate for the CIF to recommend inclusion of Graphical presentation of mean performance results when there is more than one user group or task. Additional information Only one CIF report included user comments and observations. This information is not required by the CIF format, as it is information for designers rather than purchasers. However in some contractual situations the CIF evaluation may serve a dual purpose, with design feedback needed to make improvements if the usability measures are not acceptable. It would be useful for the CIF format to explain how and when this information may be relevant, and where to include it. Satisfaction with the CIF standard Each partner using the CIF was asked to give a personal assessment of the usability of the CIF. In your opinion as a user of the CIF standard, when using the CIF standard to design and document a summative test of usability, was the CIF standard: - effective in guiding the design of an appropriate test which was appropriately documented? very effective 1 2 3 4 5 6 7 very ineffective mean score: 2.3 The CIF does a good job in providing a structure for a summative test. It reminds you to document charactersitics of the test such as how subjects were recruited for repeatability. Need clearer guidance to select correct metrics for the application. It would be very helpful if more examples were provided. It would be useful to have references to on-line resources. A public library of CIFs supported by NIST would be helpful if the standard is to be taken up in future. It would be nice if an overview diagram / time schedule of the whole CIF process was provided. Following the CIF would avoid some of the methodological problems identified by Rolf Molichs comparative usability evaluation work. [8] - efficient in the time required to document this type of test? very efficient 1 2 3 4 5 6 7 very inefficient mean score: 2.7 Producing the first report is time consuming. Once you have understood the format it is easy An enterprising user could create a template for themselves to use in the future to speed up the process more and provide a standard format for a usability group to follow. It is well-structured and straightforward to follow. - how satisfied are you with the CIF standard? very satisfied 1 2 3 4 5 6 7 very dissatisfied mean score: 2.7 We believe it is a very handy tool for usability experts and consultants, and plan to use it again in the future. I quite like the structure, but would like to make it simpler, provide guidance for interpretation of results, and provide an electronic version. It would be nice to be able to include formative data within a recognised section or sections of the report rather than just in an appendix. Trial conclusions Much early usability work used summative methods [14], but was not always supported by other user centred design activities. It therefore gained the reputation for being an expensive way to identify problems when it was too late to fix them! So the emphasis moved to formative evaluation (so-called discount usability methods) that could be used earlier in development [9]. It is essential to introduce usability early in the development preocess, but without subsequent summative testing, it is difficult to judge the effectiveness of the usability work [2]. Summative usability testing using the CIF has advantages to both consumer and supplier organisations during procurement. It is one of the most effective ways to enrich the consumers requirements document with objective user performance and satisfaction goals (based on the existing system/product used). It provides a platform on which to evaluate potential competitive products from a number of supplier organisations during the procurement of a new system/product. By using the CIF test structure, suppliers are able to demonstrate that their product complies to the usability metrics defined in the requirements document. The PRUE trials have demonstrated the potential value of the CIF in a wide range of European environments, and this experience with the CIF contributed to the final version of the text for the CIF standard. The adoption of the format as an international standard should provide a strong case for the wider use of summative testing reported in the Common Industry Format. The PRUE case studies provide strong evidence for the benefits of this approach. The CIF builds on the methods developed by the EU MUSiC project [3,7], and it is appropriate that it should be used in Europe. References Bevan N (1999) Industry standard usability tests. In: Human-Computer Interaction INTERACT 99 (Volume II). S. Brewster, A. Cawsey & G. Cockton (editors). British Computer Society. Bevan N (1999) Quality in use: meeting user needs for quality. Journal of Systems and Software,49(1), 89-96 Bevan N and Macleod M (1994) Usability measurement in context. Behaviour and Information Technology, 13, 132-145 Blanchard H (1998) The application of usability testing results as procurement criteria for software. SIGCHI Bulletin, July 1998. ISO 9241-11 (1998) Ergonomic requirements for office work with visual display terminals (VDT)s - Part 11 Guidance on usability. ISO 13407 (1998) User centred design process for interactive systems. Macleod M, Bowden R, Bevan N and Curson I. (1997) The MUSiC Performance Measurement Method. Behaviour and Information Technology, 16. Molich, R., Kaasgaard, K., Karyukina, B., Schmidt, L., Ede, M., van Oel, W., Arcuri, M. Comparative Evaluation of Usability Tests, CHI99 Extended Abstracts, ACM Press, pp 83-84. Nielsen J (1993) Usability Engineering. Academic Press. Nielsen J (2001) www.useit.com PRUE (2001) www.usability.serco.com/prue Public Accounts Committee (1999) Improving the Delivery of Government IT Projects http://www.publications.parliament.uk/pa/cm199900/cmselect/cmpubacc/65/6502.htm Spool J, Schroeder W (2001) Testing Web Sites: Five Users is Nowhere Near Enough.. Proceedings of ACM CHI 2001. Whiteside J, Bennett J, Holzblatt K (1988) Usability engineering: our experience and evolution. In: Handbook of Human-Computer Interaction, M Helander (ed). Elsevier. PRUE Final Report Version 1.0 Page  PAGE \* MERGEFORMAT 2 Public  SAVEDATE \@ "d-MMM-yy" \* MERGEFORMAT 12-Mar-02 Public %*STUWd79 A  q r | }   2 3 4 5 6 m n jwUjUj}UjU jU CJ OJQJCJCJ5mH j5U5CJ55CJ,OJQJ5CJ$OJQJ 5OJQJOJQJ56CJ(OJQJ5CJ5CJ8%*TUVWdxaaZZ$da$$d]a$  ]dd%dO]9$d$d%d&d'dNOPQ]a$  ]6$$d%d&d'dNOPQ]a$  ]] %*TUVWd7k8 9 f h i    3 r | 7  g  N pE}$wdGwJ N;4fV(a7k8 9 f ] $d]a$  ]$d]a$  ]d]  ]f h i    3 r | 7  g (h .]. & .].  ].].    G H b c d e f    . / I J K L M u v OPjjUj_UjUjeUjUjkUjUjqUjU jUE  N pE}$wdGwJ (jklnopst|}~$%?@ACD\]wxy{| "#VWqjA Uj UjG UjUjMUjUjSUjUPJ jUjYUCqrsuv    CD^_`bc&'ABCEFVWqrsuvj#UjUj)Uj Uj/ Uj Uj5 Uj Uj; U jUj UC)*DEFHIcd~ -.HIJLM5679:opjUjUj UjUjUjUjUjUjUjU jUC N;4fV5egh{sB & F$a$ (./023EF`abde56PQRTU/jjUjUjpUjUjvUjUj|UjUjU jUE/0134DE_`acdefghB` k &('(r(s(t(((//>0?0@0}0~0>>"?#?$?G?H?HH#I$I%I`IaILMMN B*mH phmH  >*B*phjUjUjNU0J#CJjUPJ5hhCJCJmHjCJUmHjdU jUjU<5egh{sB !! ""Z#s##$$$$0%}%%%ʾvl^Te  rk   rWe  e  e  hk   h@e   e k  k  k  e   &  k  !! ""Z#s##$$$$0%}%%%%7&i&& & FCEƀ%%7&i&&j'h)))+9+G+t+++,_,o,,,-1-ĸ~tf\RH>Ie  e  e  e  +k   +$e  Ye  e  e  e  Qk  Qk  k  e  ;e  k   e  &j'h)))+9+G+usqoomF EƀCEƀG+t+++,_,o,,,-1-N-t---Z...1b2 & FKEƀ^`1-N-t---Z...1b22'44445666X77l8|888}znkeb_XND`  `   ` '  .k  .k  k  ;k   k   3e  k   e  e  Yk   Yb22'44445666X77l8|888889i::;<<#=S==== & F8889i::;<<#=S====>I?l?@@XAABBCvlbXNDCe  e  <e  {e  e  vk  v6k   6`  `  ^`  `  k   Dk   D`  `  =>I?l?@@XAABBCCDoDDDD8EtEEEFCEƀCCDoDDDD8EtEEEFFFGG{GhHHbIIK&LLvlbXNKC@=:<k  Xe  (Ee  'ye  &e  %k   e  $e  #e  "*e  !fe  e  e  k   de  e  >e  e  FFFGG{GhHHbIIK&LLLkEdEƀ^CEƀ LLMNNN]P}QRRSUVVVVVVVVWWWWW&W'WY]YYN[[\\?\B\]^(fffGgqgvgggijljlllwwQyz+)lPQ46WOJQJ 5hmH hmH OJQJhmH 5mH hmH PJ@OJQJmH  B*CJphCJ5B*CJOJQJph5CJOJQJ B*phDVVVVVVVWWWWW&Whw$$Ifr6j $44664$a $1$7$8$H$If$1$7$8$H$If^` &W'WY]YYYYYYYYþ{vqlgb^[        %  O  no        ;  TU  f  w             !           (  ./  0"WWWWWWWXXxLdM!d(($1$7$8$H$If P$1$7$8$H$If^`w$$Iflr6j $44664$a (($1$7$8$H$IfXX!X+X5XFXWXXXqXXXtddddXdd $1$7$8$H$If(($1$7$8$H$If$1$7$8$H$If^`w$$Ifr6j $44664$a XY=Y>Y]YYYYYYYYxhxf$1$7$8$H$If^w$$Iffr6j $44664$a (($1$7$8$H$If YZ[N[W[\Q]]^_^^w`ab:cduddNeffahiYii-jjj>kk¿|tld\WTf  w    K    Cyk   yV    N      N    k  k  YZ[N[W[\Q]]^_^^w`ab:cduddNef & F & FH & FdEƀffahiYii-jjj>kkkljllH & FdEƀ & F$a$kkljlllFmmmpp\qqorrYtwt*uu_vvwdwwTxxxxQyyǹ}upkh\WO  hk  h  @  {/k   ;Nsk   s^      :      llFmmmpp\qqorrYtwt*uu_vvwdw & F$a$ & FH & FdEƀdwwTxxxxQyyz{{k|||<}~}~~+@Qh$a$ & Fyz{{k|||<}~}~~+@Qh|)@K̄Ⱥtqc`RFAk  k   k   k   k  k   ]     k   Ek   ^  h|)@K̄քf؅) l܍{F6 $]a$ % & F$dd̄քf؅) l܍{F6Ly6½xurolid]Se    C4k   4ak  k  k     2  f]k   ] 6Ly6sݧp%7έۭ=gm & Fsݧp%7έۭ=gm0ŴӶG˽}umebTQKC    Lk   L  7  z    k   /'k   '1k  k   *mk         m0ŴӶG1ܸ/չWXY]d $$$Ifa$$$Ifo]o & F & FG1ܸ/չWXY]del ̾}xsnje`[VQ                        &  -  =  i  pq  x  |  }~k   z    x      b  WX)+HJce~45@BP./pq3./x!;D]hiGQi~Xm%& " 1   @mH  CJOJQJ>*6CJB*hmH phhmH  CJOJQJOJQJ 5OJQJ55CJOJQJ CJOJQJB*CJOJQJhph5B*CJOJQJhph5B*OJQJph;del`Ff $$$Ifa$g$$If4F%         %%%    a "$&()+.479<BEGHJMRTVY^`b|lFf"Ff $$$Ifa$ "$&()+.479<BEGHJMRTVY^`bcehmoqty{}Ŀ{vqlgb]X  Z  \  a  d  f  h  m  p  rs  u  w  |                                      "bcehmoqty{}~Z459@l $$$Ifa$$a$Ff& $$Ifa$Ff$}~Z459@ABIPQRY\_nowz}ɿʿ׿ڿ}xsnie`[             *+  6  9  <  GH  X  [  ^  fg  v  y  |              {(WX#@ABIPQD|qq| $$$Ifa$$$Ifz$$IfTl0 E%      &&04 la QRY\_nRtG<<4$$If $$$Ifa$ $$$Ifa$$$IfTl\ E)        (&&&&04 la nowz}m|bWWO$$If $$$Ifa$ $$$Ifa$$$IfTl\ E)   04 la mtbWWOm|bWW$$If $$$Ifa$ $$$Ifa$$$IfTl\ E)   04 la ɿʿ׿ڿݿeZOO $$$Ifa$ $$$Ifa$$$IfTl\ E)   04 la $$Ifڿݿ G6!./¿|uqjc\X                  34  ;  ?  @B  L               m|bWWO$$If $$$Ifa$ $$$Ifa$$$IfTl\ E)   04 la  m(gg$If$$IfTl\ E)       04 la G6$$If$a$k$$IfTl0 E%     04 la kccc$$If$$IfTlFQ h %%%0    4 la!./gkpqxxxxxxxxx\$$If~$$IfTlFQ h0    4 la /gkpq" ?!3/}y !Ķxje^TJ      k       [  N     g k   "     $  de  j  n  q" ?nigeeca"$a$~$$IfTlFQ h0    4 la$$If dd$If !3/}y !,]vAi & F & F & F!,]vAi>J>J¼|yvhebZR  w  k   Pk   e+     A +     6 |-8bk  k  k   >J>J93,w & Fh^h h & Fh^h h & F93,wplk{kx,K% " ŷ}zli[XUR7k   k   k   k   xk  xHk   Hk   p  =    p  plk{kx,K% " c   ! " 1 O n   !$dd$If P$$If" c   ! " 1 O n                    :<>?RT½zvqlgc^Y          /0  2  4  ;<  >  @  MN  P  R  de  g  i  vw         !     k   !     mHeee$$If$$IflF H D  P 0    4 laH              :\zzzHzzz0zzz0z$$If|$$IflF H D  P 0    4 laH:<>?RTVWz`zzt$If|$$IflF H D  P 0    4 laH$$If TVWy W R    ¿zwrmhc^YEf)k  k  `k   ¿k   ¿Oabc  e  g            !y |rppppnppp ^`|$$IflF H D  P 0    4 laH$If '*,,,-(.l33444^4h4444"5#55::==>0>u>>>>>?)???zDDH IJKRV W&Y[Y\^+`aaXbbbBcyccdpmmno2rru+vwwy{&h6 CJOJQJOJQJCJ>*mH 5mH 5B*mH phmH mH  CJOJQJQW R    !!["##$CEƀ'  & F,^, !!["##$P%'**,,,-.).u/0U2l3m3344^4i44444#55ĺ|wolda^[V  ]h  '  %YZqDQk   \e  3_e  2e  1e  0e  /je  .' $P%'**,,,-.).u/0U2l3m334$h^ha$$$a$h^h$a$CEƀ44^4i44444#55 66679G999:::<A<$h^ha$ & Fh^h & F^$a$$^a$' $ & F^a$5 66679G999:::<A<==>1>u>>>>?*???c@AzD{DDEGHH II¿}zwtqnkheb_\-KL<c      @Q      561            $A<==>1>u>>>>?*???c@AzD{DDEGL & FhdEƀ^h & Fh^hGHH III#JGJkJJJKKzMOQR\UzU-VYVV^ & FRx^R`^h^hII#JGJkJJJKKzMOQR\UzU-VYVVV WXXX&Y[YY&[|nb_XPM[    k  k   #>   ` /k  /k  k   Rk   1  [          VV WXXX&Y[YY&[:[[gb & FH & FdEƀH & FdEƀ &[:[[[\] ^^^____+`D`[`|````aaaĶ{qg]SND)  )  )  )  )L  )m   )  ) k  k   k       k   Zk   Z[[\] ^^^____+`igH & FdEƀH & FdEƀ +`D`[`|````aaaabXbjb|ccdyggh@iKi2k/mFmm!o8o$dd) & F) & FaabXbjb|ccdyggh@iKi2k/mFmm!o8oq7qq1r2rrrrsstUt~pmjgda^[XUR7~6FzIk   Ink   n0`k   `Ok   Ok   7k   7)  )  )  8oq7qq1r2rrrrsstUtt!uuuu+v;vvJwwwwwxtxy^Utt!uuuu+v;vvJwwwwwxtxyyy{U~&]Ɂ<@~tj`V5  5  5  5[  5  5_  5  5>   5 k  qBk   #bx&fgS!yyy{U~&]Ɂ<@EW!"$ 9r !$5 & F^ $)*@ABCKLMtu~0J%CJOJQJCJOJQJmHjCJOJQJU CJOJQJ6mH mH NHEW !"#$ABuwx{|~&$5  5  5V  5  5  "#$5 & F &$$dNa$$ &1$] & # 0/ =!"#$%}DyK _Toc414417994}DyK _Toc414417995}DyK _Toc414417996}DyK _Toc414417997}DyK _Toc414417998}DyK _Toc414417999}DyK _Toc414418000}DyK _Toc414418001}DyK _Toc414418002}DyK _Toc414418003}DyK _Toc414418004}DyK _Toc414418005}DyK _Toc414418006}DyK _Toc414418007}DyK _Toc414418008}DyK _Toc414418009}DyK _Toc414418010}DyK _Toc414418011}DyK _Toc414418012}DyK _Toc414418013}DyK _Toc414418014}DyK _Toc414418015}DyK _Toc414418016}DyK _Toc414418017}DyK _Toc414418018}DyK _Toc414418019}DyK _Toc414418020}DyK _Toc414418021}DyK _Toc414418022}DyK _Toc414418023}DyK _Toc414418024}DyK _Toc414418025}DyK _Toc414418026}DyK _Toc414418027}DyK _Toc414418028}DyK _Toc414418029}DyK _Toc414418030}DyK _Toc414418031}DyK _Toc414418032}DyK _Toc414418033}DyK _Toc414418034}DyK _Toc414418035}DyK _Toc414418036}DyK _Toc414418037}DyK _Toc414418038}DyK _Toc414418039}DyK _Toc414418040}DyK _Toc414418041}DyK _Toc414418042}DyK _Toc414418043}DyK _Toc414418044}DyK _Toc414418045}DyK _Toc414418046mDyK >http://www.usabilitynet.org/methods/requirements/existing.aspyK |http://www.usabilitynet.org/methods/requirements/existing.aspmDyK >http://www.usabilitynet.org/methods/requirements/existing.aspyK |http://www.usabilitynet.org/methods/requirements/existing.aspDyK $http://www.usabilitynet.org/methodsyK Hhttp://www.usabilitynet.org/methodsDyK $http://www.usabilitynet.org/methodsyK Hhttp://www.usabilitynet.org/methods$$If L &x!%             R S    Z%%%%%%%%%$$$$a$$If L &x!%   R S    Z%$$$$a$$If L &x!%  RS  Z%$$$$a$$If L &x!%  RS  Z%$$$$a$$If L &x!%        R S    Z%$$$$a7 iD@D Normal,paragraph ddCJmH @ Heading 1,H1,number,Annex+$$ & Fkh&d@&P5CJ KH OJQJn@n Heading 2,1.1,H2,heading 2$ & Fk@&5CJOJQJd@d Heading 3,h3,H3'$$ & Fk x@&a$ 5OJQJL@L Heading 4$ & Fk<@&5CJOJQJHH Heading 5,h5 & Fk<@& 56CJFF Heading 6,h6 & Fk<@&5CJDD Heading 7 & Fk(@& 5OJQJ<< Heading 8 & Fk<@&6D D Heading 9 & Fk<@& CJOJQJ<A@< Default Paragraph Font>B@> Body Text,bt $xa$OJQJ66 Paragraphe $a$CJ:: bullets & FZ CJJO"J Body Text Keep$$PPa$OJQJ>O2> Bullet$ & F`(a$OJQJOB bullet & FePEƀ**IFCmH PRP Table large bullet  & FdP CJOJQJ:b: NomAuteurs$xxa$64r4 Adresse$a$ CJOJQJXX Bibliographie'$$ x^`a$CJ Section HeadingX$$ & Fxd$d0%d&dN0OP^x5@CJKHOJQJmH "@ Captiony$  ]dP$d%d&d'dNOPQ]a$5CJ @Q@@ Body Text 3 $xa$ CJOJQJ4>@4 Title$a$5CJOJQJB@BTOC 1 f# 5OJQJmHnHPS@P Body Text Indent 3 ^CJOJQJmH BB Picture $8^8@CJOJQJmH ZRZ Body Text Indent 2 $S^Sa$@CJOJQJmH JOJ TOC Base!d P @CJOJQJmH DP@"D Body Text 2"$a$@CJOJQJmH ,U@1, Hyperlink >*B*CJ4@B4 Header $ 9r OJQJ&)@Q& Page Number4 @b4 Footer & !OJQJZC@rZ Body Text Indent'$^a$CJOJQJmH :@:TOC 2( p# ^mHnH@O@ alpha list ) & Fx CJOJQJH0H List Bullet*$ & Fa$OJQJ&& TOC 3 +^&& TOC 4 ,^&& TOC 5 -^&& TOC 6 .^&& TOC 7 /^&& TOC 8 0^&& TOC 9 1^.". Footnote Text28&@18 Footnote ReferenceH*XTBX Block Text"4 %Sh]^SOJQJhmH RORR References 5d$^` CJOJQJT^bT Normal (Web)6$P[$\$a$CJOJPJQJmH `/!z!z!z!z!z!z!z!z! z! z! z! z! z!z!z!z!z!z!z!z!z!z!z!z!z!z!z!z!z!z!z! z!!z!"z!#z!$z!%z!&z!'z!(z!)z!*z!+z!,z!-z!.z!/zgsG hh#).6t?bCLU^gq+{l;+ > !,4>3I\OWA_jqsz 0".r $  1 V/__d*O !0"#w$f%&~'(h)8*+,- .!# jq/NW -f &G+b2=FLV&WWXXYfldwh6mdb@Qn q  :$4A<GV[+`8oy"     "#%'(*,/%1-8CL|WYkȳG }ڿ/!" T 5I&[aUt !$&)+.|35m Gce.JLuOkn} $ @ C \ x {   " V r u C _ b  & B E V r u ) E H c  -IL69o/2Ead5QT03D`ce&"s"")?*}*8#9G9B$C`C %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%XXXX(?AKs}!?b$hj:bk>,r`qb$ ΋o}b$槐Li(v@0( ' B S  ?I _Toc441052920 _Toc414417994 _Toc406035765 _Toc414417995 _Toc414417996 _Toc414417997 _Toc414417998 _Toc414417999 _Toc414418000 _Toc414418001 _Toc414418002 _Toc414418003 _Hlt504571271 _Toc414418004 _Toc414418005 _Toc414418006 _Toc414418007 _Toc414418008 _Toc414418009 _Toc414418010 _Toc414418011 _Toc457293996 _Toc465752691 _Toc465841295 _Toc457294002 _Toc465752696 _Toc465841300 _Toc414418012 _Toc414418013 _Toc414418014 _Toc414418015 _Toc406035766 _Toc414418016 _Toc414418017 _Toc414418018 _Toc414418019 _Toc414418020 _Toc526692663 _Toc414418021 _Toc414418022 _Toc414418023 _Toc414418024 _Toc406035767 _Toc414418025 _Toc414418026 _Toc414418027 _Toc414418028 _Toc414418029 _Toc414418030 _Toc414418031 _Toc414418032 _Toc414418033 _Toc414418034 _Toc414418035 _Toc414418036 _Toc414418037 _Toc414418038 _Toc414418039 _Toc414418040 _Toc406384346 _Toc414418041 _Toc406384347 _Toc414418042 _Toc457294007 _Toc465752701 _Toc465841305 _Toc406384348 _Toc414418043 _Toc406384349 _Toc414418044 _Toc414418045 _Toc414418046 _Hlt409247162fh7 h##(b,.. 1I9bCLUNU`gYnrrruuux+{)~܇yݡΧճ !,"$EK\O-P-PRR&S&S&SYYkksz'  !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGH@gy h ##(,.. 1k9CLMUMUVU`gvnrrruuux?{?~ۇۇ$ڧ>.+I2Jb$EKyOWPWPRR:S:S:SYY5k5ksz)7>?FHRTV`hkst}7PYv//@@AACCDD^MbMNNKPRPZPcPPPPPQQQQ_QhQ] ]N_W_abjbbbbbdd+e4eppttuu@{I{v{{{{||||Z}f}h}p}}}@~I~~~~~u~4=GO.8fpbkǐӐ9CΚؚښCLǝѝ T^ʹֹݹz}?H/9 %,5;D#,LU#(GP)2%`iw  V_$FF~[[^^}__``vdzdddklmmAmPmjnnnwoossduluuuuuyyMzRzzz${*{0{7{]{b{{{{{}}}}}}}}~~~$~*~3~F~I~S~V~\~b~(Dox-M:o3Ee5U4Ddt""@*}*$9G9 "$)CL Nigel BevanOMacintosh HD:Nigel:Nigel 1:European Commission:PRUE:PRUE Final Report v0.4. doc Nigel BevanOMacintosh HD:Nigel:Nigel 1:European Commission:PRUE:PRUE Final Report v0.4. doc Nigel BevanOMacintosh HD:Nigel:Nigel 1:European Commission:PRUE:PRUE Final Report v0.4. doc Nigel BevanOMacintosh HD:Nigel:Nigel 1:European Commission:PRUE:PRUE Final Report v0.4. doc Nigel BevanOMacintosh HD:Nigel:Nigel 1:European Commission:PRUE:PRUE Final Report v0.4. doc Nigel BevanOMacintosh HD:Nigel:Nigel 1:European Commission:PRUE:PRUE Final Report v0.9. doc Nigel BevanOMacintosh HD:Nigel:Nigel 1:European Commission:PRUE:PRUE Final Report v0.9. doc Nigel BevanOMacintosh HD:Nigel:Nigel 1:European Commission:PRUE:PRUE Final Report v0.9. doc Nigel BevanOMacintosh HD:Nigel:Nigel 1:European Commission:PRUE:PRUE Final Report v0.9. doc Nigel BevanOMacintosh HD:Nigel:Nigel 1:European Commission:PRUE:PRUE Final Report v0.9. doc|}(p~!XDjjDʜRsvi`M#        OΊb"d Cx6B bC #  :2–vrDN81V  [8f(-?!Q  z* S |0ΥmDi NC {JM  P  __ .l   =i ܺp_D  u| Ps1gVR. =ZP{g–ey_   E v0P|QNm=Lz3x `p-B: h/|ȴ ! bMuΥmD0dzRj28PN>6S14NR\SiLzxfD)x1*V?  ZC 35E! h:P!  \!қ(z#c#B`)T&%f.%6"& n& a&s ^L& N!'BqW{'Z^ q' u:(->N ("\(2(\s) q')@ U00*Υm W*'ڰ5ua* 6+B7+]JfZ+()&}+ ,+,tY,,wY, /-(r- J-Υm/z. 7z.Kg)/xpL/ ZU0Nt*!0 }0 a0 -WE1 (572F JY2~ )5 :D5@*#6nE168 q>:84]B5J8S8r 8{`8Ept8}ou9V&rm:Ĺ _n:^g(:F);BDkp@QJA(9/CxTlhD RmDz@E cMEjV}pFΥmFΥmzFk.IdT.yIΥmL:JBf"aJ J 3J 6cKJ=L g2 MR9kML^X-Nt?NWNΥm]YXNƖh;&hN6d[OĽJ-QO > Pte,P HQf};R *XS>VDpS nS }T "4T d9VT|ؽXTl/ 2r;U Uxq@%BMVoWvXfiX>*)Z C8[PB,\ @@\ r_\ n!\Υms]h8#!]l/ 0]<& |_ |C_F,rbf +Ec Akc  d rdDBKldQr??De2yezL peܣP-f fQN&gLl1si2% 0iMj>uLkt-J k yLl ~l&|3m:-,cmy|mHGmȶ7n =vn<Vn o ]bo dq=9q =q NqtZ3xrдr t *1uL0u(`Ou-lu^|ew |2x  y yΥmNDzF+z y*z,S({ 4Q{Υm9P)}Nk|}\(d~woT1([^`.^`.^`.^`. ^`OJQJo( ^`OJQJo( ^`OJQJo( ^`OJQJo(hh^h`. hh^h`OJQJo(@7^`.@7^`.@7^`..@...@ ....@ .....@ ......@ .......@ ........* hh^h`OJQJo( hh^h`OJQJo( hh^h`OJQJo( hh^h`OJQJo( hh^h`OJQJo( hh^h`OJQJo( hh^h`OJQJo( hh^h`OJQJo(*hh^h`.h ^`OJQJo(h ^`OJQJo(oh pp^p`OJQJo(h @ @ ^@ `OJQJo(h ^`OJQJo(oh ^`OJQJo(h ^`OJQJo(h ^`OJQJo(oh PP^P`OJQJo(hh^h`o(. hh^h`OJQJo( hh^h`OJQJo(h ^`OJQJo(h ^`OJQJo(oh pp^p`OJQJo(h @ @ ^@ `OJQJo(h ^`OJQJo(oh ^`OJQJo(h ^`OJQJo(h ^`OJQJo(oh PP^P`OJQJo(h ^`OJQJo(h pp^p`OJQJo(oh @ @ ^@ `OJQJo(h ^`OJQJo(h ^`OJQJo(oh ^`OJQJo(h ^`OJQJo(h PP^P`OJQJo(oh   ^ `OJQJo( hh^h`OJQJo( 7e^`7OJQJo(%h ^`OJQJo(h ^`OJQJo(oh pp^p`OJQJo(h @ @ ^@ `OJQJo(h ^`OJQJo(oh ^`OJQJo(h ^`OJQJo(h ^`OJQJo(oh PP^P`OJQJo(h ^`OJQJo(h ^`OJQJo(oh pp^p`OJQJo(h @ @ ^@ `OJQJo(h ^`OJQJo(oh ^`OJQJo(h ^`OJQJo(h ^`OJQJo(oh PP^P`OJQJo( hh^h`OJQJo(hh^h`. hh^h`OJQJo(@h ^`OJQJo(h ^`OJQJo(oh pp^p`OJQJo(h @ @ ^@ `OJQJo(h ^`OJQJo(oh ^`OJQJo(h ^`OJQJo(h ^`OJQJo(oh PP^P`OJQJo(^`QJo(o ^`OJQJo(o pp^p`OJQJo( @ @ ^@ `OJQJo( ^`OJQJo(o ^`OJQJo( ^`OJQJo( ^`OJQJo(o PP^P`OJQJo(h ^`OJQJo(h ^`OJQJo(oh pp^p`OJQJo(h @ @ ^@ `OJQJo(h ^`OJQJo(oh ^`OJQJo(h ^`OJQJo(h ^`OJQJo(oh PP^P`OJQJo(h pp^p`OJQJo(h @ @ ^@ `OJQJo(oh ^`OJQJo(h ^`OJQJo(h ^`OJQJo(oh ^`OJQJo(h PP^P`OJQJo(h   ^ `OJQJo(oh ^`OJQJo( hh^h`OJQJo(808^8`0o(()^`.pLp^p`L.@ @ ^@ `.^`.L^`L.^`.^`.PLP^P`L. hh^h`OJQJo(h 88^8`OJQJo(h ^`OJQJo(oh   ^ `OJQJo(h   ^ `OJQJo(h xx^x`OJQJo(oh HH^H`OJQJo(h ^`OJQJo(h ^`OJQJo(oh ^`OJQJo( hh^h`OJQJo(h ^`OJQJo(oh pp^p`OJQJo(h @ @ ^@ `OJQJo(h ^`OJQJo(h ^`OJQJo(oh ^`OJQJo(h ^`OJQJo(h PP^P`OJQJo(oh   ^ `OJQJo(h ^`OJQJo(h ^`OJQJo(oh pp^p`OJQJo(h @ @ ^@ `OJQJo(h ^`OJQJo(oh ^`OJQJo(h ^`OJQJo(h ^`OJQJo(oh PP^P`OJQJo( hh^h`OJQJo(h ^`OJQJo(h ^`OJQJo(oh pp^p`OJQJo(h @ @ ^@ `OJQJo(h ^`OJQJo(oh ^`OJQJo(h ^`OJQJo(h ^`OJQJo(oh PP^P`OJQJo(h ^`OJQJo(h ^`OJQJo(oh pp^p`OJQJo(h @ @ ^@ `OJQJo(h ^`OJQJo(oh ^`OJQJo(h ^`OJQJo(h ^`OJQJo(oh PP^P`OJQJo(hh^h`. hh^h`OJQJo(@h^`56CJOJQJo()h ^`OJQJo(h pp^p`OJQJo(h @ @ ^@ `OJQJo(h ^`OJQJo(h ^`OJQJo(oh ^`OJQJo(h ^`OJQJo(h PP^P`OJQJo(oh   ^ `OJQJo(808^8`0o(.^`.pLp^p`L.@ @ ^@ `.^`.L^`L.^`.^`.PLP^P`L. ^`OJQJo(- --^-`OJQJo(h ^`OJQJo(h ^`OJQJo(oh pp^p`OJQJo(h @ @ ^@ `OJQJo(h ^`OJQJo(oh ^`OJQJo(h ^`OJQJo(h ^`OJQJo(oh PP^P`OJQJo(h ^`OJQJo(h pp^p`OJQJo(oh @ @ ^@ `OJQJo(h ^`OJQJo(h ^`OJQJo(oh ^`OJQJo(h ^`OJQJo(h PP^P`OJQJo(oh   ^ `OJQJo(hh^h`o(.h ^`OJQJo(h ^`OJQJo(oh pp^p`OJQJo(h @ @ ^@ `OJQJo(h ^`OJQJo(oh ^`OJQJo(h ^`OJQJo(h ^`OJQJo(oh PP^P`OJQJo(hh^h`.P^`P..^`...xp^`x....  ^` .....  X ^ `X ......  ^ `....... 8^`8........ `^``.........h ^`OJQJo(h ^`OJQJo(oh pp^p`OJQJo(h @ @ ^@ `OJQJo(h ^`OJQJo(oh ^`OJQJo(h ^`OJQJo(h ^`OJQJo(oh PP^P`OJQJo(@hh^h`o(h ^`OJQJo(h ^`OJQJo(oh pp^p`OJQJo(h @ @ ^@ `OJQJo(h ^`OJQJo(oh ^`OJQJo(h ^`OJQJo(h ^`OJQJo(oh PP^P`OJQJo(h ^`OJQJo(oh pp^p`OJQJo(h @ @ ^@ `OJQJo(h ^`OJQJo(h ^`OJQJo(oh ^`OJQJo(h ^`OJQJo(h PP^P`OJQJo(oh   ^ `OJQJo(h ^`OJQJo(h ^`OJQJo(oh pp^p`OJQJo(h @ @ ^@ `OJQJo(h ^`OJQJo(oh ^`OJQJo(h ^`OJQJo(h ^`OJQJo(oh PP^P`OJQJo( s^`sOJQJo(^`CJOJQJo(^`56.pp^p`CJOJQJo(@ @ ^@ `CJOJQJo(^`CJOJQJo(^`CJOJQJo(^`CJOJQJo(^`CJOJQJo(PP^P`CJOJQJo( --^-`OJQJo(h^`OJPJQJo(-h ^`OJQJo(oh pp^p`OJQJo(h @ @ ^@ `OJQJo(h ^`OJQJo(oh ^`OJQJo(h ^`OJQJo(h ^`OJQJo(oh PP^P`OJQJo(h ^`OJQJo(h ^`OJQJo(oh pp^p`OJQJo(h @ @ ^@ `OJQJo(h ^`OJQJo(oh ^`OJQJo(h ^`OJQJo(h ^`OJQJo(oh PP^P`OJQJo( ^`OJQJo(%hh^h`o(.@hh^h`.h ^`OJQJo(h ^`OJQJo(oh pp^p`OJQJo(h @ @ ^@ `OJQJo(h ^`OJQJo(oh ^`OJQJo(h ^`OJQJo(h ^`OJQJo(oh PP^P`OJQJo( hh^h`OJQJo( hh^h`OJQJo(h ^`OJQJo(h ^`OJQJo(oh pp^p`OJQJo(h @ @ ^@ `OJQJo(h ^`OJQJo(oh ^`OJQJo(h ^`OJQJo(h ^`OJQJo(oh PP^P`OJQJo(h ^`OJQJo(h ^`OJQJo(oh pp^p`OJQJo(h @ @ ^@ `OJQJo(h ^`OJQJo(oh ^`OJQJo(h ^`OJQJo(h ^`OJQJo(oh PP^P`OJQJo( s^`sOJQJo(%h ^`OJQJo(oh pp^p`OJQJo(oh @ @ ^@ `OJQJo(h ^`OJQJo(h ^`OJQJo(oh ^`OJQJo(h ^`OJQJo(h PP^P`OJQJo(oh   ^ `OJQJo(h ^`OJQJo(h ^`OJQJo(oh pp^p`OJQJo(h @ @ ^@ `OJQJo(h ^`OJQJo(oh ^`OJQJo(h ^`OJQJo(h ^`OJQJo(oh PP^P`OJQJo(hh^h`.hh^h`.h hh^h`OJQJo(h 88^8`OJQJo(oh ^`OJQJo(h   ^ `OJQJo(h   ^ `OJQJo(oh xx^x`OJQJo(h HH^H`OJQJo(h ^`OJQJo(oh ^`OJQJo( hh^h`OJQJo(h^`CJOJQJo(nh ^`OJQJo(oh ^`OJQJo(h ^`OJQJo(h   ^ `OJQJo(oh \ \ ^\ `OJQJo(h ,,^,`OJQJo(h ^`OJQJo(oh ^`OJQJo(h ^`OJQJo(h ^`OJQJo(oh pp^p`OJQJo(h @ @ ^@ `OJQJo(h ^`OJQJo(oh ^`OJQJo(h ^`OJQJo(h ^`OJQJo(oh PP^P`OJQJo( hh^h`OJQJo(  ^`OJQJo(-^`.^`.^`.hh^h`.88^8`.^`.  ^ `.  ^ `.xx^x`.h ^`OJQJo(h ^`OJQJo(oh pp^p`OJQJo(h @ @ ^@ `OJQJo(h ^`OJQJo(oh ^`OJQJo(h ^`OJQJo(h ^`OJQJo(oh PP^P`OJQJo( hh^h`OJQJo( hh^h`OJQJo(-@h ^`OJQJo(h ^`OJQJo(oh pp^p`OJQJo(h @ @ ^@ `OJQJo(h ^`OJQJo(oh ^`OJQJo(h ^`OJQJo(h ^`OJQJo(oh PP^P`OJQJo( hh^h`OJQJo(h ^`OJQJo(h ^`OJQJo(oh pp^p`OJQJo(h @ @ ^@ `OJQJo(h ^`OJQJo(oh ^`OJQJo(h ^`OJQJo(h ^`OJQJo(oh PP^P`OJQJo(h ^`OJQJo(h   ^ `OJQJo(oh   ^ `OJQJo(h xx^x`OJQJo(h HH^H`OJQJo(oh ^`OJQJo(h ^`OJQJo(h ^`OJQJo(oh ^`OJQJo(h^`)h^`.hpLp^p`L.h@ @ ^@ `.h^`.hL^`L.h^`.h^`.hPLP^P`L. hh^h`OJQJo(h hh^h`OJQJo(h 88^8`OJQJo(oh ^`OJQJo(h   ^ `OJQJo(h   ^ `OJQJo(oh xx^x`OJQJo(h HH^H`OJQJo(h ^`OJQJo(oh ^`OJQJo(h pp^p`OJQJo(h @ @ ^@ `OJQJo(oh ^`OJQJo(h ^`OJQJo(h ^`OJQJo(oh ^`OJQJo(h PP^P`OJQJo(h   ^ `OJQJo(oh ^`OJQJo( hh^h`OJQJo( s^`sOJQJo( hh^h`OJQJo(@hh^h`o(.h ^`OJQJo(h pp^p`OJQJo(oh @ @ ^@ `OJQJo(h ^`OJQJo(h ^`OJQJo(oh ^`OJQJo(h ^`OJQJo(h PP^P`OJQJo(oh   ^ `OJQJo(h^`B*OJQJo(phh ^`OJQJo(oh pp^p`OJQJo(h @ @ ^@ `OJQJo(h ^`OJQJo(oh ^`OJQJo(h ^`OJQJo(h ^`OJQJo(oh PP^P`OJQJo( hh^h`OJQJo(h ^`OJQJo(h ^`OJQJo(oh pp^p`OJQJo(h @ @ ^@ `OJQJo(h ^`OJQJo(oh ^`OJQJo(h ^`OJQJo(h ^`OJQJo(oh PP^P`OJQJo( hh^h`OJQJo( hh^h`OJQJo(hh^h`o(. hh^h`OJQJo(808^8`0o(.^`.pLp^p`L.@ @ ^@ `.^`.L^`L.^`.^`.PLP^P`L.h ^`OJQJo(h ^`OJQJo(oh pp^p`OJQJo(h @ @ ^@ `OJQJo(h ^`OJQJo(oh ^`OJQJo(h ^`OJQJo(h ^`OJQJo(oh PP^P`OJQJo(hh^h`.h pp^p`OJQJo(h @ @ ^@ `OJQJo(oh ^`OJQJo(h ^`OJQJo(h ^`OJQJo(oh ^`OJQJo(h PP^P`OJQJo(h   ^ `OJQJo(oh ^`OJQJo(hh^h`5o()^`CJOJQJo(^`CJOJQJo(opp^p`CJOJQJo(@ @ ^@ `CJOJQJo(^`CJOJQJo(^`CJOJQJo(^`CJOJQJo(^`CJOJQJo(PP^P`CJOJQJo(@hh^h`) s^`sOJQJo(h^`o(.^`o(.^`o(..^`o( ^`o( .... ^`o( ..... ^`o( ...... ^`o(....... ^`o(........h ^`OJQJo(h ^`OJQJo(oh pp^p`OJQJo(h @ @ ^@ `OJQJo(h ^`OJQJo(oh ^`OJQJo(h ^`OJQJo(h ^`OJQJo(oh PP^P`OJQJo( ^`OJQJo(%h ^`OJQJo(h pp^p`OJQJo(oh @ @ ^@ `OJQJo(h ^`OJQJo(h ^`OJQJo(oh ^`OJQJo(h ^`OJQJo(h PP^P`OJQJo(oh   ^ `OJQJo(^`o(.^`o(.^`o(..^`o(... ^`o( .... ^`o( ..... ^`o( ...... ^`o(....... ^`o(........g^`gCJOJQJo(^`CJOJQJo(opp^p`CJOJQJo(@ @ ^@ `CJOJQJo(^`CJOJQJo(^`CJOJQJo(^`CJOJQJo(^`CJOJQJo(PP^P`CJOJQJo(h ^`OJQJo(h ^`OJQJo(oh pp^p`OJQJo(h @ @ ^@ `OJQJo(h ^`OJQJo(oh ^`OJQJo(h ^`OJQJo(h ^`OJQJo(oh PP^P`OJQJo(h pp^p`OJQJo(h @ @ ^@ `OJQJo(oh ^`OJQJo(h ^`OJQJo(h ^`OJQJo(oh ^`OJQJo(h PP^P`OJQJo(h   ^ `OJQJo(oh ^`OJQJo(h ^`OJQJo(oh pp^p`OJQJo(oh @ @ ^@ `OJQJo(h ^`OJQJo(h ^`OJQJo(oh ^`OJQJo(h ^`OJQJo(h PP^P`OJQJo(oh   ^ `OJQJo(h pp^p`OJQJo(h @ @ ^@ `OJQJo(oh ^`OJQJo(h ^`OJQJo(h ^`OJQJo(oh ^`OJQJo(h PP^P`OJQJo(h   ^ `OJQJo(oh ^`OJQJo( h^h`OJQJo(hhh^h`.h88^8`.hL^`L.h  ^ `.h  ^ `.hxLx^x`L.hHH^H`.h^`.hL^`L. hh^h`OJQJo(*h ^`OJQJo(h ^`OJQJo(oh pp^p`OJQJo(h @ @ ^@ `OJQJo(h ^`OJQJo(oh ^`OJQJo(h ^`OJQJo(h ^`OJQJo(oh PP^P`OJQJo(h ^`OJQJo(h ^`OJQJo(oh pp^p`OJQJo(h @ @ ^@ `OJQJo(h ^`OJQJo(oh ^`OJQJo(h ^`OJQJo(h ^`OJQJo(oh PP^P`OJQJo(808^8`0o(.^`.pLp^p`L.@ @ ^@ `.^`.L^`L.^`.^`.PLP^P`L. hh^h`OJQJo(h ^`OJQJo(h ^`OJQJo(oh pp^p`OJQJo(h @ @ ^@ `OJQJo(h ^`OJQJo(oh ^`OJQJo(h ^`OJQJo(h ^`OJQJo(oh PP^P`OJQJo( hh^h`OJQJo(h ^`OJQJo(h ^`OJQJo(oh pp^p`OJQJo(h @ @ ^@ `OJQJo(h ^`OJQJo(oh ^`OJQJo(h ^`OJQJo(h ^`OJQJo(oh PP^P`OJQJo(@@h ^`OJQJo(h ^`OJQJo(oh pp^p`OJQJo(h @ @ ^@ `OJQJo(h ^`OJQJo(oh ^`OJQJo(h ^`OJQJo(h ^`OJQJo(oh PP^P`OJQJo(@h^`56CJOJQJo()h ^`OJQJo(h ^`OJQJo(oh pp^p`OJQJo(h @ @ ^@ `OJQJo(h ^`OJQJo(oh ^`OJQJo(h ^`OJQJo(h ^`OJQJo(oh PP^P`OJQJo( s^`sOJQJo(h ^`OJQJo(h ^`OJQJo(oh pp^p`OJQJo(h @ @ ^@ `OJQJo(h ^`OJQJo(oh ^`OJQJo(h ^`OJQJo(h ^`OJQJo(oh PP^P`OJQJo(hh^h`o(.hh^h`.P^`P..^`...xp^`x....  ^` .....  X ^ `X ......  ^ `....... 8^`8........ `^``.........h ^`OJQJo(h ^`OJQJo(oh pp^p`OJQJo(h @ @ ^@ `OJQJo(h ^`OJQJo(oh ^`OJQJo(h ^`OJQJo(h ^`OJQJo(oh PP^P`OJQJo(@h ^`OJQJo(h ^`OJQJo(oh pp^p`OJQJo(h @ @ ^@ `OJQJo(h ^`OJQJo(oh ^`OJQJo(h ^`OJQJo(h ^`OJQJo(oh PP^P`OJQJo(hh^h`. hh^h`OJQJo(^`o(.^`o(.7^`7o(..S^`So(... ^`o( .... ^`o( ..... ^`o( ...... ^`o(....... ^`o(........^`CJOJQJo(nh ^`OJQJo(h ^`OJQJo(oh pp^p`OJQJo(h @ @ ^@ `OJQJo(h ^`OJQJo(oh ^`OJQJo(h ^`OJQJo(h ^`OJQJo(oh PP^P`OJQJo(h ^`OJQJo(h ^`OJQJo(oh pp^p`OJQJo(h @ @ ^@ `OJQJo(h ^`OJQJo(oh ^`OJQJo(h ^`OJQJo(h ^`OJQJo(oh PP^P`OJQJo(^`o(.^`o(.7^`7o(..S^`So(... ^`o( .... ^`o( ..... ^`o( ...... ^`o(....... ^`o(........h LL^L`OJQJo(h ^`OJQJo(oh ^`OJQJo(h ^`OJQJo(h   ^ `OJQJo(oh \ \ ^\ `OJQJo(h ,,^,`OJQJo(h ^`OJQJo(oh ^`OJQJo(h hh^h`OJQJo(h 88^8`OJQJo(oh ^`OJQJo(h   ^ `OJQJo(h   ^ `OJQJo(oh xx^x`OJQJo(h HH^H`OJQJo(h ^`OJQJo(oh ^`OJQJo(@pp^p`CJOJQJo(@ @ ^@ `CJOJQJo(o^`CJOJQJo(^`CJOJQJo(^`CJOJQJo(^`CJOJQJo(PP^P`CJOJQJo(  ^ `CJOJQJo(^`CJOJQJo(h ^`OJQJo(h ^`OJQJo(oh pp^p`OJQJo(h @ @ ^@ `OJQJo(h ^`OJQJo(oh ^`OJQJo(h ^`OJQJo(h ^`OJQJo(oh PP^P`OJQJo(h^`CJOJQJo(nh ^`OJQJo(oh ^`OJQJo(h ^`OJQJo(h   ^ `OJQJo(oh \ \ ^\ `OJQJo(h ,,^,`OJQJo(h ^`OJQJo(oh ^`OJQJo( hh^h`OJQJo(h ^`OJQJo(h ^`OJQJo(oh pp^p`OJQJo(h @ @ ^@ `OJQJo(h ^`OJQJo(oh ^`OJQJo(h ^`OJQJo(h ^`OJQJo(oh PP^P`OJQJo( hh^h`OJQJo(h ^`OJQJo(h ^`OJQJo(oh pp^p`OJQJo(h @ @ ^@ `OJQJo(h ^`OJQJo(oh ^`OJQJo(h ^`OJQJo(h ^`OJQJo(oh PP^P`OJQJo(hh^h`.[ ^`OJQJo(- hh^h`OJQJo(hh^h`o(. hh^h`OJQJo( hh^h`OJQJo(h ^`OJQJo(h ^`OJQJo(oh pp^p`OJQJo(h @ @ ^@ `OJQJo(h ^`OJQJo(oh ^`OJQJo(h ^`OJQJo(h ^`OJQJo(oh PP^P`OJQJo(h ^`OJQJo(h ^`OJQJo(oh pp^p`OJQJo(h @ @ ^@ `OJQJo(h ^`OJQJo(oh ^`OJQJo(h ^`OJQJo(h ^`OJQJo(oh PP^P`OJQJo( hh^h`OJQJo(@h^`56CJOJQJo()@h ^`OJQJo( s^`sOJQJo(h^`B*OJQJo(phh ^`OJQJo(oh pp^p`OJQJo(h @ @ ^@ `OJQJo(h ^`OJQJo(oh ^`OJQJo(h ^`OJQJo(h ^`OJQJo(oh PP^P`OJQJo(h^`B*OJQJo(phh ^`OJQJo(oh pp^p`OJQJo(h @ @ ^@ `OJQJo(h ^`OJQJo(oh ^`OJQJo(h ^`OJQJo(h ^`OJQJo(oh PP^P`OJQJo( hh^h`OJQJo(h ^`OJQJo(h pp^p`OJQJo(h @ @ ^@ `OJQJo(h ^`OJQJo(h ^`OJQJo(oh ^`OJQJo(h ^`OJQJo(h PP^P`OJQJo(oh   ^ `OJQJo( hh^h`OJQJo( hh^h`OJQJo(hh^h`.@h ^`OJQJo(h ^`OJQJo(oh pp^p`OJQJo(h @ @ ^@ `OJQJo(h ^`OJQJo(oh ^`OJQJo(h ^`OJQJo(h ^`OJQJo(oh PP^P`OJQJo(h ^`OJQJo(h ^`OJQJo(oh pp^p`OJQJo(h @ @ ^@ `OJQJo(h ^`OJQJo(oh ^`OJQJo(h ^`OJQJo(h ^`OJQJo(oh PP^P`OJQJo(^`o(.^`o(.^`o(..^`o(... ^`o( .... ^`o( .....?^`o( ^`o(....... ^`o(........^`o(.^`CJOJQJo(^`CJOJQJo(opp^p`CJOJQJo(@ @ ^@ `CJOJQJo(^`CJOJQJo(^`CJOJQJo(^`CJOJQJo(^`CJOJQJo(PP^P`CJOJQJo(P^`Po(@@^@`o(.^`o(..``^``o(... ^`o( .... ^`o( ..... ^`o( ...... `^``o(....... 00^0`o(........ hh^h`OJQJo( hh^h`OJQJo( hh^h`OJQJo(*h ^`OJQJo(h ^`OJQJo(oh pp^p`OJQJo(h @ @ ^@ `OJQJo(h ^`OJQJo(oh ^`OJQJo(h ^`OJQJo(h ^`OJQJo(oh PP^P`OJQJo(h hh^h`OJQJo(h 88^8`OJQJo(oh ^`OJQJo(h   ^ `OJQJo(h   ^ `OJQJo(oh xx^x`OJQJo(h HH^H`OJQJo(h ^`OJQJo(oh ^`OJQJo(h ^`OJQJo(h ^`OJQJo(oh pp^p`OJQJo(h @ @ ^@ `OJQJo(h ^`OJQJo(oh ^`OJQJo(h ^`OJQJo(h ^`OJQJo(oh PP^P`OJQJo(P^`Po(@@^@`o(.0^`0o(..``^``o(... ^`o( .... ^`o( ..... ^`o( ...... `^``o(....... 00^0`o(........ hh^h`OJQJo( ^`OJQJo(-h ^`OJQJo(h ^`OJQJo(oh pp^p`OJQJo(h @ @ ^@ `OJQJo(h ^`OJQJo(oh ^`OJQJo(h ^`OJQJo(h ^`OJQJo(oh PP^P`OJQJo(h ^`OJQJo(h ^`OJQJo(oh pp^p`OJQJo(h @ @ ^@ `OJQJo(h ^`OJQJo(oh ^`OJQJo(h ^`OJQJo(h ^`OJQJo(oh PP^P`OJQJo( s^`sOJQJo(h ^`OJQJo(h ^`OJQJo(oh pp^p`OJQJo(h @ @ ^@ `OJQJo(h ^`OJQJo(oh ^`OJQJo(h ^`OJQJo(h ^`OJQJo(oh PP^P`OJQJo(^`CJOJQJo(^`CJOJQJo(opp^p`CJOJQJo(@ @ ^@ `CJOJQJo(^`CJOJQJo(^`CJOJQJo(^`CJOJQJo(^`CJOJQJo(PP^P`CJOJQJo( hh^h`OJQJo( hh^h`OJQJo(h ^`OJQJo(h pp^p`OJQJo(oh @ @ ^@ `OJQJo(h ^`OJQJo(h ^`OJQJo(oh ^`OJQJo(h ^`OJQJo(h PP^P`OJQJo(oh   ^ `OJQJo(h ^`OJQJo(h ^`OJQJo(oh pp^p`OJQJo(h @ @ ^@ `OJQJo(h ^`OJQJo(oh ^`OJQJo(h ^`OJQJo(h ^`OJQJo(oh PP^P`OJQJo(h ^`OJQJo(h ^`OJQJo(oh pp^p`OJQJo(h @ @ ^@ `OJQJo(h ^`OJQJo(oh ^`OJQJo(h ^`OJQJo(h ^`OJQJo(oh PP^P`OJQJo(h ^`OJQJo(h ^`OJQJo(oh pp^p`OJQJo(h @ @ ^@ `OJQJo(h ^`OJQJo(oh ^`OJQJo(h ^`OJQJo(h ^`OJQJo(oh PP^P`OJQJo(h ^`OJQJo(h ^`OJQJo(oh pp^p`OJQJo(h @ @ ^@ `OJQJo(h ^`OJQJo(oh ^`OJQJo(h ^`OJQJo(h ^`OJQJo(oh PP^P`OJQJo( hh^h`OJQJo(* hh^h`OJQJo( hh^h`OJQJo([ ^`OJQJo(-hh^h`o(. hh^h`OJQJo(h ^`OJQJo(h ^`OJQJo(oh pp^p`OJQJo(h @ @ ^@ `OJQJo(h ^`OJQJo(oh ^`OJQJo(h ^`OJQJo(h ^`OJQJo(oh PP^P`OJQJo(h ^`OJQJo(h ^`OJQJo(oh pp^p`OJQJo(h @ @ ^@ `OJQJo(h ^`OJQJo(oh ^`OJQJo(h ^`OJQJo(h ^`OJQJo(oh PP^P`OJQJo( hh^h`OJQJo( hh^h`OJQJo(h88^8`B*OJQJo(phh ^`OJQJo(oh   ^ `OJQJo(h   ^ `OJQJo(h xx^x`OJQJo(oh HH^H`OJQJo(h ^`OJQJo(h ^`OJQJo(oh ^`OJQJo(h ^`OJQJo(h^`.hpLp^p`L.h@ @ ^@ `.h^`.hL^`L.h^`.h^`.hPLP^P`L. s^`sOJQJo(808^8`0o(.^`.pLp^p`L.@ @ ^@ `.^`.L^`L.^`.^`.PLP^P`L. hh^h`OJQJo( hh^h`OJQJo( s^`sOJQJo(@h ^`OJQJo(h ^`OJQJo(oh pp^p`OJQJo(h @ @ ^@ `OJQJo(h ^`OJQJo(oh ^`OJQJo(h ^`OJQJo(h ^`OJQJo(oh PP^P`OJQJo(hh^h`.^`o(. hh^h`OJQJo(@h ^`OJQJo(h ^`OJQJo(oh pp^p`OJQJo(h @ @ ^@ `OJQJo(h ^`OJQJo(oh ^`OJQJo(h ^`OJQJo(h ^`OJQJo(oh PP^P`OJQJo(h ^`OJQJo(h ^`OJQJo(oh pp^p`OJQJo(h @ @ ^@ `OJQJo(h ^`OJQJo(oh ^`OJQJo(h ^`OJQJo(h ^`OJQJo(oh PP^P`OJQJo(h ^`OJQJo(h pp^p`OJQJo(oh @ @ ^@ `OJQJo(h ^`OJQJo(h ^`OJQJo(oh ^`OJQJo(h ^`OJQJo(h PP^P`OJQJo(oh   ^ `OJQJo(h ^`OJQJo(h^`.hpLp^p`L.h@ @ ^@ `.h^`.hL^`L.h^`.h^`.hPLP^P`L.h ^`OJQJo(h ^`OJQJo(oh pp^p`OJQJo(h @ @ ^@ `OJQJo(h ^`OJQJo(oh ^`OJQJo(h ^`OJQJo(h ^`OJQJo(oh PP^P`OJQJo(/C-luCW{'1si/z.5E!0]V D ({*)ZAkc]boe,PL/h:P!|2x}T dQ 3xxG&hN \!JY2 W*HQ2(:H]YXNg2 MXT#!]Di 9P)}yek|} RmD> P!H-,-d9VT3xrL:J=L>p@8{`8>I!:2s1gcME{g! -JA(:rm:-,cmNqzFm|m=Zu| v0C8[)&gT&%k<C -bC-WE1-q')a06B"aJS1_n:l-rb};R P?J k7n9kM__ 3JS8QO.l -ewB,\E# ~}|q>:8H- BMVKld-cKU<-.S]SSSSSSdnpv{{|~XY]del "$&()+.479<BEGHJMRTVY^`bcehmoqty{}~Z459@ABIPQRY\_nowz}ɹʹ׹ڹݹ һ˾ϾԾվ!./gkpqǿȿ+BP c 1n:<>?RTVWc  E"08[SIY|]!i@m~"@w G Q>0%w { -F0 @ GTimes New Roman5Symbol3 Arial7Textile;Helvetica3Times? Courier New? Arial Black;WingdingsEMonotype Sorts"qhbcF|c\c&7=//!0d^|2q#IST Support Measure IST-1999-29067 Nigel Bevan Nigel Bevan Oh+'0   < H T `lt|'$IST Support Measure IST-1999-29067 ST  Nigel BevanigeNormale Nigel Bevan6geMicrosoft Word 9.0 @r[!@kb.@@0N/7= ՜.+,D՜.+,h$ hp  'Serco Usability Servicesli^ $IST Support Measure IST-1999-29067 Title 8@ _PID_HLINKS'A\9N$http://www.usabilitynet.org/methodsD9N$http://www.usabilitynet.org/methodsDOk>http://www.usabilitynet.org/methods/requirements/existing.aspDOk>http://www.usabilitynet.org/methods/requirements/existing.aspD  !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~      !"#$%&'()*+,-./023456789:;<=>?@ABCDEGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~     !"#$%&')*+,-./6Root Entry F]9N8Data 1(1TableFWordDocument%`SummaryInformation( DocumentSummaryInformation8(CompObjX FMicrosoft Word DocumentNB6WWord.Document.8