PRUE: Italian usability trial

PRUE has helped the Italian Ministry of Justice introduce usability requirements into the procurement of a new enhanced version of the existing dial-up ITALGIURE-FIND legal information service. SEMA has a contract with the Ministry to produce a new system accessible from internet browsers and mobile phones.

No information was previously available on the usability of the existing system, and there were no usability requirements in the contract (although the contract required ISO 13407 [6] to be used).

PRUE planned three activities to introduce usability to the procurement process:

A stakeholder meeting to identify the importance of usability and the intended context of use
An evaluation of the existing system to provide measures of usability as baseline requirements
An evaluation of a prototype of the new system to establish whether the requirements have been met

The stakeholder meeting was attended by senior representatives of the Ministry, SEMA and other stakeholders. Two user groups were identified: expert users with legal knowledge need functionality to support fast and efficient searches, consistent with the natural logic of juridical research. The system will also be accessible to new users without any legal knowledge who should be able to carry out simple searches without training.

In the evaluation of the existing system, five experts and four non-experts attempted to complete different sets of typical tasks. Expert users successfully accomplished 63% of tasks in comparison with 52% of non-expert users. The evaluation also provided the opportunity for SEMA designers to observe real usage of the system.

An important objective is for the new system to at least equal and if possible improve on these success rates. If the measured success rates and user satisfaction scores are not acceptable, the Ministry could negotiate with SEMA to provide an improved system.

The trial has demonstrated several benefits of introducing summative usability testing into the procurement process:

It provides a concrete benchmark for user performance and satisfaction, thus reducing the risk that the new system is more difficult to use (and therefore less successful) than the existing system.
It highlights usability problems with the existing system that need to be addressed in the design of the new system.
It provides specific goals for usability and gives developers the opportunity to became familiar with typical user task scenarios.
It provides the framework for the more detailed usability work required by ISO 13407.

Summative evaluation results reported in the CIF format have thus been very useful in helping the Ministry understand the needs of different user groups, and in establishing usability requirements for the new system. The user testing has also clear benefits in helping the supplier (SEMA) better understand Ministry of Justice requirements.

Background

Italgiure is the information retrieval software of the Corte di Cassazione. It returns sentences, case law, rules, summaries, etc. coming from the five databases of the Court:

Database of the Senior Council of Magistrates: sentences of the Council
Databases of the jurisprudence: sentences of Supreme Court, case law, civil jurisprudence, penal jurisprudence, etc.
Normative laws databases: national laws, regional laws, circular letters, Ministry rules, etc.
Doctrine databases: doctrine and juridical debates, juridical papers and journals, etc.
Other databases

Two versions are currently in use: ITALGIURE-FIND uses a command line interface, while EasyFind 5.0 is a windows-based form-filling interface for the Italgiure system.

The new version of ITALGIURE-FIND will replace the existing services. The main enhancements are:

Extend the number and size of files
Widen availability by making access possible from internet browsers and mobile phones
Improve the search interface, with versions for expert users and general users.

PRUE planned three activities:

A stakeholder meeting to identify the importance of usability and the intended context of use
An evaluation of the existing system to provide measures of usability as baseline requirements
An evaluation of a prototype of the new system to establish whether the requirements have been met

Context of use

The stakeholder and context meetings were attended by senior representatives of the Ministry, SEMA and other stakeholders. It was concluded that expert users with legal knowledge need functionality to support fast and efficient searches, consistent with the natural logic of juridical research. The system will also be accessible to new users without any legal knowledge who should be able to carry out simple searches without training. To carry out more complex searches, these users may need access to online training about the legal system and the use of efficient search strategies.

Longer term objectives include packaging the legal information in a format more intelligible to the non-legal user, and incorporating additional databases.

The environment in which the product is used is mainly an office and professional environment. Access will normally be by a web browser, but mobile phone access will also be supported.

The new system should enable existing users to complete their searches at least as quickly and as accurately as before, with at least as much satisfaction

The type of user work that is supported by the product is mainly the retrieval of legal information. Typical tasks are:

expert users: whether the supreme court has decided a question, and if so how many times and what laws were applied
general users: find a specific legal text by reference number, find judgements that relate to a defined legal problem. Potential user groups include people concerned with industrial relations, taxes and wages, damages claims, property disputes, marriage and divorce problems, patronage (insurance and pensions), university professors regarding university legislation.

Evaluation of existing system

The new system "New ITALGIURE-FIND" currently only has a temporary "developer" interface, so is not yet ready for evaluation. Design of the new user interface is just starting.

The main objective of the evaluation was to establish a baseline for development of the new interface and to identify existing usability problems.

To develop an easy to use interface developers needs to have a good understanding of how users perform their tasks at the moment, and any difficulties they have.

In order to fill this gap an evaluation focused on the current interface of ITALGIURE-FIND named EasyFind 5.0 was carried out.

Two different databases were considered in the evaluation:

"Civil jurisprudence", coming from the general databases of jurisprudence;
the Normative Laws Databases.

The user population is composed of: internal expert employees; internal inexperienced employees; different kinds of subscribers such as lawyers, juridical professional and practitioners, researchers, citizens.

No groups with special needs are supported by the product.

9 persons divided in two categories (5 expert and 4 non expert ITALGIURE-FIND users) were tested.

Tasks

Simple tasks for non expert (n=10) and more difficult tasks for experts (n=14) were used:

expert tasks are typically related with decisions of the Supreme Court together with the detail of the laws that were applied and the frequency pursuance;

non expert tasks are typically related with finding specific legal text by specifying its reference number or retrieving judgement concerning a certain legal problem.

The selected tasks were defined by the training manager, based on her experience in teaching the EasyFind interface in the training course. The tasks were selected as representative of general tasks for both expert and non expert users.

Procedure

Users were informed that the usability of EasyFind 5.0 was going to be tested to check whether the system met the needs of the users. They were told that it was not a test of their abilities. No pre-task training was used.

Users were then given 20 minutes to complete all tasks.

When each task finished the correctness was checked and time completion was recorded.

If the user required assistance the task was considered as failed.

At the end of the session users completed the SUMI questionnaire.

After the experimental session a debriefing was conducted in order to collect information about how to improve the new interface.

Performance results

Results in terms of effectiveness, efficiency, and satisfaction were obtained. In particular, effectiveness was measured as "number of tasks successfully accomplished"; efficiency in terms of "time to complete successfully all the task", satisfaction as SUMI scores.

The main reason for conducting the test was to get a baseline for the evaluation of the new system by comparing the users' performances and satisfaction whenever carrying on the same tasks in the old and new system.

Results can be summarized as follows:

Effectiveness: non expert users successfully accomplished an average of 52% of tasks in comparison with 63% of tasks by expert users;
Satisfaction: non expert users judged the interface as satisfactory as expert users (SUMI score is 55 for the two categories of users).

Satisfaction results

Italgiure survey

A SUMI survey of 7 existing internal Italgiure users showed a very consistent positive attitude to Italgiure. Users liked it very much and found it efficient, though they did not find it easy to learn, in particular they did not feel that the software helped them overcome problems they had in using it.

Easyfind survey

In contrast, although a SUMI survey of 8 existing external Easyfind users showed that they quite liked it, they had a consistently negative opinion of the efficiency, helpfulness, controllability and learnability.

Easyfind evaluation

In the Easyfind evaluation, the opinions of the expert and inexpert internal users were similar and generally quite positive despite the difficulty in completing all tasks.

Evaluation conclusions

Italgiuri

The survey results confirmed that existing internal expert users are very satisfied with the Italgiure command line interface.

Easyfind

The success rates for expert and inexpert tasks of 63% and 52% were not high, and there was dissatisfaction with some aspects of the interface.

Results show that the new interface has to be both:

At least as good as Italgiure Find for internal expert users;
Better than Easy Find 5.0 for both internal non expert users and general public.

This is the most important feedback for designers of the new interface (New Italgiure Find) as these two search interfaces (Italgiure Find and EasyFind 5.0) are very different in terms of required skills and working experience.

Starting from the test results the baseline objectives for the new system will be provided within next December. Such objectives (both for expert and non expert) will be the baseline in order to compare usability characteristics of the New Italgiure Find interface with the existing interfaces (Italgiure Find and EasyFind 5.0). A planning for future work has been defined as well.

Conclusions

Summative evaluation results reported in the CIF format have been very useful in helping the Ministry understand the needs of different user groups, and in establishing usability requirements for the new system.

Ministry of Justice benefits

The work done within the PRUE project so far has proven to be of great value for Ministry of Justice. The initial tests lead to development of a usability requirements document for the new interface. It has been seen as a new and positive input to the Ministry of Justice decision process when selecting a supplier of a new system. The usability requirements are regarded to be good compliment to the functional and technical specifications. The benefits are the increased understanding of user performance in current system, understanding of what user think of the current system and what they require from the new.

Ministry of Justice is considering products from other supplier organizations than the one tested, and the usability requirements document will be one factor supporting the assessment of these alternatives.

Supplier organization benefits

The user testing of the supplier organization's product (SEMA) has also clear benefits. They have been able to understand Ministry of Justice requirements, assess their product and improve the interface.

Last updated 12-Mar-02

Usability requirements: how to specify, test and report usability

Case study: Italian Ministry of Justice/SEMA: Legal Web Site

Page contents

Overview