Keywords

1 New ISO Standards for Usability

Standards from the International Organization for Standardization (ISO) are produced by international groups of experts after a rigorous review process, and represent a consensus on the current state-of-the art. Because of the number of international experts involved in their development they provide a more balanced perspective than is typically found in textbooks or individual publications. Several new and revised standards relating to usability will be published in 2016/7:

  • ISO 9241-11: Usability: Definitions and concepts [6], replacing the 1988 version of ISO 9241-11.

  • ISO 9241-220: Processes for enabling, executing and assessing human-centred design within organizations [9], replacing the earlier ISO TR 18529.

  • ISO/IEC 25066: Common industry Format for Usability — Evaluation Reports [17].

  • ISO/IEC 25022: Measurement of quality in use [11], (includes measures of effectiveness, efficiency and satisfaction), replacing ISO TR 9126-4.

  • ISO/IEC 25023: Measurement of system and software product quality [12], (includes measures for usability attributes), replacing ISO/IEC TR 9126-2 and ISO/IEC TR 9126-3.

The paper summarises some of the new content included in these standards.

2 ISO 9241-11: Usability: Definitions and Concepts

The new version of ISO 9241-11 retains and elaborates on the concepts in the 1988 version of the standard, with the original definition of usability extended to apply to systems and services: “the extent to which a system, product or service can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use”. The content has been extended to include current approaches to usability and user experience [2]. Some examples are given below.

Goals.

Historically, usability has been associated with achieving predetermined practical goals, but the concept has been widened in the new version of ISO 9241-11 to include achieving personal outcomes such as entertainment or personal development. It is also recognised that users can have several interrelated goals. For example the goal of completing a report could include the potentially conflicting sub-goals to (a) please the client by sending the report today, (b) ensure a quality of content that indicates professional expertise, and (c) finish the report in time to get home for a family event.

Absence of Negative Consequences.

Effectiveness has been associated with completing a task completely and accurately, but it is also important to take account of the potential negative consequences if the task is not achieved correctly. The new definition of effectiveness: “accuracy, completeness and lack of negative consequences with which users achieved specified goals” takes account of negative consequences such as:

  • economic harm

    e.g. high costs for accidently selected the roaming option on a mobile phone

  • harm to health

    e.g. injury of a patient resulting from use errors with a medical device

  • harm to the environment

    e.g. unnecessary use of energy through inability to set central heating controls appropriately

Objectively Achieved Outcomes and Subjectively Perceived Outcomes.

For a successful outcome of interaction with an interactive system (effectiveness) both objective and perceived success is typically necessary. There can be negative consequences if objective success is perceived as failure, for example, if you book a flight but don’t get a confirmation, so assume the booking was not made, and book again. Now you have two reservations! Perceiving objective failure as success can also have negative consequences, for example, you use a voting machine to make your vote. You think it was made, but it is not counted because you did not press hard enough. If you had realised it was not successful, you would have tried again.

Satisfaction.

Satisfaction has been redefined to take account of the wider range of concerns that are now recognised as important for user experience: “positive attitudes, emotions and/or comfort resulting from use of a system, product or service”. These three aspects relate to the cognitive, affective and psychomotor responses of an individual.

Wide Applicability.

The new draft makes it clear that usability applies to all aspects of use, including:

  • Learnability, to enable new users to be effective, efficient and satisfied when learning to use a new system.

  • Regular use, to enable users to achieve their goals effectively, efficiently and with satisfaction.

  • Accessibility, so that the system if effective, efficient and satisfying for users with the widest range of capabilities.

  • Maintainability, to enable maintenance tasks to be completed effectively, efficiently and with satisfaction.

3 ISO 9241-220: Processes for Enabling, Executing and Assessing Human-Centred Design Within Organizations

According to ISO 9241-210, human-centred design is an “approach to systems design and development that aims to make interactive systems more usable by focusing on the use of the system and applying human factors/ergonomics and usability knowledge and techniques”.

ISO 9241-220 elaborates on ISO 9241-210 to provide a comprehensive description of the processes that support the activities that are required as part of human-centred design. Figure 1 (adapted from ISO 9241-220) summarises the processes that need to be in place in each area of an organisation that has some responsibility for human centred design. The groups of processes related to these levels are called “Human Centred Process categories” (HCP). HCP.1 addresses what organizations need to do to enable human-centred design on a corporate level. HCP.2 describes the required infrastructure and management of human-centred design across projects and systems, and HCP.3 details the project-specific aspects of human-centred design during development or change of a system. HCP.4 covers the specific processes during introduction and operation of a system. Together the implementation of these four sets of processes ensures that the systems produced, acquired and operated by an organization have appropriate levels of usability, accessibility, user experience and mitigation of risks that could arise from use. This combined objective is referred to as human-centred quality, the: “extent to which requirements relating to usability, accessibility, user experience and minimizing risks arising from use are met”.

Fig. 1.
figure 1

Human centred design process categories and contents

Full size image

The current draft of the standard proposes a broadened definition of user experience to take account of the way the term is commonly used: “a person’s perceptions and responses resulting from the use and/or anticipated use of an interactive system, and from the user’s interaction with the organization that supplies or delivers the interactive system; from discovering the system, adopting and using it, through to final use”.

ISO 9241-220 can be used for:

  • implementing human-centred design based on the process outcomes needed to achieve human centred quality as part of a system development or procurement process and/or support lifecycle;

  • assessing an enterprise’s existing capability to carry out the human-centred processes;

  • improving the effectiveness of human-centred design as part of an existing system development process;

  • specification and development of necessary competence in human-centred design.

4 ISO/IEC 25066: Usability Evaluation Reports

ISO/IEC 25062 Common Industry Format (CIF) for usability test reports, published in 2008, specifies the information to be included in a summative test report that is intended to enable a supplier to demonstrate the usability of their system to a potential purchaser, and the purchaser to judge whether the system would be usable in their own context of use. The information required by this standard is more detailed than is necessary for most formative evaluation reports.

The industry working group that developed the format that became ISO/IEC 25062 [1] worked on the production of guidance for the content of a formative report based on user testing [18] but concluded that the extent of the variation needed in practice was too great to support the development of a single recommended format.

When the new ISO/IEC 25066 Common Industry Format for usability (CIF) – Evaluation reports was developed to cover a wider range of approaches to usability evaluation, rather than focusing on the summative or formative purpose of evaluation, the approach taken was to specify the contents that should be included for different types of usability evaluation:

  1. (a)

    Inspection to identify usability defects and the corresponding potential usability problems.

  2. (b)

    User observation

    • Qualitative: Observing user behaviour to identify actual usability problems.

    • Quantitative: Measuring user performance and responses to obtain data on effectiveness and efficiency.

  3. (c)

    Obtaining subjective information from users including:

    • Qualitative: Problems, opinions and impressions given during or after a usability evaluation.

    • Quantitative: Measures of user satisfaction or perception.

The potential content items for a report that included each of these types of evaluation is categorised under the headings:

  1. 1.

    Executive summary (if used)

  2. 2.

    Description of the object of evaluation

  3. 3.

    Purpose of the evaluation

  4. 4.

    Method

    1. a.

      General

    2. b.

      Evaluators/participants

    3. c.

      Tasks (if used in the evaluation)

    4. d.

      Evaluation environment

  5. 5.

    Procedure

    1. a.

      Design of the evaluation

    2. b.

      Data to be collected

  6. 6.

    Results

    1. a.

      Data analysis

    2. b.

      Presentation of the results

  7. 7.

    Interpretation of results and recommendations

For each type of content, the standard lists the potential content items relevant to each type of usability evaluation, indicating whether they are required, recommended or optional. Table 1 is an example of some of the items for the evaluation environment.

Table 1. Evaluation environment
Full size table

The 64 content items in the standard can be used as a checklist to ensure that for each type of evaluation that is reported, all the required content and appropriate recommended items are included.

5 ISO/IEC 25022: Measurement of Quality in Use

Quality in use which is defined in ISO/IEC 25010 System and software quality models includes: effectiveness, efficiency, satisfaction, freedom from risk and context coverage. The measures for quality in use in ISO/IEC 25022 include measures for the components of usability that are defined in ISO 9241-11: effectiveness, efficiency and satisfaction (Table 2).

Table 2. Measures of effectiveness, efficiency and satisfaction
Full size table

ISO/IEC 25022 provides suggested measures for freedom from risk, which is defines as the: “degree to which the quality of a product or system mitigates or avoids potential risk to the user, organisation or project, including risks to economic status, human life, health, or the environment” (Table 3). This recognises that it is not sufficient simply to consider successful interaction, it is also important to minimise the possibility of any adverse consequences that could arise from poor usability (such as an individual failing to purchase the correct transport ticket, adverse economic consequences for a company, or environmental consequences resulting from poor usability of home-heating controls). The importance of the risk of negative consequences that can result from poor quality was first introduced in the ISO/IEC 9126-1 standard in 2001, and was updated in ISO/IEC 25010 in 2011. The concept of the risk of negative consequences that can result from use is being incorporated into ISO 9241-220 and is addressed in the revised version of ISO 9241-11 as part of effectiveness.

Table 3. Measures of risk
Full size table

An example measure is: “Proportion of usage situations where there are human or system errors with economic consequences”. Although it is usually not possible to control other factors in the context of use that could influence freedom from risk, it is often possible to provide evidence for the potential risks that could result from poor usability or poor product quality, and to suggest target values for usability that would mitigate these risks.

ISO/IEC 25022 also provides measures for context coverage: “the degree to which a product or system can be used with effectiveness, efficiency, satisfaction and freedom from risk in both specified contexts of use and in contexts beyond those initially explicitly identified” (Table 4). Context coverage was introduced into ISO/IEC 25010 to support specification and evaluation of usability in all defined and anticipated contexts of use. The measure of context completeness is defined as: “The proportion of the intended contexts of use in which a product or system can be used with acceptable usability and risk”. It could for example be measured by the proportion of the intended contexts of use for which there are measures showing that the product can used with acceptable usability and risk.

Table 4. Measures of context coverage
Full size table

Flexibility measures are used to assess the degree to which a product or system can be used with acceptable levels of effectiveness, efficiency, satisfaction and freedom from risk in contexts beyond those initially specified in the requirements for the system. Flexibility enables products to take account of circumstances, opportunities and individual preferences that might not have been anticipated in advance. Flexibility can be measured by analysing the characteristics of the product and the context of use to assess the extent to which a product can be used by additional types of users to achieve additional goals with effectiveness, efficiency, satisfaction and freedom from risk in additional contexts of use, or by testing the product with users in these additional contexts of use, or by the capability of the product to be modified to support adaptation for new types of users, tasks and environments by means of suitability for individualization as defined in ISO 9241-110.

Proficiency independence assesses the extent to which the product can be used by people who do not have specific knowledge, skills or experience. The product could be primarily intended for a user group with specific knowledge, skills or experience, but potentially usable by a wider range of types of user.

6 ISO/IEC 25023: Measurement of System and Software Product Quality

ISO/IEC 25023 contains measures for system and software product quality, including measures of usability as a product attribute (Tables 5 and 6). The main reason for including the usability measures was to complement the better-established measures for software qualities such as system performance, reliability, security and maintainability. Measures such as response time, system availability, fault tolerance and data integrity are widely used for specifying, evaluating and monitoring software quality. Without equivalent measures for usability that can be used in early systems development, usability is likely to be ignored in favour of the hard software qualities.

Table 5. Measures of usability attributes
Full size table
Table 6. Measures of usability attributes (continued)
Full size table

This creates a problem, as usability in early systems development typically focuses on identifying and fixing usability defects, and it is not clear whether detailed measures of usability properties would be of any benefit to a usability engineer. So the initial audience for detailed usability measures is likely to be the software testers who will only be able to use quite simple measures in a restricted context of use, but at least it will flag usability as an important issue in systems development.

In principle the evaluation of almost any usability guideline (of which there are hundreds in the literature) could be treated as a measure, which caused a problem deciding which usability measures to include in ISO/IEC 25023. Eventually it was decided to use selected examples of “dialogue principles” in ISO 9241-110 as a source.

7 Conclusions

The ISO 9241-11, ISO 9241-220 and ISO/IEC 25066 standards provide rich resources for usability practitioners and researchers, who may also find the measures in ISO/IEC 25022 and 25023 useful. ISO/IEC 25023 is primarily intended for use in a software development environment, and cross-references the ISO 9241 series of standards for additional information.

The standards described are part of the ISO 9241 series developed by the ISO Ergonomics subcommittee TC159/SC4 and the ISO 25000 series developed by the Systems and software engineering committee ISO/IEC JTC1/SC7. The ISO/IEC 2506x Common Industry Format standards are developed jointly by the two committees and also include:

  • ISO/IEC 25063: Context of use description

  • ISO/IEC 25064: User needs report

  • ISO/IEC 25065: User Requirements Specification [in preparation]

If you would like to contribute to the development of future standards related to usability, or to comment on drafts, you can either do this via your national standards body [3], or if you are a member of one of the ISO TC159/SC4 liaison organisations [4] such as UXPA [19] you can participate through the liaison organisation.