Development of a knowledge-based self-assessment system for measuring organisational performance

https://doi.org/10.1016/S0957-4174(02)00192-6Get rights and content

Abstract

Effective performance measurement is an important task in the discipline of engineering management. With the support of City University of Hong Kong, a research project was initiated to develop a knowledge-based expert self-assessment (KES) training toolkit on measuring and assessing organisational performance based on the evaluation criteria of a renowned Business Excellence Model—the Malcolm Baldrige National Quality Award (MBNQA). This paper explains the development of and elaborates the system framework, requirements, design and validation of the toolkit. The project results shows that the toolkit could facilitate the teaching of students of engineering management courses by providing a stimulating learning environment and practical experiences in measuring and assessing enterprise performance. Incorporating the KES model and toolkit into the engineering management curriculum can provide students and industrial users with hands-on experience and insights of organisational PM.

Introduction

There are various dimensions of enterprise performance measurement (PM), such as financial versus non-financial and qualitative versus quantitative. Traditionally, many organisations rely largely on financial measures and process outcomes using self-referenced objective data from internal sources. Meanwhile, there are also many organisations adopting the total quality management (TQM) philosophies to foster continuous performance improvements. In this context, it is essential for organisations to monitor their performance on a regular basis. Since the 1990s, the introduction of various international standards and quality/business excellence awards has helped thousands of organisations measure and assess their performance through exercises of management reviews, internal and external audits. Self-assessment against the compliance requirements of standards (e.g. ISO 9001:1994, ISO 14001:1996, and OHSAS 18001:1999) or criteria of renowned awards (e.g. the Malcolm Baldrige National Quality Award (MBNQA), the European Quality Award (EQA), and the Deming Prize (DP)) provides this type of assessment framework. The topic has received considerable attention from academic researchers and is well-defined in the literature (Bemowski and Stratton, 1995, Conti, 1997, Coulambidou and Dale, 1995, Lascelles and Peacock, 1996, Hakes, 1998). However, little use has been made of them to develop decision models and analysis tools for supporting the organisational performance assessment process. It is also found that a number of organisations do not have sufficient expertise and knowledge to carry out their own self-assessment process. Knowledge-based and action-learning approaches thus provide a feasible solution to apply these frameworks for self-assessment.

The idea of developing a knowledge-based or expert self-assessment training toolkit with a prototype system is useful, because it means that there is a ready-made methodology for applying the performance excellence model (e.g. MBNQA) to a business. Supported by the City UHK's quality enhancement fund, the purpose of this project was to develop such a knowledge-based expert self-assessment (KES) training toolkit and prototype system that facilitate teaching and student learning of engineering management courses in enterprise PM. This paper reviews the concepts and development of PM that help safeguard continuous performance improvement in organisations. It discusses the concepts and applications of knowledge-based expert systems and web-based training for measuring and assessing enterprise performance. It describes the framework of the model and explains the system requirements and design of the KES training toolkit, and draws the conclusions based on the evaluation results of the KES system. The project affirms that the proposed KES training toolkit and system can facilitate organisational learning and performance improvement processes.

Section snippets

Notion of performance measures

According to Neely, Gregory and Platts (1995), PM is a process of quantifying the efficiency and effectiveness of action that leads to performance. In the past, the focus of attention has been on measuring financial performance, such as sales turnover, profit, debt and return on investment. These financial measures do not match entirely with the competencies and skills required by companies for today's changing business environment (Geanuracos and Meiklejohn, 1993, Medori et al., 1995). It is

Concepts of total quality management and business excellence

The concepts of TQM and business excellence (BE) have come to the fore in recent times, being adopted by organisations as the means of understanding and satisfying the needs and expectations of their customers and taking costs out of their operations (Dale, 1999). TQM is an integrated management philosophy and set of practices that emphasise among continuous improvement, meeting customer requirements, reducing rework, long-range thinking, increased employee involvement and teamwork, process

The TQM–BE–PM integration and self-assessment

Recent research suggests that both TQM and PM can produce economic value to many firms (Dale, 1999, Kermally, 1997, Neely, 1998). One of the best indicators is the achievement or competitive advantage obtained from integrating TQM–BE concepts into performance measures. The integration has to comprise a thorough definition of measures and indicators to monitor the TQM implementation process and corporate performance from a stakeholder's perspective. Many researchers and practitioners believe

A systems framework

The proposed KES model has seven categories of evaluation criteria with respect to that of MBNQA (NIST, 2002). These categories are leadership, strategic planning, customer and market focus, information and analysis, human resource focus, process management, and business results. The criteria provide a systematic framework for assessing and measuring performance on a composite of key indicators of organisation performance (Fig. 1). This includes evaluating performance, identifying areas for

System requirements of KES

The system requirements for the KES training toolkit and prototype system development addressed three main areas, including (1) the self-assessment requirements, (2) the training requirements, and (3) the web design requirements. The self-assessment requirements describe what elements the system needs to have in order to serve as a self-assessment tool that is based on MBNQA (NIST, 2002). The training requirements describe elements in achieving the training purpose that acknowledges the user

Structuring acquired knowledge with Microsoft Access

The knowledge acquisition is to find out all the information for the KES training system. The knowledge was acquired from literature and interviews with industrial experts and academics. After the knowledge acquisition, the acquired expert knowledge was to be stored into different tables of the database with the aid of the Microsoft Access software. There are seven tables storing the questions, strengths, weaknesses and recommendations for seven categories of criteria, respectively. There are

System structure of KES

The KES system is a knowledge-based/expert system (KBS/ES). It is designed to execute through Internet, and the web server plays an important role in the system design. Fig. 4 shows the KBS/ES structure of the KES training system. The active server pages (ASP) scripting will be executed when receiving a request to the server. The execution of the scripts can be treated as the inference engine of the KBS/ES (Evans and Lindsay, 1987, Boyle, 1997). The working memory is the temporary memory that

System maintenance

This web-based system has two major components for ease maintenance. The first component is the ASP scripting that controls the overall functions of the system, and the second one is the database that stores all relevant information. In order to change or improve the features, the ASP scripting is required to modify based on any necessary changes. The modification of ASP scripting is a difficult task because wrong scripting might lead to the malfunction of the system. Regarding the

Development of training features

According to Gandell, Weston, Finkelstein and Winer (2000), several web capabilities can be used as action-based teaching and learning strategies. They include content presentation; searchable information; information exchange; guidance, practice, and feedback; discussion, and simulations. Gibbons and Fairweather (1998) also argue that web-based training can offer several advantages as compared with traditional modes of training. The KES system provides a web-based environment for the training

Evaluation of the KES system

In order to evaluate the applicability of the KES model and accompanying training toolkit, a web-based user satisfaction survey was conducted. A group of seven invited respondents (including industrial users and postgraduates) were asked to complete a survey questionnaire after a trial of the KES system. The questionnaire contained 10 questions that addressed three main areas, including the contents, interface design, and willingness to use. A five-point Likert-scale was employed.

Most

Conclusion

The success and continuity of an organisation depend on its performance. Recent business literature gives much prominence to balanced scorecards, TQM, BE models, and other similar approaches for assessing enterprise performance. Most of these systems and/or frameworks share the basic understanding that success in today's global marketplace requires measures of the critical aspects of performance. The interest in TQM–BE has been fuelled with a range of national and regional awards (such as,

Acknowledgements

The authors would like to thank City University of Hong Kong for supporting this project under the Quality Enhancement Fund (Project no. 8710199). The authors also appreciate the contribution from Mr I K Leung of the Department of Manufacturing Engineering and Engineering Management of City University of Hong Kong in the software development.

References (38)

  • D. Adebanjo

    TQM and business excellence: is there really a conflict?

    Measuring Business Excellence

    (2001)
  • K. Beasley

    Self-assessment: A tool for integrated management

    (1994)
  • K. Bemowski et al.

    How do people use the Baldrige criteria?

    Quality Progress

    (1995)
  • G. Bounds et al.

    Beyond total quality management: Towards the emerging paradigm

    (1994)
  • T. Boyle

    Design for multimedia learning

    (1997)
  • T. Conti

    Organisational self-assessment

    (1997)
  • L. Coulambidou et al.

    The use of quality management self-assessment in the UK: A state of the art study

    Quality World Technical Supplement

    (1995)
  • B.G. Dale

    Managing quality

    (1999)
  • R. Eccles et al.

    Creating a comprehensive system to measure performance

    Management Accounting

    (1992)
  • EFQM (2002). The European Quality Award, http://www.EFQM.org/...
  • J.R. Evans et al.

    Expert systems for statistical quality control

  • T. Gandell et al.

    Appropriate use of the web in teaching higher education

  • J. Geanuracos et al.

    Performance measurement: The new agenda

    (1993)
  • A.S. Gibbons et al.

    Computer-based instruction: Design and development

    (1998)
  • C. Hakes

    The corporate self-assessment handbook

    (1997)
  • C. Hakes

    Total quality management: The key to business improvement

    (1998)
  • R. Heimpel

    Elements of web course design

  • S. Henderson

    Black swans don't fly double loops: the limits of the learning organisation?

    The Learning Organisation

    (1997)
  • R. Hill

    A measure of the learning organisation

    Industrial and Commercial Training

    (1996)
  • Cited by (0)

    View full text