A systematic review of hospital accreditation: the challenges of measuring complex intervention effects

Our search of electronic databases identified a considerable increase in studies addressing the effect of accreditation and/or certification. In 2006, 672 studies were identified [11]. Over the next 3 years 522 new studies were published. In 2013 we identified another 910 relevant studies. Of the identified studies in 2013 fifteen citations were considered potentially eligible based on the inclusion criteria. Two additional articles were identified, and an additional three references were identified by manually searching the articles’ reference lists. Twenty references were considered potentially eligible and were retrieved for a full text assessment. Of these, 16 articles were excluded because they did not fulfil the inclusion criteria; Table 4 presents the excluded studies and the detailed reasons for their exclusion.

The agreement between reviewers for study eligibility was complete. As only one original study was included a meta-analysis was not possible (Fig. 1).

We included systematic reviews as well as controlled studies. A total of four references, three systematic reviews and one primary study met the inclusion [15‐18]. The aims and the inclusion criteria of the three reviews were slightly different. However, their inclusion criteria overlapped with the inclusion criteria for this review. Please see Table 1 for included systematic reviews in this review.

The qualities of the systematic reviews were assessed using the AMSTAR quality checklist framework, the standard for assessing methodological quality of systematic reviews [19]. The results of the assessment are shown in Table 2. Two of the reviews were of moderate quality scoring 6/11 [17], and 7/11 [16], respectively, whereas the third review was scored as high quality with a score of 9/11 [15]. Our review scored 9/11 in the AMSTAR assessment. The primary study [18] was assessed as having a high risk of bias after using the risk of bias assessment as described in the Cochrane Handbook for randomized controlled trails [20]. The assessment is shown in Table 3.

The Cochrane review by Flodgren et al. has the best quality AMSTAR score [15]. The authors identified two studies which met their inclusion criteria that focused on the effect of external inspection on: a) compliance with standards improving healthcare organizations; b) healthcare professional behavior; and, c) patient outcomes.

The first study was a cluster-randomized controlled trial by Salmon et al. [18] that involved 20 South African public hospitals. The other study was an interrupted time- series conducted to identify the effects of the NHS Healthcare Commissions Infection Inspection program on the MRSA rates in UK trusts hospitals, but did not meet our inclusion criteria. Flodgren et al. concluded that the results could not be used to draw firm conclusions on the effectiveness of external inspection.

The Matrix Knowledge group searched the literature in 2010 and found 56 articles that addressed the impact of hospital accreditation [16]. The vast majority of these studies used surveys with standardized questionnaires, and reported staff, patient and stakeholders’ perceptions of impact. Overall they reported a positive impact of accreditation on hospital and professional practice. Only the South African cluster-randomized controlled trial was consistent with the inclusion criteria of our study.

Alkhenizan and Shaw searched the literature in 2009 and included 26 studies that assessed either the general impact of accreditation on hospitals or impact on a single aspect of performance of healthcare services, and on sub-specialty accreditation programs. The authors found a positive effect of accreditation on improving the process of care and clinical outcomes [17]. Sixteen (62 %) of the 26 included studies reported significant positive results attributed to accreditation, mainly related to better compliance with guidelines. Ten studies (38 %) reported weak or no improvement after accreditation. Alkhenizan and Shaw included the one RCT by Salmon et al. [18].

There was one primary study identified that met all of our criteria, the randomized controlled trial from South Africa by Salmon et al. [18]. This study was not identified through the database search, but by searching reference lists (snowballing); it was missed by our literature review in 2009. The authors included 20 hospitals in their study. The hospitals were randomly selected and stratified into groups according to hospital size (number of beds). Ten hospitals were randomized to start an accreditation program, while the other 10 served as controls. Two sets of data, before and after measures, were collected by the Council for Health Services Accreditation of Southern Africa (COHSASA), and by independent research teams. Initially, 12 indicators of hospital care quality were identified and used for the first data collection – this number was reduced to eight in the second data collection. Of these indicators, surgical wound infection, time to surgery, neonatal mortality rate and financial solvency were left out due to challenges in data collection. It is unclear whether the four indicators that were abandoned would have influenced the overall magnitude, range of results or conclusions of the study.

The compliance with the COHSASA accreditation standard was found to have increased substantially in the accredited hospitals (p < 0.001), whereas the control hospitals maintained their score throughout the study. Eight hospital quality indicators were reported. The nurses’ perceptions of clinical quality was increased in the accredited hospitals (p = 0.031); however, the other seven indicators showed little or no effect on the quality indicators; patient satisfaction with care (p = 0.484); patient medication education (p = 0.395); accessibility of medical records (p = 0.492); completeness of medical records (p = 0.114); completeness of peri-operative notes (p = 0.489); labelling of ward stock (p = 0.112); and, composite assessment of hospital sanitation (p = 0.641).

Sixteen of the 20 studies were excluded after they were independently evaluated by two researchers (GVE, KB). The reasons for exclusion were as follows: four studies had no control groups [21‐24]; two performed the study outside hospitals [25, 26]; four studies did not report on the effects of accreditation [27‐30]; two studies lacked baseline measurements [31, 32]; one study lacked description of the accreditation intervention [33]; two studies did a comparison of the clinical outcome in accredited hospitals with outcome in non-accredited hospitals, but did not assess the effect of the intervention per se. [34, 35]; and, one systematic review conducted a qualitative assessment of healthcare professionals’ attitude toward accreditation, but the effect of the intervention was not assessed [36]. A complete list of the excluded studies and the reasons for their exclusion is presented in Table 4.

We identified one primary study and three systematic reviews. Notably, we had strict inclusion criteria and found few studies that met these strict criteria. A summary of the methods used in the excluded studies is relevant to the discussion on assessing the full measure of complex interventions. Accreditation was addressed in several ways in the publications that failed to fulfil the criteria for inclusion in the present review. Seven of the 16 excluded studies conducted cross-sectional studies comparing patient outcome in accredited and non-accredited hospitals [23, 25, 31, 32, 34, 35]. In general, performance in accredited hospitals was higher than in non-accredited hospitals and showing higher compliance to standards also affecting outcome positively [29]. The study by Lichtman et al. identified the risk of selection bias as certified hospitals had better outcome than non-certified hospitals even before the program began [28]. Four studies used survey to assess the staff and patient’ perception of patient safety culture, quality and patient satisfaction pre- and post accreditation [21, 22, 26, 30]. Nurses and patients reported that positive changes in their organization were a result of accreditation, while physicians in general were more sceptical. This is consistent with the findings in Alkhenizan and Shaw’s systematic review studying healthcare professionals’ attitude toward accreditation. Nurses in general were more favourably inclined than physicians indicating the necessity of special education schemes to involve staff in the accreditation process [36]. Another systematic review did not address accreditation directly, but found that physician specialization had effect on outcome of ovarian cancer patients [34]. A study on implementing nurse accreditation in one hospital reports increased staff and patient satisfaction, improved nurse-physician relationship, improved nursing quality and reduced turnover and vacancy rates [24]. The last study aims at proposing an integrated research model of the accreditation and quality measurement/reporting systems, providing more supportive information on the system weakness [27].