Figures
Abstract
Background
Surgical site infection (SSI) surveillance is a key factor in the elaboration of strategies to reduce SSI occurrence and in providing surgeons with appropriate data feedback (risk indicators, clinical prediction rule).
Aim
To improve the predictive performance of an individual-based SSI risk model by considering a multilevel hierarchical structure.
Patients and Methods
Data were collected anonymously by the French SSI active surveillance system in 2011. An SSI diagnosis was made by the surgical teams and infection control practitioners following standardized criteria. A random 20% sample comprising 151 hospitals, 502 wards and 62280 patients was used. Three-level (patient, ward, hospital) hierarchical logistic regression models were initially performed. Parameters were estimated using the simulation-based Markov Chain Monte Carlo procedure.
Results
A total of 623 SSI were diagnosed (1%). The hospital level was discarded from the analysis as it did not contribute to variability of SSI occurrence (p = 0.32). Established individual risk factors (patient history, surgical procedure and hospitalization characteristics) were identified. A significant heterogeneity in SSI occurrence between wards was found (median odds ratio [MOR] 3.59, 95% credibility interval [CI] 3.03 to 4.33) after adjusting for patient-level variables. The effects of the follow-up duration varied between wards (p<10−9), with an increased heterogeneity when follow-up was <15 days (MOR 6.92, 95% CI 5.31 to 9.07]). The final two-level model significantly improved the discriminative accuracy compared to the single level reference model (p<10−9), with an area under the ROC curve of 0.84.
Citation: Saunders L, Perennec-Olivier M, Jarno P, L’Hériteau F, Venier A-G, Simon L, et al. (2014) Improving Prediction of Surgical Site Infection Risk with Multilevel Modeling. PLoS ONE 9(5): e95295. https://doi.org/10.1371/journal.pone.0095295
Editor: Cornelius J. Clancy, University of Pittsburgh, United States of America
Received: October 21, 2013; Accepted: March 25, 2014; Published: May 16, 2014
Copyright: © 2014 Saunders et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Funding: The French national surgical site infection surveillance system is supported by the French Ministry of Health. The funder had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Introduction
Surgical site infection (SSI) is one of the most frequent hospital-acquired infections occurring in surgical patients and leads to increased morbidity, mortality and costs [1]–[3].
Since the early 1980s, SSI surveillance which provides appropriate data feedback (risk indicators, clinical prediction rule, etc.) to surgeons is considered an important component of the strategies developed to reduce SSI occurrence [4]–[6].
In this context, many countries have developed a national system for the surveillance of nosocomial infections. In France, a national surgical site infection surveillance system (the RAISIN surveillance system) based on a pyramidal organization (local, regional and national) was implemented in 1999 [7], using standard guidelines established in 1992 by the American Centers for Disease Control and Prevention (CDC)’s National Nosocomial Infections Surveillance (NNIS) system [4]. These guidelines were recently updated by the French Technical Committee for Nosocomial Infections and Healthcare Associated Infections [8]. The RAISIN surveillance system relies on volunteer surgical wards from public or private hospitals that routinely collect nosocomial infection data. This surveillance system reported a crude incidence rate of one SSI per 100 procedures in 2009–2010 [9].
To improve the accuracy of surveillance, successive risk indicators have been suggested [10]–[12]. They are mainly based on stratification or adjustment of SSI risk such as the NNIS risk index (consisting of 4 categories based on the American Society of Anesthesiologists [ASA] physical status score, wound class and duration of surgery). The SSI incidence rate among patients with an NNIS risk index of 0 is now provided by many surveillance systems. A more recent approach uses the Standardized Infection Ratio (SIR), calculated as the number of reported SSI divided by the number of expected SSI for a given ward and year [13]. The latter is the sum of individual SSI probabilities calculated from a logistic regression reference model including only individual risk factors [9]. As reported by Rioux et al. [14], the SIR was found to be a more reliable indicator when estimating the reduction in SSI incidence than the NNIS-0 SSI incidence rate, as it includes individual characteristics. Thus, each ward participating in the RAISIN surveillance system currently uses this model to rate its performance (a SIR greater than 1 indicates a lower performance).
The common feature (and limitation) of these risk indicators is that they exclusively take into account individual factors to assess SSI risk. Nevertheless, it seems important to consider data hierarchy in order to determine the role played by other specific risk factors, such as ward or hospital characteristics. A suitable statistical method for analyzing grouped or clustered data is multilevel modeling, which has the following advantages: correction of standard error underestimation, examination of cross-level interactions, estimation of the coefficient variability at group level, and analysis of contextual effects after adjusting for individual variables, while also accounting for the non independence of within-group observations [15]–[17].
The aim of this study was, therefore, to improve the predictive performance of an individual-based SSI risk model by investigating the adequacy of a multilevel hierarchical structure.
Methods
Data Collection
The RAISIN active surveillance system has been fully described elsewhere [18]. Every year, volunteer surgical wards are asked to complete a survey over a three-month period chosen at their own discretion between January 1, and June 30. In each participating facility, all consecutive surgery procedures occurring during the first 2 months are included with the exception of reoperations due to SSI. Then, a 30 day follow-up is theoretically organized after the procedure (including post hospital discharge) and stopped in case of SSI occurrence. To avoid reporting bias, a standardized questionnaire is completed for each patient by the surgical team with the help of an infection control practitioner to document peri- and post-operative data [19]. An SSI diagnosis is made according to standardized CDC criteria. Every type of surgery is taken into account, including ambulatory surgery (provided a follow-up is possible). Quality control is supervised by the local Coordination Centre for Nosocomial Infection Control (CCLIN). At national level, the database is maintained by the 5 CCLINs involved in the RAISIN surveillance system, in cooperation with the French Institute for Public Health Surveillance.
The 2011 national SSI data consisted of 330202 patients from 2433 wards and 756 hospitals (27.9% of all French hospitals). Among these participating facilities, 50.8% were private hospitals, 8.2% were semi-private hospitals and 41.0% were public hospitals.
To generate unbiased and accurate estimates from multilevel models, we limited our study to wards of at least 10 patients. We randomly selected 20% of the hospitals from within this subset, thus providing a study population of 151 hospitals, 502 wards and 62280 patients (Figure 1).
To assess the added-value of a multilevel model as opposed to conventional single level models, the following patient characteristics were considered: gender (male vs. female), age (<65 years vs. ≥65 years), ASA physical status score [20], duration of preoperative hospitalization (<48 hours vs. ≥48 hours), wound class according to the Altemeier classification, endoscopic surgery (yes vs. no), type of surgical procedure (gastrointestinal, gynecologic, cardiovascular, orthopedic, ophthalmic, others), duration of surgery (≤75th percentiles vs. >75th percentiles), emergency status of surgery (yes vs. no), ambulatory surgery (yes vs. no), duration of post surgery follow-up (<15 days vs. ≥15 days), time of SSI occurrence (during hospitalization vs. after discharge), depth of infection (superficial, deep, organ/space), mean time between surgery and SSI diagnosis, second surgical procedure (yes vs. no), patient outcome (alive vs. deceased within 30 days). Individual variables with a proportion of missing values greater than 10% were discarded from analyses. No ward-level explanatory variable was available. Regarding hospital level, two characteristics were considered: status (public, private, semi-private) and hospital category (private hospital, teaching hospital, district hospital, regional hospital, military healthcare facility, cancer hospital).
Statistical Analysis
To take into account the hierarchical structure of data, analyses were initially performed using a 3-level (patient, ward, hospital) hierarchical logistic regression model. The binary outcome was the occurrence of an SSI during the first 30 days post-surgery. Three successive multilevel models were fitted to the data. An empty model (model 1) was built with a random intercept at patient, ward and hospital levels containing no independent variables at any level, in order to determine the initial distribution of the variance of the dependent variable between the three levels. Heterogeneity within levels was considered significant at the p value threshold of 0.05. Patient characteristics with a p value <0.20 in a prior univariate multilevel analysis were included in a multivariate multilevel model, allowing for the probability of SSI to vary across wards but assuming that the effects of individual explanatory variables were the same for each ward (model 2 or random intercept model). Furthermore, in the third model (model 3 or random coefficient model), we introduced random coefficients to allow explanatory variable effects to vary between wards. Parameters were estimated using the simulation-based Markov Chain Monte Carlo (MCMC) procedure for discrete response multilevel models in a Bayesian framework [21]. To quantify heterogeneity between wards, the median odds ratio (MOR) was calculated. The aim of this measure is to translate the group level variance in the widely used odds ratio scale, which has an intuitive interpretation, and is statistically independent from the prevalence of the outcome [22]. Ninety-nine percent credibility intervals (CI) for the MOR were calculated using the 2.5th and 97.5th percentiles of the posterior distribution of the ward variance.
The final model was cross-validated in an independent 20% hospital sample, randomly selected from the same 2011 national database after exclusion of the original 151 hospitals from the study population. This validation sample comprised 51348 patients. Coefficients from multilevel modeling performed on the study population were applied to the validation population, as well as the coefficients from a single level logistic regression model, considered as the reference model (also using the 2011 data to allow direct comparison) [9].
Discriminative accuracy of both models was compared by the means of their respective area under the receiver operating characteristic (ROC) curve, using the Hanley and Mc Neil method [23], [24].
Descriptive analyses were performed using the software SAS version 9.3 (SAS Institute, Cary, NC, USA) and multilevel modeling was carried out using the software MLwiN version 2.23 (Centre for Multilevel Modelling, Bristol University, UK).
Ethics
This study was given ethical clearance by the French data protection authority (n°326452). As only anonymous data were extracted from patients’ medical files for this surveillance study, the French data protection authority deemed that verbal informed consent was acceptable provided patients were supplied with, as was the case, a document describing the objectives of the study, their right to access the data collected concerning them, and their right to have it rectified (in compliance with act n°78–17 of 6 January 1978 on Information Technology, Data Files and Civil Liberties, amended by act n° 2004–801 of 6 August 2004 relating to the protection of individuals with regard to the processing of personal data).
Results
Study Population
Patients’ characteristics are reported in Table 1. The study population was mainly female (56.3%) and age <65 years (58.6%). The vast majority of patients were hospitalized less than 48 hours (92.3%) prior to surgical procedure and had an Altemeier wound class ≤2 (93.0%). Follow-up after surgery was ≥15 days for 65.0% patients.
Of the 62280 patients included, 623 SSI were diagnosed (1%), 40.3% of which were diagnosed during hospitalization. Forty-eight percent of SSI were superficial, 33.4% were deep incisional infections and 16.8% were organ/space infections. The median time between surgery and SSI diagnosis was 10 days, and 35.5% of the patients required a second surgical procedure. Ten patients died during hospitalization. The main statuses of the hospitals were private (50%) or public (46.2%).
Multilevel Modeling
Results from the empty model revealed a significant heterogeneity of SSI occurrence between individual wards (p<10−6), but not between individual hospitals (p = 0.32). Patient and ward levels were therefore kept in the subsequent analyses. The hospital level was discarded as it did not contribute to variability of SSI occurrence. In the resulting two-level empty model, ward-level variance could be characterized by a MOR of 3.02 (95% CI 2.47 to 4.71).
Multivariate results are presented in Table 2. In model 2, SSI occurrence was significantly higher among males (p = 0.04), patients with an ASA score >2 (p<10−8), patients with a duration of preoperative hospitalization ≥48 hours (p<10−3), patients whose surgical wound was contaminated or dirty according to the Altemeier wound class (p<10−6), and patients whose surgery duration was greater than the 75th percentile value for similar procedures (p<10−9). In contrast, SSI occurrence was lower among patients who underwent endoscopic surgery (p<10−2); patients who underwent orthopedic surgery (p<10−6), ophthalmic surgery (p<10−2), or other types of surgery (p = 0.002) (versus gastrointestinal surgery); patients who underwent ambulatory surgery (p<10–8); patients with a follow-up≥15 days after the operation (p<10−9). When including these patient-level variables, ward-level variance remained significantly different from zero (p<10−9) with a MOR of 3.59 (95% CI 3.03 to 4.33).
The final model (model 3) was as follows:
with:
πij: the probability of SSI occurrence in patient i in ward j.
i: 1.number of patients.
u0j: random intercept effect.
u15j: random coefficient effect.
Corresponding exponentiated estimates (odds ratios) are reported in Table 2. The random coefficient for follow-up duration (u15j) was statistically significant (p<10−9), while the conclusions regarding the effects of patient variables were unchanged. For a duration of follow-up<15 days, the MOR was 6.92 (95% CI 5.31 to 9.07), whereas for a follow-up≥15 days, the MOR was 3.32 (95% CI 2.49 to 4.84). No significant random effect was highlighted for the remaining patient-level variables.
Validation
ROC curves for the multilevel logistic regression model and the reference single level logistic regression model (both for 2011) are reported in Figure 2. The risk factors taken into account are the same in both models. However, variables in the reference single level model have no subscript j that would indicate to which ward the patients belong (and therefore no random effects are estimated). When exclusively taking into account individual factors in the reference model, the area under the ROC curve was 0.73. The multilevel logistic model (considering ward variability) significantly improved the discriminative accuracy (p<10−9), with an area under the ROC curve of 0.84.
Discussion
This study sheds new light on the respective contribution of patient-, ward- and hospital-levels to SSI occurrence and demonstrates a significant impact of the ward level in itself (i.e. with no ward-level explanatory variable) after adjusting for patient-level variables.
The strengths of the study stem from a large population base, a high completeness rate, and a sound statistical methodology. To avoid convergence problems, we randomly selected a 20% sample from the 2011 database, comprising a significant number of wards (502) and a sufficient number of patients per ward (≥ 10 patients). These criteria are recognized as crucial for valid and reliable estimates of the fixed and random effects from multilevel analyses [25], [26]. Successive controls at local, regional and national level resulted in good quality data in terms of completeness (less than 1% missing data) and consistency. Thus we were able to use a large number of the variables available in the French RAISIN national database. Assuming that the organization or activity of a ward or hospital might have an impact above individual characteristics, we used multilevel modeling techniques with the distribution-free MCMC method, to take into account the hierarchical structure of the data, the non-independence of within-group observations, and to obtain better parameters estimation.
There are a few limitations to this study that should be noted. First, as involvement in the surveillance system is voluntary, wards paying more attention to infection prevention measures may be overly-represented in the database. Nevertheless, this should not affect our main results as we have highlighted the added-value of a two-level model compared to a single level reference model, all else being equal. Second, lack of information regarding some risk factors difficult to collect in a routine surveillance system (preoperative skin preparation, antibiotic prophylaxis, hand hygiene compliance) may be the source of potential residual confounding. Third, the completeness of the follow-up is questionable as inferred by the proportion of patients without SSI and with a follow-up<15 days (34.7%, Table 1), illustrating an insufficient compliance with the surveillance requirements. However, as SSI surveillance policy is implemented at the ward level in the RAISIN system, any difference in follow-up completeness is for the most part accounted for by the between-ward heterogeneity. Fourth, a 30 day follow-up is quite a short period of time considering SSI may occur some months or years after surgery (especially in the case of surgical implant). Lastly, between-ward heterogeneity was revealed but we were unable to further explore ward-level factors potentially explaining this variability. In this respect, the variable “ward’s post-hospitalization follow-up procedure” (optional in the surveillance network) could not be introduced as a ward-level risk factor because of the excessive amount of missing data (probably in relation with the above-mentioned follow-up completeness issue).
This multilevel study highlights individual factors (patient history, surgical procedure and hospitalization characteristics) as significant risk factors for SSI occurrence, in line with previous studies [9], [18]. It confirms the relevance of collecting these individual data in national surveillance systems such as the surveillance system established by the CDC in the United States (the effectiveness of which has been proven) [4] and at European level by the European Centre for Disease Prevention and Control (ECDC) [27]. It goes, however, one step further by providing a case-mix adjusted model allowing proper between-ward comparisons as individual risk factors (notably the Altemeier classification) are accounted for.
In an empty model, we found a between-patient and between-ward heterogeneity. Conversely, hospital-level had no significant effect on SSI risk. This result emphasizes the importance of taking into account the organizational context in which care is delivered to predict the SSI risk [28], both from a clinical and statistical point of view. This approach is consistent with recent studies that have reported the benefit of simultaneously including both individual and aggregated characteristics in health sciences, particularly in the context of infectious health outcomes [29]–[35]. In the multivariate multilevel model 2, the MOR of 3.59 (95% CI, 3.03–4.33) shows that, in the median case, the residual heterogeneity between wards increases by 3.59 times the individual odds of SSI when randomly picking out two persons in different wards. We also found an increased heterogeneity between wards with follow-up<15 days (MOR = 6.92 [95% CI, 5.31–9.07]). This may reflect a lack of systematic reporting procedure in some wards. As the vast majority (67%) of SSI occur in the first 15 postoperative days [9], reported SSI may subsequently be concentrated in wards that comply with the surveillance requirements, increasing the between-ward heterogeneity during this period.
In conclusion, the present study provides evidence of the existence of ward-level effects over and above risk factors present at patient level (i.e. independently from patient case-mix). As the ward-level significantly improved the discriminative accuracy, a new two-level risk model (possibly stratified by surgical procedure type) should be implemented to allow clinicians both to predict risk in individual patients and to have at their disposal a better estimate of the expected number of SSI in their respective wards. From a pragmatic point of view, the ward housing a given patient is an easy-to-collect and reliable data element, making the two-level risk model applicable to other settings. Some external validation should, however, be conducted to confirm the robustness of the results stated above. Further work is also required to establish the potential added-value of ward characteristics (post-discharge follow-up process, preoperative skin preparation, hand hygiene compliance, antibiotic prophylaxis policy) to be collected systematically.
Acknowledgments
We wish to thank all the professionals from the French hospitals who contributed to data collection, and Maryvonne Dixon for editorial assistance.
Contributors
Members of the RAISIN group are: Olivia Ali-Brandmeyer (Coordination Centre for Nosocomial Infection Control, Eastern Regions, Nancy, France), Pascal Astagneau (Coordination Centre for Nosocomial Infection Control, Northern Regions, Paris, France), Patrice Baillet (District hospital, Eaubonne, France), Claude Bernet (Coordination Centre for Nosocomial Infection Control, South-Eastern Regions, Lyon, France), Caroline Bervas (Coordination Centre for Nosocomial Infection Control, South-Western Regions, Bordeaux, France), Franck Bruyere (University hospital, Tours, France), Emmanuelle Caillat-Vallet (Coordination Centre for Nosocomial Infection Control, South-Eastern Regions, Lyon, France), Cécilia Campion (Coordination Centre for Nosocomial Infection Control, Northern Regions, Paris, France), Yannick Costa (District hospital, Lagny - Marne-la-Vallée, France), Fabien Daniel (Coordination Centre for Nosocomial Infection Control, Northern Regions, Paris, France), Marine Giard (Coordination Centre for Nosocomial Infection Control, South-Eastern Regions, Lyon, France), Pascal Jarno (Coordination Centre for Nosocomial Infection Control, Western Regions, Rennes, France), Jean Louis Jost (University hospital, Pitié-Salpêtrière Paris, France), François L’Hériteau (Coordination Centre for Nosocomial Infection Control, Northern Regions, Paris, France), Sandra Malavaud (University hospital, Toulouse, France), Véronique Merle (University hospital, Rouen, France), Laurent Merlo (District hospital, Dreux, France), Camille Neels (Coordination Centre for Nosocomial Infection Control, Eastern Regions, Nancy, France), Marion Perennec (Coordination Centre for Nosocomial Infection Control, Western Regions, Rennes, France), Emmanuelle Reyreaud (Coordination Centre for Nosocomial Infection Control, South-Western Regions, Bordeaux, France), Jean-Christophe Séguier (District hospital, Saint-Germain-en-Laye, France), Loïc Simon (Coordination Centre for Nosocomial Infection Control, Eastern Regions, Nancy, France), Jean-Michel Thiolet (Institute for Public Health Surveillance, Saint Maurice, France), Anne-Gaëlle Venier (Coordination Centre for Nosocomial Infection Control, South-Western Regions, Bordeaux, France).
Author Contributions
Conceived and designed the experiments: PJ JFV. Performed the experiments: L. Saunders MPO PJ JFV. Analyzed the data: L. Saunders MPO JFV. Contributed reagents/materials/analysis tools: PJ FLH AGV L. Simon MG JMT. Wrote the paper: L. Saunders PJ JFV. Critical revision of the manuscript for important intellectual content: MPO FLH AGV L. Simon MG JMT.
References
- 1. Astagneau P, Rioux C, Golliot F, Brücker G (2001) Morbidity and mortality associated with surgical site infections: results from the 1997–1999 INCISO surveillance. J Hosp Infect 48: 267–274.
- 2. Kirkland KB, Briggs JP, Trivette SL, Wilkinson WE, Sexton DJ (1999) The impact of surgical-site infections in the 1990s: attributable mortality, excess length of hospitalization, and extra costs. Infect Control Hosp Epidemiol 20: 725–730.
- 3. Mangram AJ, Horan TC, Pearson ML, Silver LC, Jarvis WR (1999) Guideline for Prevention of Surgical Site Infection, 1999. Centers for Disease Control and Prevention (CDC) Hospital Infection Control Practices Advisory Committee. Am J Infect Control 27: 97–132.
- 4. Horan TC, Gaynes RP, Martone WJ, Jarvis WR, Emori TG (1992) CDC definitions of nosocomial surgical site infections, 1992: a modification of CDC definitions of surgical wound infections. Infect Control Hosp Epidemiol 13: 606–608.
- 5. Haley RW, Culver DH, White JW, Morgan WM, Emori TG, et al. (1985) The efficacy of infection surveillance and control programs in preventing nosocomial infections in US hospitals. Am J Epidemiol 121: 182–205.
- 6. Condon RE, Schulte WJ, Malangoni MA, Anderson-Teschendorf MJ (1983) Effectiveness of a surgical wound surveillance program. Arch Surg 118: 303–307.
- 7. Astagneau P, Brücker G (2001) Organization of hospital-acquired infection control in France. J Hosp Infect 47: 84–87.
- 8.
French Ministry of Health (2007) Updating the definition of nosocomial infections [In French]. Available: http://www.sante.gouv.fr/IMG/pdf/rapport_vcourte.pdf. Accessed 2013 Sep 30.
- 9.
Perennec-Olivier M, Jarno P (2012) Surveillance of surgical site infections, France, 2009–2010 [in French]. French Institute for Public Health Surveillance. Available: http://www.invs.sante.fr/Publications-et-outils/Rapports-et-syntheses/Maladies-infectieuses/2012/Surveillance-des-infections-du-site-operatoire-en-France-en-2009-2010. Accessed 2013 Sep 30.
- 10. Culver DH, Horan TC, Gaynes RP, Martone WJ, Jarvis WR, et al. (1991) Surgical wound infection rates by wound class, operative procedure, and patient risk index. National Nosocomial Infections Surveillance System. Am J Med 91 Suppl 2S152–157.
- 11. Haley RW, Culver DH, Morgan WM, White JW, Emori TG, et al. (1985) Identifying patients at high risk of surgical wound infection. A simple multivariate index of patient susceptibility and wound contamination. Am J Epidemiol 121: 206–215.
- 12. Gaynes RP, Culver DH, Horan TC, Edwards JR, Richards C, et al. (2001) Surgical site infection (SSI) rates in the United States, 1992–1998: the National Nosocomial Infections Surveillance System basic SSI risk index. Clin Infect Dis 33 Suppl 2S69–77.
- 13.
Perennec-Olivier M, Jarno P (2012) Surveillance of surgical site infection, France 2011: results [in French]. French Institute for Public Health Surveillance. Available: http://www.invs.sante.fr/Publications-et-outils/Rapports-et-syntheses/Maladies-infectieuses/2012/Surveillance-des-infections-du-site-operatoire-France-2011. Accessed 2013 Sept 30.
- 14. Rioux C, Grandbastien B, Astagneau P (2006) The standardized incidence ratio as a reliable tool for surgical site infection surveillance. Infect Control Hosp Epidemiol 27: 817–824.
- 15.
Goldstein H (2011) Multilevel Statistical Models. 4th ed. Chichester (UK): John Wiley & Sons. 358 p.
- 16.
Leyland AH, Goldstein H (2001) Multilevel Modelling of Health Statistics. 1st ed. Chichester (UK): John Wiley & Sons. 246 p.
- 17. Diez Roux AV, Aiello AE (2005) Multilevel analysis of infectious diseases. J Infect Dis 191 Suppl 1S25–33.
- 18. Astagneau P, L’Hériteau F, Daniel F, Parneix P, Venier AG, et al. (2009) Reducing surgical site infection incidence through a network: results from the French ISO-RAISIN surveillance system. J Hosp Infect 72: 127–134.
- 19.
RAISIN (2010) Surveillance of surgical site infections. French national protocol, 2011 [In French]. French Institute for Public Health Surveillance. Available: http://www.invs.sante.fr/content/download/20596/124273/version/1/file/iso_raisin_protocole_2011.pdf. Accessed 2013 Sep 30.
- 20. Keats AS (1978) The ASA classification of physical status-a recapitulation. Anesthesiology 49: 233–236.
- 21.
Browne WJ (2012) MCMC estimation in MLwiN, v2.26. Bristol (UK): Centre for Multilevel Modelling, University of Bristol. 427 p.
- 22. Merlo J, Chaix B, Ohlsson H, Beckman A, Johnell K, et al. (2006) A brief conceptual tutorial of multilevel analysis in social epidemiology: using measures of clustering in multilevel logistic regression to investigate contextual phenomena. J Epidemiol Community Health 60: 290–297.
- 23. Hanley JA, McNeil BJ (1982) The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology 143: 29–36.
- 24. Hanley JA, McNeil BJ (1983) A method of comparing the areas under receiver operating characteristic curves derived from the same cases. Radiology 148: 839–843.
- 25. Moineddin R, Matheson FI, Glazier RH (2007) A simulation study of sample size for multilevel logistic regression models. BMC Med Res Methodol 7: 34.
- 26. Theall KP, Scribner R, Broyles S, Yu Q, Chotalia J, et al. (2011) Impact of small group size on neighbourhood influences in multilevel models. J Epidemiol Community Health 65: 688–695.
- 27.
European Centre for Disease Prevention and Control (2012) ECDC surveillance report. Surveillance of healthcare-associated infections in Europe, 2007. Available: http://www.ecdc.europa.eu/en/publications/Publications/120215_SUR_HAI_2007.pdf. Accessed 2013 Sep 30.
- 28. Griffiths P, Renz A, Hughes J, Rafferty AM (2009) Impact of organisation and management factors on infection control in hospitals: a scoping review. J Hosp Infect 73: 1–14.
- 29. Venier AG, Gruson D, Lavigne T, Jarno P, L’hériteau F, et al. (2011) Identifying new risk factors for Pseudomonas aeruginosa pneumonia in intensive care units: experience of the French national surveillance, REA-RAISIN. J Hosp Infect 79: 44–48.
- 30. Gbaguidi-Haore H, Cholley P, Rabaud C, Bertrand X, Talon D (2011) Multilevel modelling of the prevalence of hospitalized patients infected with Pseudomonas aeruginosa. Epidemiol Infect 139: 886–894.
- 31. Amadeo B, Dumartin C, Venier AG, Fourrier-Réglat A, Coignard B, et al. (2011) Factors associated with the prevalence of antibiotic use for the treatment of hospital-acquired infections at 393 French hospitals: a regional variation analysis. Infect Control Hosp Epidemiol 32: 155–162.
- 32. Timsit JF, L’Hériteau F, Lepape A, Francais A, Ruckly S, et al. (2012) A multicentre analysis of catheter-related infection based on a hierarchical model. Intensive Care Med 38: 1662–1672.
- 33. Hägi M, Schémann J-F, Mauny F, Momo G, Sacko D, et al. (2010) Active trachoma among children in Mali: clustering and environmental risk factors. PLoS Negl Trop Dis 4: e583.
- 34. Mauny F, Viel JF, Handschumacher P, Sellin B (2004) Multilevel modelling and malaria: a new method for an old disease. Int J Epidemiol 33: 1337–1344.
- 35. Miliani K, L’Hériteau F, Astagneau P (2009) Non-compliance with recommendations for the practice of antibiotic prophylaxis and risk of surgical site infection: results of a multilevel analysis from the INCISO Surveillance Network. J Antimicrob Chemother 64: 1307–1315.