Development and feasibility of a set of quality indicators relative to the timeliness and organisation of care for new breast cancer patients undergoing surgery

The task of developing QIs for breast cancer management was delegated by the French authorities to the research project COMPAQH (COordination for Measuring Performance and Assuring Quality in Hospitals). COMPAQH’s remit is to develop QIs for monitoring quality in French hospitals and to design ranking methods and pay-for-quality programmes [20]. The project is run by INSERM (French National Institute for Health and Medical Research) and is sponsored by the French Ministry of Health and HAS which, together in 2003, listed 9 priority areas in need of quality improvement: pain management, practice guidelines, human resources management, iatrogenic events, nutritional disorders, access to care, taking account of patients’ views, coordination of care, and continuity of care. Quality of breast cancer care comes under the topic “practice guidelines” [21, 22].

QI development in breast cancer care began in 2007 as a partnership between COMPAQH, the French National Federation of Cancer Centres (FNLCC), HAS, INCa, the Société Française de sénologie et de pathologie mammaire (SFSPM) and the Collège National des Gynécologues et Obstétriciens français (CNGOF). Attention was focussed on “new patients with infiltrating, non-inflammatory and metastasis-free breast cancer undergoing surgery within the institution” as this was the main concern of experts in the field and of consulting physicians. “New patients” were women with unilateral breast cancer who had never undergone treatment for breast disease, and who had not yet been seen in consultation or been admitted to the hospital for breast disease. We chose “infiltrating, non-inflammatory breast cancer” as this is the most common type of breast cancer (about 75 % of all cases) and constitutes a homogeneous population. Surgery is the primary treatment for most patients with metastasis-free T0 to T2 tumours.

COMPAQ-HPST has a unique methodology for developing QIs [20, 23, 24] (Figure 1). The QIs for this study were designed with the help of experts in the field of breast cancer, submitted to SFSPM and CNGOF for their opinion, and selected by a working group of experts and consulting physicians in breast cancer. A list of criteria and items was drawn up for each QI. It was based on French practice guidelines for breast cancer management, legal regulations, and consensus-based guidance [21, 22, 25]. Factors influencing choice of criteria were: high level of evidence (level 1 whenever possible); discriminatory power (in view of variations observed in practice); practices in areas that were not subject to rapid change and where considerable improvement could be expected; applicability to all the hospitals in our sample; a precise working definition of the criterion that could be shared by all; and easy, standardised data collection. The number of criteria was restricted to 8 in order to lighten the burden of the participating hospitals. These QIs covered the care process as best possible.

Two tests were performed, first a preliminary test of QI feasibility in a small number of hospitals, then a larger scale test to measure QI performance.

For the preliminary test, performed in 2008, we asked 23 hospitals performing breast cancer surgery (including 20 comprehensive cancer centres) whether they would be willing to test the 8 QIs using their data for 2006. They assessed QI feasibility using a validated grid of 12 items exploring 5 dimensions (QI acceptability by staff, their understanding of the QI, their availability to respond in the allotted time, the ability of the hospital and the IT system to collect and handle the necessary data, and the workload) [26]. At the end of the test, we also assessed QI relevance as given by inter-hospital variability and deviation from expected performance.

For the second test (July–October 2009), we approached, via French hospital federations, all 633 hospitals performing breast cancer surgery in France (351 public, 231 private not-for-profit, and 51 private profit-making organisations). We assessed inter-hospital variability, internal validity (i.e. whether the QI really measured what it was intended to measure both from qualitative and quantitative points of view), and inter-observer reproducibility. Assessment was double-blind on 20 medical records from each of 14 hospitals. None of the QIs required adjustment.

For each test in each hospital, 80 patient records were analysed manually. This number was a compromise between acceptable workload and statistical validity [27, 28]. The records were selected randomly from the PMSI database (Programme de médicalisation des systèmes d’information) which reports diagnosis-related group (DRG) statistics in a public hospital setting. The selected DRG code was breast cancer surgery. Apart from the 80 records for analysis, 20 additional records were also selected in the event of exclusions. Each hospital received an explanatory guide on the 8 QIs and a grid with instructions for its completion (available in French at http://​www.​compaqhpst.​fr/​data/​indicateurs/​12_​GYC_​V2_​Grille_​de_​recueil_​images.​pdf).

For any given QI, at least 30 completed grids were required per hospital to support the assumption of a Gaussian distribution and to compute confidence intervals. This meant that, for any given QI, only those hospitals with at least 30 medical records for this QI were entered into the comparison among hospitals. Inter-hospital variability was given by QI score variance and the Gini coefficient which measures score dispersion. Variability (i.e. discriminatory power) is high if the Gini coefficient is under 0.2 and low if it is above 0.5 [29]. Internal validity was given by the overall concordance rate and inter-observer reproducibility by the Kappa coefficient [30, 31]. We used SAS version 8.1 software (SAS Institute Inc, Cary, NC).

The working group established a list of 8 QIs: waiting time to first appointment with surgeon, time to surgery, time to postoperative MDTM, waiting time to appointment after surgery, time to first postoperative treatment, patient information, traceability of information relating to prognosis, and conformity of postoperative MTDM (Table 1).

All the 23 hospitals (3 public, 20 cancer centres) taking part in the feasibility test completed and returned the grids. They randomly selected 2044 medical records: inclusion criteria were not met in 274 records, so 1770 (72 %) were included in our analysis. All QIs, except QI 1 (time to first appointment with surgeon), showed fair feasibility and high inter-hospital variability.

QI 1 presented ambiguities with regard to wording. Hospitals understood “time to first appointment” (QI 1) differently. Some hospitals thought that it was the date the patient’s medical record had been created, some that it was the date when the patient had called the hospital for an appointment, and some that it was the date on the GP’s letter requesting an appointment for the patient. Because of this ambiguity, QI 1 was reserved for the hospital’s information and was excluded from the hospital comparisons below.

Of the 633 hospitals performing breast cancer surgery, 70 accepted to participate in the testing of QI performance. Of the 70 participating hospitals, 60 completed and returned the grids (28 public, 10 private, and 22 not-for-profit privately owned organisations which included 20 cancer centres) (Figure 2). The reasons for the 10 drop-outs are given in the footnote to the flowchart in Figure 2. The 60 hospitals randomly selected 5215 medical records but we retained only those (n = 5043) that came from the 54 hospitals that had at least 30 records for at least one QI. From 6 to 21 hospitals, depending upon the QI, did not reach the required threshold of at least 30 medical records in order to be included in hospital comparisons.

The 54 hospitals were situated in different regions of France and differed in their status (public/private) and number of beds. All except one met the threshold volume of activity (> 30 breast cancer surgeries/year) that is required by French health authorities. The breakdown according to annual volume of activity was as follows: 28–80 operations (n = 6 hospitals), 122–197 (n = 11), 205–377 (n = 13), 476–945 (n = 18), 1036–1873 (n = 6).

We analysed 3624/5043 records (72 %) from the 54 hospitals. The main reasons for exclusions are given in the footnote to Figure 2. The incidence of missing data was by decreasing rank order: 40.6 % (1471/2153) for date of adjuvant therapy, 26.5 % (960/2664) for date of the postoperative appointment, 11.1 % (402/3222) for date of the postoperative MDTM, 8.8 % (319/3305) for the first appointment with the surgeon, and 0.4 % (9/3615) for the date of surgery.

QI 2 (“time to surgery”) was subject to misinterpretation: it was not clear whether it referred to (i) the appointment when the decision to perform surgery was taken or (ii) the appointment when the surgeon diagnosed suspected cancer and ordered tests before deciding to operate. Q2 was used, without modification, in the hospital comparisons.

Table 2 gives for each QI the number of hospitals included in hospital comparisons, the mean score (i.e. mean percentage of patients with medical records that met the criteria outlined in Table 1) with ranges and standard deviations, and the Gini coefficient which measures dispersion. Scores varied widely, revealing substantial room for improvement in hospitals for all QIs, in particular QI 6 (patient information) and QI 8 (conformity of postoperative MTDM), which had the lowest mean scores. The Gini coefficient was <0.3 for all QIs (except QI 6 and QI 8) indicating that the power of the QIs to discriminate among hospitals was high (Figure 3). In the internal validity test, the overall discordance rate (lack of reproducibility) was 6.7 % (2.5–13.6). The Kappa coefficient score of 0.66 was indicative of fairly strong inter-observer reliability.