Background
Methods
Data sources
Composite index development approach
STEPS | ||
1. Theoretical Framework: Quality Matrix | ||
2. Metric Selection: Literature Review/Expert Opinion | ||
3. Missing data imputation: imputation by mode for binary variables or mean for continuous variables | ||
4. Initial Data Analysis: Review outliers/directionality | ||
Indicators within cells | Base Case | Alternative |
5a. Normalization: | Binary categorization of non-binary cell indicators | A. Rescaling of non-binary cell indicators (Min-max) |
5b. Weighting: | Equal weighting | |
5c. Aggregation: | Additive linear aggregation of indicator scores | B. Geometric aggregation of indicator scores |
Cells within matrix | Base Case | Alternative |
6a. Normalization: | Rescaling of cell scores (Min-max) | C. Standardization of cell scores (Z-scores) |
6b. Weighting: | Equal weighting | |
6c. Aggregation: | Additive linear aggregation of cell scores | D. Geometric aggregation of cell scores |
7. Uncertainty/Sensitivity Analysis: comparison base case against alternative methods | ||
8. Deconstruction: explore individual indicators contribution to composite score |
Conceptual framework
EFFECTIVE | EFFICIENT | ACCESSIBLE/ TIMELY | PATIENT-CENTERED/ ACCEPTABLE | EQUITABLE | SAFE | |
---|---|---|---|---|---|---|
health care that is adherent to evidence based guidelines resulting in improved health outcomes | delivering health care which maximizes resource use and avoids waste | health care that is timely, geographically reasonable, with appropriate skills and resources | health care that takes into account preferences of service users and community culture | health care quality does not because of personal characteristics such as gender, race, or socioeconomic status; | delivering health care which minimizes risks and harm to service users. | |
STRUCTURE | ||||||
PROCESS | ||||||
OUTCOME |
Metric selection
Imputation of missing data
Initial data analysis
Indicators within cells: normalization, weighting and aggregation
Cells within matrix: normalization, weighting, and aggregation
Uncertainty and sensitivity analysis
Deconstruction
Results
Indicator selection
Uncertainty and sensitivity analysis
-
Base vs. alternative A: Rescaled ordinal or continuous variables are compared to binary variables (base case). Alternative A scores are slightly more condensed and therefore show less variation across facilities. This method reduces the extreme ends of the scale and extremely well performing facilities are no longer seen as potential outliers. Despite the narrow distribution of values in the underlying data, the variation in values remains minimal. This method showed no significant outliers in facility rankings compared to the base case (Fig. 1).
-
Base vs. alternative B: Additive aggregation (base case) is compared to geometric aggregation of binary indicators into cell scores. With the same maximum possible points as the base case, in the absence of perfect quality, geometric aggregation leads to substantially lower scores in each cell than additive aggregation, and, therefore to lower total scores. In addition, this method created outliers in facility rankings compared to the base case. This can be seen best with Facility N (Fig. 1), which ranked 14th in the base case, but dropped to 23rd due to not meeting all indicators in 9 of the 12 cells. Although this facility obtained some or most of the indicators in each cell, it did not obtain all indicators in the majority of cells, which caused this facility to be most affected by this method.
-
Base vs. alternative C: Rescaled cell scores using the Min-max method (base case) is compared to standardization using Z-scores for each cell, which expands the underlying scale to a normal distribution curve. Although this allows for easier identification of exceptionally good and poor performing facilities, it is a relative metric, only allowing for comparison of facilities against each other, but does not give the user a way to easily identify how well the facilities are performing in absolute terms (e.g. against some standard, against another sample). This method also shows no significant outliers in facility rankings compared to the base case.
-
Base vs. alternative D: Additive aggregation of the cell scores (base case) is compared to geometric aggregation into a composite score. The maximum scale range is now 0–1 with almost all facilities identified as poor performers. This method is extremely sensitive to a low performance in a given cell score and therefore does not allow for much differentiation between facilities. This method also showed a significant difference in facility rankings compared to the base case most notable for facility P, which was ranked 16th in the base case, but drops to 26th when using the geometric aggregation at the cell level. This facility scored “0” in the patient-centered structure cell, which then resulted in a total score of “0” using this method.
Facility | Base Case | A. Min-max Normalization (indicators) | B. Geometric Aggregation (indicators) | C. Z-score Standardization (cells) | D. Geometric Aggregation (cells) |
---|---|---|---|---|---|
Scale Range | 0–12 | 0–12 | 0–12 | not applicable | 0–1 |
A | 11.07 | 9.81 | 2.37 | 11.20 | 0.33 |
B | 10.67 | 9.59 | 2.13 | 7.97 | 0.22 |
C | 10.50 | 9.36 | 1.92 | 8.27 | 0.13 |
D | 10.03 | 8.89 | 1.92 | 8.27 | 0.13 |
E | 9.88 | 8.96 | 2.00 | 5.13 | 0.07 |
F | 9.83 | 8.75 | 1.79 | 5.39 | 0.06 |
G | 9.68 | 9.43 | 1.46 | 4.49 | 0.05 |
H | 9.37 | 8.28 | 1.79 | 4.10 | 0.02 |
I | 9.36 | 8.15 | 1.13 | 2.90 | 0.04 |
J | 9.21 | 8.74 | 1.29 | 3.75 | 0.02 |
K | 9.14 | 7.55 | 1.67 | 1.50 | 0.02 |
L | 8.99 | 8.43 | 1.00 | 1.99 | 0.02 |
M | 8.84 | 7.51 | 1.17 | 1.64 | 0.01 |
N | 8.50 | 7.50 | 0.79 | −2.35 | 0.01 |
O | 8.48 | 7.51 | 1.17 | −1.46 | 0.01 |
P | 8.28 | 7.83 | 1.29 | −1.72 | 0.00 |
Q | 8.24 | 7.37 | 1.13 | −2.77 | 0.01 |
R | 8.12 | 7.52 | 1.00 | −2.43 | 0.00 |
S | 8.10 | 7.63 | 1.00 | −2.76 | 0.00 |
T | 7.85 | 7.16 | 1.50 | −3.80 | 0.00 |
U | 7.52 | 7.37 | 0.70 | −4.79 | 0.00 |
V | 7.44 | 6.79 | 1.00 | −5.77 | 0.00 |
W | 7.39 | 7.39 | 0.50 | −7.96 | 0.00 |
X | 7.16 | 6.68 | 1.00 | −9.16 | 0.00 |
Y | 7.11 | 6.70 | 1.00 | −8.40 | 0.00 |
Z | 6.45 | 5.67 | 0.50 | −9.99 | 0.00 |
Base Case | |
---|---|
A. Min-Max Normalization (indicators) | .94 |
B. Geometric Aggregation (indicators) | .90 |
C. Z-score Normalization (cells) | .99 |
D. Geometric Aggregation (cells) | .96 |
Base Case Mean (CI) | Alternative A Mean (CI) | Alternative B Mean (CI) | Alternative C Mean | Alternative D Mean (CI) | |
---|---|---|---|---|---|
Structure | |||||
Effective | .80 (.73–.87) | .80 (.75–.85) | .27 (.09–.45) | 0 | .80 (.73–.87) |
Accessible | .80 (.75–.84) | .73 (.69–.78) | .04 (.04–.12) | 0 | .80 (.75–.84) |
Patient-centered | .60 (.48–.70) | .59 (.48–.70) | .12 (.02–.25) | 0 | .60 (.48–.70) |
Safe | .75 (.67–83) | .70 (.62–.78) | .19 (.03–.35) | 0 | .75 (.67–83) |
Process | |||||
Effective | .55 (.48–.62) | .51 (.45–.57) | 0 | 0 | .55 (.48–.62) |
Accessible | .82 (.75–.89) | .70 (.64–.75) | .38 (.18–.59) | 0 | .82 (.75–.89) |
Patient-centered | .62 (.53–.70) | .44 (.38–.52) | .04 (.04–.12) | 0 | .62 (.53–.70) |
Safe | .68 (.58–.77) | .63 (.54–.73) | .15 (.06–.30) | 0 | .68 (.58–.77) |
Outcome | |||||
Effective | .94 (.88–1.00) | .84 (.80–.89) | .88 (.75–1.02) | 0 | .94 (.88–1.00) |
Accessible | .54 (.44–.64) | .46 (.39–.54) | .15 (.01–.30) | 0 | .54 (.44–.64) |
Patient-centered | .90 (.82–.99) | .88 (.81–.95) | .81 (.65–.97) | 0 | .90 (.82–.99) |
Safe | .75 (.65–.85) | .68 (.60–.76) | .50 (.29–.71) | 0 | .75 (.65–.85) |
Total Scores | |||||
Composite Total | 8.74 (8.25–9.23) (range 0–12) | 7.97 (7.56–8.38) (range 0–12) | 3.54 (2.84–4.24) (range 0–12) | 0 | .043 (0.01–0.07) (range 0–1) |
Structure Total | 2.94 (2.73–3.14) (range 0–4) | 2.82 (2.63–3.02) (range 0–4) | .62 (.31–.92) (range 0–4) | 0 | 2.94 (2.73–3.14) (range 0–4) |
Process Total | 2.67 (2.42–2.91) (range 0–4) | 2.28 (2.05–2.51) (range 0–4) | .58 (.27–.88) (range 0–4) | 0 | 2.67 (2.42–2.91) (range 0–4) |
Outcome Total | 3.14 (2.94–3.33) (range 0–4) | 2.97 (2.72–3.01) (range 0–4) | 2.35 (1.99–2.71) (range 0–4) | 0 | 3.14 (2.94–3.33) (range 0–4) |