Validation of patient- and GP-reported core sets of quality indicators for older adults with multimorbidity in primary care: results of the cross-sectional observational MULTIqual validation study

We conducted a cross-sectional observational study based on standardised personal interviews with GPs and their patients. Patients were recruited from 35 cooperating GP practices in the cities of Hamburg and Heidelberg and their surrounding areas. The GPs were asked to compile a list of all patients in their practice who met the inclusion criteria. Patients were included if (1) they were aged 65 years or older, and (2) they had at least one consultation in the last completed quarter (i.e. 3-month accounting period) prior to the time of recruitment. From this list, patients were then randomly selected and checked for exclusion criteria until 30 eligible individuals were identified. In seven of the practices, this process was repeated to recruit additional patients.

In our study, multimorbidity was defined by the presence of three or more diseases which are (1) common, (2) chronic, (3) frequently co-occurring with other diseases and (4) potentially affecting subjective health. We operationalised this definition by chronic forms of the diseases anaemia, asthma/chronic obstructive pulmonary disease, atherosclerosis/peripheral arterial occlusive disease, cancer, chronic ischaemic heart disease, chronic low back pain, depression, diabetes mellitus, vertigo, heart failure, osteoarthritis, neuropathy, obesity, osteoporosis, rheumatoid arthritis/chronic polyarthritis and urinary incontinence.

Patients were excluded if (1) they did not meet the criterion for multimorbidity described above; (2) they had been a patient of the practice for less than 12 months or were being treated on behalf of other GPs, e.g. if their practice was currently closed; (3) participation was not recommended for patient safety reasons (according to the GP), e.g. in case of poor health; (4) they lacked capacity to consent; (5) their life expectancy was less than 3 months (according to their GP); (6) they lived in a nursing home; (7) their German language skills were insufficient to participate in the study (according to their GP); (8) they had a severe uncompensated hearing loss and (9) they were participating in other medical studies at the time of recruitment.

Eligible patients received a letter and information material from their GP inviting them to participate in our study. If they were interested, they sent a response letter to the respective study centre. Project staff then contacted the interested patients, explained the study procedure and scheduled an appointment to obtain informed consent and conduct the interview. Recruitment and data collection took place between April 2019 and June 2020.

In standardised in-person and telephone interviews, GPs provided information on their age, sex, professional qualifications and experience, and the size and type of their practice. For participating patients from their practice, GPs provided information on diagnoses and course of treatment in order to calculate 24 quality indicators. In the GP interviews, we also documented whether the patients had participated in disease management programmes (DMPs). DMPs are structured programmes for the long-term outpatient management of chronic diseases such as diabetes, coronary heart disease and breast cancer. They involve managed treatment coordinated by the GP, with regular consultations and a focus on patient education and self-management [25, 26].

Patients were visited at home or in their GP practice and were interviewed face-to-face using standardised questionnaires. The questionnaires collected data on sociodemographic characteristics, self-rated health status and health-related quality of life [27], healthcare utilisation, patient satisfaction and the degree of the patients’ commitment to their GP as coordinator of care [28]. Additionally, to calculate 14 quality indicators, patients reported data on the treatment and its outcomes. The data collection and calculation of the indicators are described in Additional file 1.

The patients’ sociodemographic data included age, sex, education level, their living situation and migration background. The education level was based on their general and vocational education and categorised into three levels according to the Comparative Analysis of Social Mobility in Industrial Nations (CASMIN) classification [29], i.e. (1) uncompleted, general elementary or basic vocational education; (2) secondary school certificate or A-level equivalent and (3) tertiary education. Migration background was assessed by the country of birth of the patients and their parents and coded in three categories, i.e. (1) patient and both parents born in Germany; (2) patient born in Germany and at least one parent born abroad and (3) patient born abroad.

The patients’ self-rated health status was rated using the EuroQoL visual analogue scale (EQ-VAS) with values between 0 (worst health status) and 100 (best possible health status). We also measured health-related quality of life using the five-level version of the EuroQol Five-Dimension Scale (EQ-5D-5L). This questionnaire includes the domains mobility, self-care, usual activities, pain or discomfort and anxiety or depression [27]. We computed the EQ-5D-5L index score based on the German value set. This gives a value of 1.000 for full health, which is reduced by up to five subtractions between − 0.026 and − 0.612 depending on the severity of limitations in each of the five domains [30]. In addition, we calculated a morbidity score comprising the number of permanent diagnoses documented in the GP practice.

Utilisation of primary care was assessed through the number of the patients’ contacts with their GP in the previous 3 months. Patient satisfaction was operationalised by asking if patients would recommend their GP to other patients with chronic conditions, which was rated on a four-point Likert scale (‘definitely yes’, ‘rather yes’, ‘rather no’ and ‘definitely no’) and dichotomised for the analyses (‘yes’ vs. ‘no’).

The patients’ willingness to use the GP as coordinator of their treatment was collected using the Questionnaire on Intensity of the Commitment to the GP (‘Fragebogen zur Intensität der Hausarztbindung (F-HaBi)’). The F-HaBi consists of six items rated on a five-point Likert scale and produces a summary score ranging from 0 to 24 points. Higher scores indicate that the patients are more likely to recognise and use the GP as coordinator of their care. Lower scores indicate that the patients prefer to navigate the healthcare system independently [28].

The recruitment process of study participants is described in Fig. 1. In the participating practices, 1243 eligible patients were contacted for informed consent, 362 patients (29.1%) agreed to participate and 346 could be included in the analyses. The median cluster size was 8 patients per practice (interquartile range: 6 to 13 patients).

Participating GPs had a median age of 57 (IQR 50 to 60) years, and 54.3% were women. GPs had been practising for a median of 20 (IQR 12 to 26) years. More than half of the GP population (57.1%) worked in individual practices, 5.7% in group practices where all physicians have their own patient base, 31.4% in joint practices where all physicians share the same patient base and 5.7% were employed or self-employed in medical care centres. The median number of physicians in the participating practices was 2 (IQR 1–3). In 17.7% of the practices, fewer than 750 patients per quarter were treated, 23.5% of the practices treated 750 to 1000 patients and 58.8% treated 1000 patients per quarter or more.

The patient population is described in Table 1. The patients had a median age of 78 (IQR 72–83) years, and 55.2% were women. More than one third of the patients (35.8%) were living alone. Most patients (56.1%) had uncompleted, general elementary or basic vocational education, and almost nine out of ten patients (88.0%) were born in Germany and had parents, which were also born in Germany. The median number of chronic conditions was 10 (IQR 7–15). The median EQ-5D-5L score was 0.84 (IQR 0.62–0.94) points, and patients rated their health with a median of 65 (IQR 50–80) points. Patients had an average of 2 (IQR 1–3) contacts with their GP in the previous 3 months, 44.6% participated in disease management programmes and nearly nine in ten (89.6%) would recommend their GP to other patients. The median willingness to use their GP as coordinator of treatment was 22 (IQR 19–24) points in the F-HaBi score. As shown in Table 2, the most prevalent diseases in our sample were hypertension (68.2%), chronic low back pain (59.2%) and osteoarthritis (47.3%).

The stepwise backward selection to define the core sets of quality indicators is detailed in the Tables 3 and 4. Twelve GP-assessed quality indicators and seven patient-assessed quality indicators were excluded. In the previous stages of the project, a measurement framework for healthcare quality [23] had been proposed. Table 5 shows that all three levels of healthcare and all nine care domains of the complete indicator set are represented by both core sets combined. The GP-reported indicators cover eight domains and the patient-reported indicators cover five domains. The final questionnaires for both core sets can be found in Additional file 2.

The characteristics of the core quality indicators are presented in Table 6. The GP-reported indicators had an item-total correlation between 0.332 and 0.576 and the patient-reported indicators between 0.339 and 0.440. Both exploratory factor analyses resulted in one extracted factor with an eigenvalue of 3.27 and 1.33, respectively. The core quality indicators had loadings between 0.416 and 0.673, and 0.311 and 0.545, respectively. The Kaiser–Meyer–Olkin measure of sampling adequacy was 0.774 and 0.758, respectively. We determined a Cronbach’s alpha of 0.806 and 0.628, respectively, for the selected core indicator sets.

Overall, the fulfilments rate of the indicators ranged from 24.6 to 89.0%. Fourteen quality indicators had floor effects between 0 and 14.7%, and the others were at 28.6%, 35.3% and 41.2%. Eleven indicators had ceiling effects between 0 and 11.8%, five were between 17.1 and 34.3% and one at 91.4%. For the ten analysed indicators, documentation rates ranged from 50.3 to 73.0%. Between 0 and 12.7% of the values were missing. With positive agreement rates between 33.3 and 75.9% and negative agreement rates between 24.1 and 61.8%, seventeen of the analysed indicators showed low to moderate agreement between patients and professionals. One indicator had 90.7% positive agreement and 5.1% negative agreement.

The results of the multivariable analyses of the associations between patient characteristics and GP- and patient-reported quality scores are shown in the Tables 7 and 8. The GP-reported quality score was lower when patients were born abroad (− 1.04, 95% CI − 2.00/ − 0.08, p = 0.033) and when they had higher health-related quality of life (− 1.37 per point in the EQ-5D-5L score, 95% CI − 2.39/ − 0.36, p = 0.008). The quality score was higher when the patients had more contacts with their GP (0.14 per contact, 95% CI 0.04/0.23, p = 0.007) and when they were more willing to use the GP as coordinator of treatment (0.13 per point F-HaBi score, 95% CI 0.06/0.20, p < 0.001). The patient-reported quality score was higher when patients visited their GP more often (0.12 per contact, 95% CI 0.04/0.20, p = 0.006) and when they had a higher level of commitment to their GP (0.16 per point, 95% CI 0.10/0.21, p < 0.001).

To our knowledge, the MULTIqual project is the first study to develop and validate quality indicators for the primary care of patients with multimorbidity in a systematic, multi-step approach. In this study, we selected core sets of twelve GP-reported and seven patient-reported quality indicators that represented all nine care quality dimensions of the complete indicator set, demonstrated good internal consistency and robust structural and construct validity and can be collected through new or already existing GP and patient surveys. Depending on access to data sources, either patient-reported or GP-reported—or both indicator core sets—can be used, allowing for broader application of the indicators. The core sets provide viable alternatives to the untested set of indicators, as the size of the set and the cost of measurement are also important considerations for implementation [38‐40].

Following a well-established methodology, our quality indicators were identified through a multistep process that combined available evidence and expert consensus [41], represent multiple systematically selected domains of care specific for populations with multimorbidity and are therefore a more valid alternative to the fragmented and disease-specific quality assessment through existing patient-reported experience and outcome measures [15] such as or the European Task Force on Patient Evaluation of General Practice Care questionnaire (EUROPEP) [42] or EQ-5D [27]. As the indicators were developed based on literature review and expert opinions without connection to a specific disease spectrum [23], our indicators are per se generic and equally applicable to all multimorbidity constellations that impact subjective health status.

Data were analysed using multivariable analyses, adjusted for potential confounders, and multilevel models, allowing for cluster effects. However, the core sets of quality indicators were selected by a backward selection algorithm, which is known to be sensitive to differences in the distribution of the included variables. As a result, the identified core sets represent coherent sets of quality indicators, but not necessarily the best possible selection. It is important to note that the data in our study were collected via self-report, which may introduce recall problems, errors and social desirability. The study design was cross-sectional, which means that the direction of associations cannot be determined, i.e. whether quality scores influence quality of life or vice versa.

The response rate for this study was a low (29.1%), possibly limiting its representativeness as certain groups (e.g. male, younger, less educated and less healthy living patients) are often underrepresented in low-response samples. Despite these differences in descriptive data between samples and the population, there is usually little effect of low response rates on the reported associations in the dataset [43‐46]. For patient safety reasons, we had to exclude patients in poor health, which narrows our construct of multimorbidity. Furthermore, only diseases that are frequently co-occurring with other diseases [47, 48] were defined as inclusion criteria. However, many common diseases that do not fall within this definition are still prevalent in our sample such as chronic gastritis/gastroesophageal reflux disease (24.6%) or liver disease (6.4%), as patients with additional conditions were not excluded.

Another potential bias in our study sample is that participating GPs are likely to be highly motivated and interested in the topic. The quality of primary care for patients with multimorbidity might therefore be overestimated. Moreover, the study was conducted in major German cities with a high density of healthcare providers. The average population of German GPs is slightly younger than our study sample (55 vs. 57 years), the proportion of women is slightly lower (49% vs. 54%) and on average practices are smaller (average of 850 patients per quarter vs. 59% treating 1000 patients or more [49]). Therefore, caution should be taken when generalising the results to medically deprived areas.

Finally, it should be mentioned that this study was observational and had multiple outcomes without prior sample size calculation. Due to the relatively small sample size of 346 patients from 35 practices, predictors of reduced healthcare quality may have been missed due to limited statistical power.

Pilot testing of quality indicators for primary care and community settings is rarely reported. Consequently, well-defined criteria and standards for empirical validation are lacking [50, 51]. In Germany, testing of quality indicators is mainly carried out by central organisations commissioned by the Federal Joint Committee [21]. This makes the application and validation of core sets by independent researchers and health experts an innovative component of the MULTIqual project.

Previous studies have shown that quality of care increases with the number of diagnoses when using disease-specific indicators, particularly in concordant conditions with similar pathophysiological profiles and disease management [52‐54]. However, patient safety and patient-centred outcomes have been found to be negatively associated with the number and severity of conditions [54, 55]. There is evidence linking higher severity of comorbidities with higher quality of care according to process measures. This is consistent with our findings of an inverse relationship between quality of life and quality of care per GP-reported measures [54]. Zulman et al. [56] hypothesise that higher healthcare utilisation by patients, e.g. due to clinical complexity and reduced health status, leads to more intensive monitoring, more frequent assessment of healthcare needs and subsequent adjustments to their treatment.

In our study, quality scores improve with commitment to the GP. In Germany, which does have a compulsory primary care system and allows free choice of healthcare provider, this relationship is based solely on mutual trust and voluntariness [57]. However, we did not find evidence supporting the link between participation in DMPs and improvements in care structure and processes [25, 58, 59], although with enrolment in a DMP, some of the criteria measured by the indicator sets should already become an integral part of the care regimen.

The results of the pilot study demonstrated that the core sets can be a useful tool for the identification of areas in primary care with potential for improvement. Although many researchers advocate for patient-centred care in the context of multimorbidity [60‐62], treatment goals or patient preferences were established in less than half of all cases. These findings suggest that patient-centred care planning is not yet fully realised. Tinetti et al. [63] were able to show that aligning care with patient preferences led to a reduction in unwanted treatments, medications and diagnostic tests. Widespread adoption of these principles could potentially have a similar impact on the German healthcare system, where patients with multimorbidity incur significant healthcare utilisation due to the lack of gatekeeping in primary care [4, 64].

Indicators of process quality are most useful for quality improvement purposes as they more directly reflect changes in practice [65, 66]. Moreover, our findings may guide the future development of electronic documentation systems, ultimately seeking to improve documentation quality and enable quality monitoring through built-in performance measurement [67]. In Germany, the digital transformation of GP practices is still in its early stages, so that despite major barriers to this development, further progress in current documentation standards can be expected in the coming years [68, 69].

While the development of the candidate indicator was informed by international evidence—most notably the multimorbidity guideline by the UK National Institute for Health and Care Excellence [70], the German College of General Practitioners [71] and the American Geriatrics Society [72]—evaluation and consensus of the indicators was obtained by a German expert panel and is thus geared to the specifics of the German healthcare system [23]. Therefore, in principle, the indicators are internationally relevant and transferable to other healthcare systems. Nevertheless, it will be necessary to adapt indicator descriptions and modes of data collection. In particular, it should be examined if easily accessible data can be used as data sources for quality indicators [73, 74], e.g. standardised documentation in medical records in the UK [75].

Longitudinal studies are required to examine the responsiveness of quality scores to change, costs and potential unintended consequences, as well as long-term benefits resulting from the implementation of these quality indicators. This should be done by conducting a cost-utility analysis and measuring changes in indicator scores over time in relation to health outcomes [21, 76, 77]. Unfortunately, there is still no robust evidence of the benefits of using quality indicators. However, improvements in care processes have been achieved by creating the conditions for the implementation of indicators, including increased use of digital solutions, prompts, recall systems and better documentation [78]. In light of future advances in multimorbidity research and corresponding changes in guideline recommendations, the indicators should be regularly updated to best reflect current evidence [79].