Testing a European set of indicators for the evaluation of the management of primary care practices

Abstract

Background. Effective practice management is an important prerequisite for offering good clinical care. Internationally valid, reliable and feasible indicators and instruments are needed to describe and compare the management of primary care practices in Europe.

Objective. This paper describes development and evaluation of the European Practice Assessment instrument and indicators (Engels Y, Campbell S, Dautzenberg M et al. Developing a framework of, and quality indicators for, general practice management in Europe. Fam Pract 2005; 22(2): 215–22).

Methods. The study design was a validation and feasibility study set in 273 general practices in Austria, Belgium, France, Germany, Israel, The Netherlands, Slovenia, Switzerland and the UK. Use was made of a set of 62 valid quality indicators derived previously from an international Delphi procedure. The EPA instrument, based on this set of indicators, was used to collect data in the 273 practices. This instrument consists of self-completed questionnaires for doctors, staff managers and patients. In addition, there is an interview schedule for use by an outreach visitor, to be held with the lead GP or manager, and a visitor checklist. The instrument was analysed using expert review by the project partners, factor and reliability analyses, ANOVA analyses and by determining intraclass correlations.

Results. Fifty-seven indicators were found to be valid, feasible, reliable and discriminative in all participating countries. The instrument was able to determine differences in practice management within and between countries. All (but one) practices completed the assessment procedure. The data collection method appeared to be feasible, although some aspects can be improved.

Conclusion. The EPA instrument provides feedback to practices that facilitates quality improvement and can compare primary care practices on a national and an international level.

Introduction

Good management of primary health care is a prerequisite for patients to receive the clinical care they need.^1,2 In addition, patients value good access, patient involvement and sufficient consultation time.^3,4 Information on practice management, collected with valid and reliable indicators and instruments, would enable consumers and providers of care to assess and improve practice management and to compare practices. However, practice management indicators to be used for research and assessment of primary care are scarce.⁵ A valid and reliable method for assessing practice management demands good coverage with quality indicators in all relevant domains and dimensions. As practice management is multidimensional and includes both structures and processes, this is a complex task.

Now that Europe gradually moves towards a situation characterised by care provision that does not recognise borders, it is important to understand differences and similarities of family practice management across Europe.⁶ In each country, the health care system is different and influenced by historical, political and cultural developments. Although several sets of indicators are available in Europe, they were all developed within specific health care systems and can not automatically be transferred between countries.⁷ It is for this reason that we aimed to develop a European set of indicators and an instrument to assess and improve practice management.

The basis for this European Practice Assessment (EPA) instrument was a framework for primary care practice management consisting of 171 quality indicators.⁸ Sixty-two of these indicators were rated face valid in a Delphi procedure with expert panels in Belgium, France, Germany, The Netherlands, Switzerland and the UK.⁹

In this paper, we present evidence of the validity and reliability of the EPA instrument.¹⁰

Methods

Development of the instrument

The instrument was developed between 2001 and 2004 by the project partners of the EPA-project, all researchers and experts in the field of quality in primary care from six European countries, i.e. Belgium, France, Germany, The Netherlands, Switzerland and the UK (Appendix 1). Practice management was defined as: ‘systems, structures and processes aimed to enable the delivery of good quality patient care’, excluding clinical processes and clinical outcomes.^1,11 A framework of 171 indicators within five domains was developed consisting of: (1) infrastructure; (2) information; (3) people; (4) finance; and (5) quality and safety. These indicators were rated on usefulness and clarity in a two-round written postal Delphi study with expert panels in the six participating countries. The result was a set of 62 indicators that were rated as useful for quality assessment by all six panels.^8,9 After consulting with the project partners by e-mail and in workshops, this set of indicators was operationalized into 202 questions and allocated to specific questionnaires for the principal GP or practice manager, for all GPs and for all staff respectively. In addition, 30 patients per practice completed the EUROPEP patient questionnaire. EUROPEP is an internationally validated instrument reflecting a set of indicators for patients to evaluate the quality of primary care.¹² An observer checklist and an interview schedule for the principal GP or PM were developed for use by an external visitor during a practice visit (Box 1). The resulting EPA instrument was developed in English and translated into relevant languages by at least two independent translators per country (an EPA partner in cooperation with an independent translator).

Design and research population

We used a cross-sectional design to test the instrument and indicators in nine European countries (Belgium, France, Germany, The Netherlands, Switzerland and the UK and also Austria, Israel and Slovenia). We aimed at a convenience sample of 30 primary care practices in each country with equal numbers of single-handed, dual and group practices, and an equal distribution between practices in rural and in urban areas. Practices were recruited from the network of the partners.

Procedure

The number of visitors for the practice visits varied from 2 to 5 per country. They had different backgrounds (GPs, practice assistants, nurses, social scientists and medical students), but they were all familiar with practice procedures in primary care and had good interpersonal skills. They received training from the EPA-partners in their country to use the instrument and received a visitor manual with written instructions.

The data collection procedure was based on a successful Dutch procedure.¹¹ The first step was to contact the practice, to explain the purpose and procedure of the assessment and to send standardized introductory letters. Next, the visitor made contact by telephone to set a date for the practice visit. Afterwards, a standardized confirmation letter was sent with a questionnaire for each staff member, each individual doctor, the practice manager, as well as with patient questionnaires. In order to get data for at least 30 patients per practice, 50 questionnaires were sent. The practices were asked to send the completed questionnaires back to the visitor. In some countries, the visitor entered all data or of the patient evaluations in a spreadsheet program and made a summary for a feedback session.

Next, the practice visit took place, taking 3–6 hours on one day. In the morning, the observer checklist was completed, after which an interview took place with the GP with management responsibilities or the practice manager. In some countries, the practice visit was completed with a feedback session on the preliminary outcomes of the assessment procedure. Since it was a study to test the instrument and procedure, national and international reference data were not available yet, except for data on patient evaluations. Therefore, the content of the feedback was not standardized.

The complete package was first pilot tested in 3–5 practices per country in Autumn 2003 and revised, followed by data collection in over 270 practices in nine European countries (about thirty practices per country). Data collection took place between January and June 2004. Figure 1 summarises the assessment procedure, from contacting the practice until the practice visit.

Analyses

After data collection each question was reviewed using comments from the practice visitors. 56 questions were excluded as they were open for misinterpretation or already covered by other questions.

Frequency distributions and descriptive statistics were computed for all remaining questions across all countries and for each country individually. Questions were omitted from analyses if there was more than 5% missing data¹³ (n = 2) or if they were met positively by more than 95% of the practices in each country indicating an inability to discriminate (n = 2).

Factor and reliability analysis were used to reduce the remaining 142 questions to a fewer number of indicators. If indicators were composed of more than two questions, only questions with a factor loading of 0.4 in a principal compound analysis or more were included. The reliability of an indicator with multiple questions was regarded as acceptable if Cronbach's alpha was 0.60 or above.¹⁴

The result was an instrument with a set of 57 indicators, considered valid, reliable, feasible and discriminative. These indicators were spread across 26 of the 32 dimensions of the five domains of the theoretical framework (Table 2).

To determine discriminative power, we computed means and standard deviations of all 57 indicators per country and of the whole sample, and performed variance analyses between countries (ANOVA). To assess the proportion of variance that is accounted for by the country, we computed intraclass correlations (ICC) by GLM (variance components). Data from the staff and patient evaluations were analysed using GLM multilevel variance analysis.

Finally, to find out whether the data collection procedure was feasible or not, we asked the partners and the visitors, after all data were collected, to complete a semi-structured written questionnaire to describe whether practice staff found the EPA procedure acceptable, whether the procedure was performed as planned, and which problems they encountered. We also asked them to mention factors that contributed to the success or failure of the data collection. Two researchers (YE and MD) analysed these process evaluations.

Results

In the nine participating countries, between 27 (UK) and 33 (Austria) practices took part (273 in total).

Only Slovenia and The Netherlands managed to recruit about the same number of single-handed, duo and group practices, as well as an equal division between rural and urban practices. Belgium and Germany had an overrepresentation of duo practices; France, the UK and Israel of group practices. Austria only had single-handed practices (Table 1).

In Belgium, France, Germany, Switzerland and Israel the samples deviated from the national situation with too few single-handed practices, whereas in Slovenia there are too many single-handed practices.

The mean number of GPs per practice varied from 1.1 (Austria) to 3.5 (UK) and the mean number of staff questioned varied from 1.7 (Belgium) to 11.3 (UK). Most countries had a sample of about 30 patients who filled in the patient questionnaire, except for Germany (68), Switzerland (40) and Austria (66), that questioned more patients per practice.

Intra- and inter-country comparisons

All but four indicators (the waiting room looks clean, the practice has at least one computer for staff, at least one consultation/examination room has a container for sharps and has hygienic hand wash facilities) showed significant differences between countries (ANOVA P < 0.05), with intraclass correlations (ICC) of 1 to 56%. This shows that ‘country’ accounted for a moderate to high part of the variance for most indicators.¹⁵Table 2 gives an overview of the whole set of indicators. Here, we present per domain the indicators with the largest differences between countries (ICC > 25) as well as the indicators on which countries did not differ significantly.

Infrastructure. Across all nine countries 73% of emergency bags were found to contain essential emergency drugs, but the country means varied between 41% and 90%. About half of the practices (42%) had a functioning refrigerator with a thermometer, varying from 13 to 88% between countries. Overall, most practices (81%) have a clear phone message when phoning the practice out-of-hours, but in one country only 26% did. The larger majority of practices (86%) has a procedure for accepting patient requests for non-emergency home visits, varying from 43 to 100% between countries. Privacy in the consultation rooms with respect to undressing and examination varies enormously, from 13 to 97%.

Countries did not differ significantly (as they all had a high score) on cleanliness of the waiting room and having at least one computer for staff.

People. Although in 67% of all practices staff have a job description, one country had a mean of 27% and another of 94%. Overall, in three out of four practices there was a social event for the whole staff in the past 12 months, varying from 27 to 100% between countries. Staff in most practices (75%) reported a high level of work satisfaction, although staff work satisfaction was extremely low (23%) in Israel.

Information. In 76% of practices medical records contain smoking status, varying from 53 to 100%. In most practices (87%) computer backups are made daily, although in France only 40% of the practices do. While more than half of the practices (57%) record outgoing requests (referrals, tests etc), in one country this was just 10% and in another 86%.

Finance. Producing an annual report (62%), varies between countries from 33 to 98%.

Quality and safety. Not so many practices (30%) have lists or inventories of medical equipment and drugs, varying from almost none (7%) and two out of three (62%) between countries. Most practices (86%) have an explicit procedure for infection control, although in one country this was not common at all (31%). The same counts for having a leak-proof container in at least one consultation/examination room (76%, varying from 28 to 100%), hygienic hand drying facilities (77%, varying from 38 to 100%) and keeping controlled drugs in a locked cupboard (61%, varying from 11 to 90%). Only 32% of the practices have a no-smoking sign in the reception area, varying from no single practice to all practices per country.

As almost all practices (96%) have a container for sharps in the consultation/examination room as well as hygienic hand wash facilities (98%), these indicators did not differ significantly between countries.

Acceptability and feasibility of EPA

Observers. There were no specific difficulties related to the types of the visitors: all appeared to be capable to complete the assessment. There were no indications that the background of the visitors affected the outcomes.

Preparation of the assessment. For a number of practices in some countries there was a requirement for the visitors to overcome suspicion to this highly novel proposition. To overcome this resistance among practice staff, it was important to explain the assessment process in detail before the practice visit. The researchers in some countries solved this by visiting practices (UK, France) or contacting staff or practice manager by telephone (The Netherlands, Israel) to explain the study method. Others recruited practices by letter accompanied by a written informed consent form (Germany). After this introduction, there was a good level of practice cooperation.

All practices were contacted by telephone in order to obtain the names and job titles of staff members, which appeared labour-intensive. In larger practices it was often difficult to locate an individual who knew all relevant details about all staff, so that several calls were necessary.

Questionnaires. Most GPs and staff completed the questionnaires, sometimes after a reminder by phone or during the pilot visit. The agreed procedure for administering the patient questionnaires was to distribute them to the patients while they were waiting for their consultation, to fill them in on-site and to leave them in a sealed box at the reception area. However, in Switzerland patients received the questionnaires by post and were requested to mail them back.

Practice visit. The checklists did not cause many difficulties, although inspecting the emergency bag during the practice visit was easier when the owner was present, as some GPs consider the emergency bag to be a confidential property. However, this provided an opportunity to give immediate feedback. Inspecting the emergency bags was also often problematical for non-clinician visitors as they needed considerable time to recognize drugs and check expiry dates, particularly in larger practices, where up to four emergency bags had to be checked.

The interview with the principal GP or practice manager usually went well, although it sometimes took longer than planned and often one person was not able to answer all the questions. For that reason, some visitors conducted group interviews (Belgium), or an additional interview with other staff members (Israel). Some doctors reported spontaneously that they recognized areas for organizational improvement as a direct result of the interview and practice visit.

Practice staff asked for rapid specific feedback, but in this testing phase practices had to wait until all data were analyzed and national and international comparison data were available before receiving feedback.

All practices but one (in Belgium) completed the whole procedure.

Factors that contributed to the success or failure of the data collection

According to the visitors and project partners, the visitor needs to be well-trained, to have good interpersonal skills and to be able to handle sensible issues during the practice visit. Secondly, practice staff need to return all questionnaires on time. Thirdly, the practice needs to be well-prepared, implying that staff know that the visitor is coming and that they have allocated sufficient time for the visit. Fourthly, there should be an open, non-defensive atmosphere during the visit. Fifthly, GPs and staff need to be open-minded to the feedback that is provided, and motivated to find areas for improvement. Lastly, it is important that the National Health Service or professional body supports the assessment as this functions as a trustworthy recommendation.

Discussion

This study demonstrates that we were able to develop and use a feasible, valid educational instrument to measure the organisation and management of primary care practices within nine European countries with diverse health care systems. All practices, except one, that agreed to take part in the study completed the assessment. We developed a set of 57 valid, reliable and feasible indicators within the EPA instrument, of which 53 showed differences in practice management between and within countries.

Some aspects of the assessment procedure can be improved. Concerning the first contact with the practice, a good information package is required for the GPs and staff that describes explicitly what they can expect from an assessment visit. A form also needs to be developed to simplify the collection of names and affiliations of staff. Secondly, all countries should use the same method to sample and collect patient questionnaires. Thirdly, checking the emergency bag needs more attention in the training of non-clinician visitors.

There were no indications that the background of the visitor affected the outcomes. One important reason for this may be the highly standardized procedure; all visitors were selected on good interpersonal skills, they were trained to use the instrument and were familiar with practice procedures and systems in primary care.

Strengths and limitations

As we used small convenience samples of practices, and samples varied per country, we only used comparisons of the performance of the practices to validate the instrument. In France, for example, many group practices took part in our study, while in reality they are a minority, and in several other countries there was an under representation of single-handed practices. Therefore, it is not appropriate to make judgements about the relative performance of practices in each country on the basis of this data. As such we have only described the means of the whole research group, as well as the lowest and the highest country mean. However, we found differences in scores between single-handed and larger practices, confirming earlier findings that larger practices are better organized in a number of ways, but that patients prefer single-handed practices.¹⁶ We found that the UK practices had a good management with respect to staff, information and quality and safety. This is no surprise, since the UK has mostly large practices and practices are motivated to improve their practice management as a consequence of the new NHS contract. Surprisingly Slovenian practices, being a transitional country, had a good practice management as well as a good patient evaluation. In Switzerland and The Netherlands, GPs and staff reported the highest work satisfaction, while in Israel staff were not satisfied at all.⁸

Developing an international set of quality indicators means that compromises have to be made, which may overlook or mask national issues. Only indicators that were considered valid by national expert panels in all six countries were included in this paper.⁹ While questions based on indicators not rated valid in all countries were included in the EPA instrument, they were excluded from these analyses as the Delphi procedure was an essential part of the validation procedure.⁸ Many of the remaining indicators are, at this moment, not relevant or applicable in all countries. For example, recalling populations or patients at risk for primary or secondary care is not allowed in France and hardly feasible in countries where there is no patient list.

On the other hand, the EPA instrument lacks several indicators that were rated valid in the Delphi procedure, as we were not able to operationalize them into questions or items. We were, for example, not able to reach agreement upon assessing the essential basic equipment needed in a practice.

Implications for policy and practice

While good practice structure provides the opportunity to provide better quality of care,^1,9 evidence for a positive correlation between practice management and improved clinical outcomes is lacking.¹⁷ One reason for this has been a paucity of instruments to assess good practice organization. EPA provides such an instrument.

The EPA instrument could be used to compare primary care practices within or between countries, to compare certain types of practices (e.g. single-handed versus larger) within countries, or to look for trends in time.

Each country can chose to use the EPA instrument for a summative or a formative assessment, or a combination of both. The EPA instrument can be used to provide practices with tailored feedback including minimal standards (summative), benchmarks and suggestions for practice improvement (formative), on the initiative of the practice itself, or as part of accreditation.¹⁸ Each practice received a feedback report with their own practice scores compared to the national and the international mean. As no international standards were available yet, each country is free to develop or use their own national standards or benchmarks. In combination with other tools, this detailed assessment and feedback can be the starting point for quality improvement, as practice staff need help to select feasible improvement objectives, define barriers, make a change plan and to evaluate the result.^19,20

The EPA instrument could also be used as a starting point for improving practice management on a national level.

The future of EPA

The ‘EPA version 2005’, adapted on the basis of this study, will be owned by the ‘TOPAS-Europe association’, of which all EPA-partners became members. Implementation of the instrument has already started in Germany and Switzerland. In The Netherlands, it will be used as part of a practice accreditation instrument of the professional body.

The EPA-instrument will be made available for other countries as well. Although the instrument has been developed within a European context, expending the users to non-European country should be possible, as we used indicators from others continents in the development of the instrument.¹ As European unification continues and health care systems and procedures change, and as additional European or even non-European countries with different health care systems will use the instrument, the EPA instrument will be updated every three years.

Conclusion

The EPA instrument represents the first instrument that assesses the quality of practice management across Europe. The EPA indicators and instrument will be used for both formative and summative purposes. More research is needed to evaluate whether the instrument is suitable for both purposes, and to develop international standards and benchmarks.

Declaration

Funding: Bertelsmann Foundation, Germany.

Ethical approval: obtained in countries as appropriate, for example from a Multi-Centre Research Ethical Committee in the UK.

Conflicts of interest: none.

Appendix 1

Members of the EPA-collaboration

Centre for Quality of Care Research WOK, Radboud University Nijmegen Medical Centre, the Netherlands:

Richard Grol, Yvonne Engels, Maaike Dautzenberg and Pieter van den Hombergh

Bertelsmann Foundation, Gütersloh, Germany:

Henrik Brinkmann, Eckhart Volbrecht, Brigitte Mohn

AQUA Institute, Göttingen and Institutes for General Practice, Universities of Heidelberg and Frankfurt, Germany:

Joachim Szecsenyi, Ferdinand Gerlach, Björn Broge and Petra Wippenbeck

Société de Formation Thérapeutique du Généraliste, Paris, France:

Marianne Samuelson and Hector Falcoff

Swisspep Institute for Quality and Research in Healthcare, Gümligen, Switzerland:

Beat Künzi and Georg von Below

Scientific Society of Flemish General Practitioners, Berchem (Antwerp), Belgium:

Luc Seuntjens and Nicole Boffin

Department of Primary Care, University of Wales Swansea Clinical School:

Glyn Elwyn and Melody Rhydderch

National Primary Care Research and Development Centre, University of Manchester, UK:

Martin Marshall and Stephen Campbell

Partners that joined up at a later stage of the project:

Slovene Family Medicine Society, Ljubljana, Slovenia: Janko Kersnik and Vlasta Vodopivec-Jamsek

Clalit Health Services, Family Medicine Department of Technion, Haifa, Israel: Margalit Goldfracht

Austrian Medical Association and the Austrian Society for General Practitioners, Austria:Reinhold Glehr

References

Author notes

aCentre for Quality in Care Research (WOK), Radboud, University Nijmegen, The Netherlands bfor the EPA working party (Appendix 1) cNational Primary Care Research and Development Centre, University of Manchester, UK.