Instrument development, data collection, and characteristics of practices, staff, and measures in the Improving Quality of Care in Diabetes (iQuaD) Study

The study was a predictive study over a 12-month period. In summary, practice attributes and a wide range of individually reported measures were measured at baseline; measures of clinical outcome were collected over the ensuing 12 months, and a number of proxy measures of behaviour were collected at 12 months (detailed in Table 1).

At baseline we collected:

At 12 months we collected:

Practices were recruited from within the UK Medical Research Council General Practice Research Framework (MRC GPRF). When conducting similar previous studies with individually recruited primary care doctors [5], we had experienced low response rates in the face of long questionnaires. In order to be able to describe, characterise, and explore whole primary care practices, we wanted to achieve as close as possible to a 100% team response rate for the survey instruments from each practice. MRC GPRF practices volunteer to be research active and can directly receive funding to support their participation in research studies; practices were offered full reimbursement for the staff time taken to complete all study activities (including questionnaire completion) on condition that practice completion rates were satisfactory.

Recruitment was by postal invitation via the GPRF administration, with telephone follow-up of interested practices by the study research associate. Participants were all the clinical and non-clinical members of the primary care team in the practices recruited to the study.

To investigate the care offered to patients we identified six clinical behaviours (Table 2) performed in the management of patients with diabetes. These were chosen to: cover a range of clinical activities (prescribing, non-prescribing); reflect decisions that were not necessarily straightforward (controlling BP that was above target despite other drug treatment); and reflect recommended best practice as described by national guidelines [19]. The behaviours were precisely specified (according to the 'TACT' principle [20]: Target, Action, Context, Time or Who does What, Where and When) in order to provide consistency of measurement across practices and to reduce ambiguity when they were described to survey respondents.

Two primary care practices in northeast England took part in piloting the questionnaires. The first section (organisational questions) was piloted with seven administrative staff (practice managers, secretarial and reception staff) and seven healthcare professionals (primary care physicians, practice nurses, and one healthcare assistant). Piloting was by postal survey for all administrative staff and for five clinical staff. Participants were provided with the questionnaire and a stamped addressed envelope to return the questionnaire to the study research associate. They were given written guidance that asked them to complete the questions in their own time, noting how long it took to complete and to comment freely on the clarity and acceptability of the questions. The questions were found to be acceptable, there were no missing responses and the time taken to complete the instrument varied from seven to 25 minutes (median 20 minutes). No adjustments were made to the questions following piloting.

The second and third sections were initially piloted using postal methods as described above with one primary care physician and two practice nurses. One lead primary care physician for diabetes and one diabetes specialist nurse also piloted the questionnaire during a face-to-face session with the study research associate using 'think aloud' technique [41]. Based on the feedback received and concerns expressed during the 'think aloud' sessions, adjustments were made to minimise repetition in the wording of the items, and two behavioural scenarios (see Measures of behaviour below) were removed (leaving four in the final version) to shorten the questionnaire and to keep the completion time within an estimated maximum of two hours. The amended questionnaire was then re-piloted using postal methods with the two original 'think aloud' participants and an additional two primary care physicians and two practice nurses. No further amendments were suggested as a result of the re-piloting. All pilot participants received book vouchers (£10 for administrative staff, £20 for nursing staff, and £50 for doctors) for returning a completed questionnaire.

A 'self-reported behaviour' questionnaire, asked individual clinicians about their performance of each of the six clinical behaviours over the previous 12 months (see Additional File 3: Self Reported behaviour questionnaire). The items used in this very brief questionnaire (one item for each of the six clinical behaviours) were worded: Over the past 12 months, given 10 patients with diabetes < attributes of patients >, for how many did you < perform behavior >? (scored 0 to 10). Such measures of behaviour are commonly used and are well predicted by social cognition models [2].

The baseline postal questionnaire survey ran between September and December 2008. All the questionnaires for a practice were delivered to the nominated study contact in the practice who then distributed the questionnaires to practice colleagues. All participants were provided with written information about the study, asked to complete their questionnaires individually, and provided with a pre-paid envelope to return their questionnaire directly to the study research associate. Reminders were sent to non-responders at two and four weeks. Individuals not wishing to complete the study questionnaire and who wanted this to be confidential from their practice colleagues were given the option of returning a blank questionnaire.

This was administered 12 months after the baseline questionnaire and using the same method as described above.

Five different, complementary measures of the performance of the six study behaviours were collected. The first two provide individual level measures of behaviour, while the latter three give aggregated practice level behavioural data.

This 'simulated behaviour' measure derived from clinical scenarios (described above) provided the first of two measures of individual clinicians' self-reported performance of the six study behaviours. Clinicians could endorse that they 'would do' (score 2) or 'would do if time' (score 1) each behaviour plus add explanatory text. Scores for one of the simulated behaviours were adjusted to reflect current best practice-prescribing additional drug therapy for the management of HbA1c was, at the time of the study, advised for individuals whose HbA1c was above 8.0%. Therefore, for scenarios in which the simulated patient's HbA1c was ≤8.0%, the correct decision was not to prescribe additional therapy, and respondents who did not indicate that they would act on this were credited with having made the evidence-based decision.

The 12-month self-reported behaviour questionnaire (described above) provided the second measure of individual clinicians' self-reported performance of the six study behaviours.

Anonymised individual patient biochemical, physiological, and drug data were extracted from practice computer systems for all patients with a diagnosis of type 2 diabetes registered with the practice (see Additional File 4: List of Read Codes for the data items). For each of the computer systems used by the practices, search queries were written by an experienced National Health Service (NHS) performance data manager. Data were extracted for a 25-month period (i.e., 12 months prior to and 12 months after the month within which the baseline survey was launched). The search queries were sent to each practice along with written guidance on running the query, a process that practices were familiar with. The performance data manager also provided practices with telephone and email support if needed.

We anticipated that information on some of the study behaviours of interest might be recorded poorly, if at all, in the computer records, specifically those on the provision of advice on weight management, self-management, and general education. A single relevant question about each was included in a patient satisfaction questionnaire previously used by the Healthcare Commission [42]. In order to increase the specificity of the measure, as well as the single item, we identified additional items that assessed specific aspects of each behaviour with the aim of producing a composite score for each behaviour. We examined the internal consistencies and ran principle components analyses on the items within each behaviour and then across behaviours. Performance of foot examination was also asked about and so provided an additional, single item, measure of this behaviour.

Using a single posting, anonymous (to the research team) survey (for the questionnaire see Additional File 5), we asked patients in the study practices about their experiences of their clinicians providing advice about weight management, self-management, and general education about their diabetes. Aiming to achieve a final sample size of 25 respondents per practice, 86 practices approached 100 randomly selected patients anticipating a 25% response rate. Questionnaires were distributed from the practice and returned to the study research associate.

The Quality and Outcomes Framework (QOF) is a voluntary annual reward and incentive programme for all primary care practices in UK, detailing practice performance across a number of clinical areas (of which diabetes mellitus is one) plus organisational areas [43, 44]. The data are extracted from practice computer systems by the local primary healthcare administrative authority on an annual basis using a standard data extraction query. The data are publically available and QOF data on the diabetes and organisational domains were obtained from the NHS Information Centre http://​www.​ic.​nhs.​uk/​. The QOF data for diabetes mellitus and practice organisation were collected for each of the participating practices for the 12-month period of QOF data collection (May 2008 to April 2009) that best matched the 12-month period after baseline questionnaire completion. Where available, practice level numerators and denominators were obtained for diabetes mellitus indicators and percentage achievement levels were calculated; where they were not available, the calculated point score is reported.

The study was approved by Newcastle and North Tyneside 2 Research Ethics Committee, REC reference number 07/H0907/102.

The process of recruitment of primary care practices is shown in Figure 1. The initial invitation went to all GPRF practices in Scotland, Wales, Northern Ireland, and a random sample of practices in England up to a total of 500 practices. One hundred practices were recruited and all took part in the telephone interview, baseline, and follow-up phases of the study. One practice was subsequently excluded from all analyses due to low completion rates for all data collection; we subsequently report on 99 practices. All practices completed a telephone interview. Informants were GPs for 47 practices, nurses for 37 practices and the practice manager for 15 practices. All practices were invited to verify their data summaries and 75 did so.

The baseline questionnaire (organisational questions) was sent to all clinical and administrative staff (2,079 in total). Usable completed questionnaires were returned by 946/1,236 (76.5%) administrative staff, 423/529 (80.0%) primary care doctors, and 255/314 (81.2%) nurses (see Figure 2). One thousand and fifty-five staff members indicated that providing care for patients with diabetes was part of their routine role and 890/1,055 (84.4%) went on to complete the diabetes-specific versions of the measures in the questionnaire.

The baseline questionnaire (clinical questions) was sent to all clinical staff within each of the 99 practices (843 in total). Of clinicians who indicated that they were involved in providing diabetes care, usable completed questionnaires were returned by 326/361 (90.3%) primary care doctors and 163/186 (87.6%) nurses (see Figure 2). Three hundred and ten primary care doctors and 162 primary care nurses responded to at least one area of care in every clinical scenario. Table 5 presents the practice level response rates for the two baseline questionnaires by staff type (excluding 146 questionnaires that were returned blank). We achieved 100% overall response rates from clinicians in 40 practices and achieved responses from over 80% of clinicians in 67 practices. We achieved 100% response from 38% of practices for at least one of the generic organisational questionnaires and from 84% of practice for at least one of the two diabetes-specific organisational questionnaires. Sixty percent of practices had a 100% response for questions on at least one individual-level psychological model.

The follow-up questionnaire was sent to 843 clinical staff. Six hundred and ninety-four (82.3%) completed questionnaires were returned. Of those involved in providing diabetes care, 427/547 (78.1%) could be paired with a completed baseline clinical questionnaire (see Figure 2).

Practices were supplied with a total of 8,600 patient questionnaires. Given the anonymous nature of the survey and the fact that practices with less than 100 patients with diabetes will have sent out fewer questionnaires a precise response rate cannot be calculated. A total of 3,591 analysable questionnaires were received (41.8% return rate).

Seventy-four of the recruited practices were located in England, 13 in Scotland, four in Wales, and eight in Northern Ireland. Thirty-seven were rural practices and 62 were urban; 15 had branch surgeries (range 2 to 5 sites); 18 were dispensing practices; 62 were training practices. The mean (SD) patient list size was 7,431 (4,040), with a mean (SD) proportion of patients aged over 65 years of 18% (7%). Most practices served patients of mainly 'White British' origin (84/99), and 63 practices 'never' or 'rarely' used interpreters. Tables 6 and 7 summarise the structural and functional characteristics of the study practices, both in general and in relation to diabetes care. There was a mean (SD) of 5.4 (2.7) doctors per practice covering a mean (SD) of 36.4 (20) half-day (notionally 3.5 hour) sessions and providing a mean (SD) of 515 (315) appointments per week. Similarly 3.1 (1.6) nurses per practice offered 17.7 (10.5) half-day sessions. Though only compared descriptively, study practices were of an equivalent size to MRC GPRF practices overall (mean list size 7,696). Since devolution in 1998, comparative UK data is hard to find but, compared to all general practices in England, the study practices were larger and had more doctors (2007 England mean list size: 6,487; mean number of practitioners per partnership: 4) and, at 4%, the study sample also contained a low proportion of single-handed practices [45].

Table 9 presents the mean (SD) scores and internal consistency for each theoretical construct included in the clinical questionnaire. The internal reliability measures are all acceptable. Across the six behaviours, the scores for the constructs were all generally well towards the positive end of the seven point scoring scale. For each of the theories the median (range across behaviours) was:

Within the theories, whilst overall no Theory of Planned Behaviour construct was scored below five, the control item had the lowest scores across all six behaviours, a similar pattern to the self-efficacy item scores within Social Cognitive Theory suggested that clinicians had stronger motivational than action cognitions. Coping planning was scored lower than action planning for all six behaviours, suggesting that clinicians were clearer how to initiate behaviours than to cope with problems should their initial plans not succeed.

Intention (measured either as strength of intention or direct estimation) to perform the behaviour was highest for 'giving advice about weight management' and was lowest for 'prescribing additional anti-hypertensive drugs' (strength of intention) and 'foot examination' (direct estimation). The highest habit score was also for 'giving advice about weight management' and the lowest was for 'prescribing additional anti-hypertensive drugs.' For action planning and coping planning the highest scores were both for 'foot examination'; the lowest action planning score was for 'giving advice about self-management' and the lowest coping plan score for 'giving advice about weight management.'

The mean (SD) rates of performance of the six behaviours are shown in Table 10 along with the patient responses to the questions about the three receiving advice behaviours and foot examination. Within the self-report questions, for both groups of clinicians, although reported rates of performing the behaviours were high, with two-thirds of rates being above seven out of ten, there was variation within the rates with standard deviations generally being just over two. Nurses reported performing the three 'giving advice' behaviours more often than doctors did, reporting performing the behaviour for almost 9 out of 10 patients. For foot examination, there was the widest difference between doctors and nurses, potentially reflecting different agreed roles and different patient populations seen.

The single-item patient report data are directly comparable to the clinician report data and, for foot examination, the patients' reports matched the nurses self-report almost exactly. For the other three advising behaviours, the patient-reported rates of receiving advice are consistently lower than the clinician-reported rates of giving it. For advice about self-management and providing advice about general education (converting the clinician n/10 scores into percentages) the gap is 21% and 14%, respectively. For advice about weight management, the gap is 52% with clinicians reporting that they gave advice about twice as often as patients reported receiving it.

When testing the composite items, the principal components analysis (PCA) on items within each behaviour suggested that each involved more than one component. For providing weight management advice and providing general education, these did not outweigh the clinical face validity of the initial scales nor did removing items improve the internal consistency. For providing self-management advice, PCA results informed the decision to remove three items. For the resulting composite measures, there were eight items for providing weight management advice (Cronbach's alpha 0.80), three items for providing self-management advice (Cronbach's alpha 0.66), and 18 items for providing general education (Cronbach's alpha 0.91). Details of the items and the analysis are in Additional File 6.

The mean (SD) scores for the composite items are shown in Table 10. For providing weight management advice, 51% of patients endorsed the single item but the mean number of items endorsed was 2.5/8, although 71% responded 'yes' to at least one item. Similarly, for providing self-management advice, 67.5% of patients endorsed the single item, the mean score on the composite measure was 1.5/3 and 83.4% responded 'yes' to at least one item; for providing general education, 72.3% endorsed the single item, the mean score on the composite measure was 7.4/18 and 93% responded 'yes' to at least one item.

The rates of the study behaviours are in Table 10. The data extracted from the practice computers are usually of the form of process (recording that a behaviour was done such as issuing a prescription) or intermediate patient outcome measures (such as recorded BP). The links between this data and the study behaviours are more or less direct. For behaviour one (providing advice on weight management), data for weight/height/body mass index (BMI) was available from all practices and reflects the physiological endpoint of the behaviour we asked about. However, assuming such advice is given, there are a number of clinician (how well was it given) and patient (was it heard, accepted, acted upon) factors that intervene before any effect of performing the behaviour plays out through a change in a measure such as BMI. Unfortunately, the available computer codes for offering advice about weight though present were infrequently used and hence cannot be used as an outcome measure in this project. Behaviour two (prescribing additional antihypertensive drugs) and behaviour five (prescribing additional therapy for managing glycaemic control) relate to drug prescription in relation to physical examination or laboratory test results. Values for BP and HbA1c were universally available, and drug data that was available from 81 practices. The analysis is currently computing the eligible patient populations (BP > 145/85; HbA1c > 8.0) and whether or not relevant treatment was increased or added at relevant consultations. This is entailing a considerable amount of coding of frequency of dose data (usually entered as text rather than coded data) and coding of maximum doses of drugs to allow the identification of a population of patients who most closely match the behaviour. Although time consuming, this will provide a much more precise measure of a prescribing behaviour than we have been able to achieve in previous studies where we relied on routine data [5]. Data on the rates of performing behaviour three (examining feet) was available from 86 practices. For behaviours four and six, we found low rates of relevant computer codes both within and across practices. For behaviour four (providing advice on self management), we have computer code data for 68 practices (and from only 63 of these in the year following completion of the questionnaires); in addition, we have coded data on the provision of diabetes self-monitoring equipment (the use of which can form part of self-management) recorded from 47 practices. Patient education codes (behaviour six) were recorded in only 33 practices (and in 19 in the year following completion of the baseline questionnaires). Therefore, for these two behaviours we will be using the patients report data as our main measure of the behaviour.

The QOF data are shown in Table 11. The QOF scores give a routinely available measure of clinical and organisational performance, though the rates of achievement against the organisational indicators are almost maximal, suggesting that these indicators will not usefully discriminate. QOF is also limited in terms of how the indicators relate to the clinical behaviours of interest within this project. Neither the behaviour 'giving advice about self-management' nor 'providing general education' have any useful match within the QOF data. For 'giving advice about weight management' the only indicator related to weight is 'patients' notes recording BMI' and, although this might reflect on the organisation of a practice, with mean achievement levels of 96% and a standard deviation of three, like the other organisational indicators, it too is unlikely to have sufficient variation to be discriminating. There is a good match for 'foot examination' and the mean achievement levels of 92% match the clinician and patient report well. For the other two behaviours 'prescribing additional anti-hypertensive drugs' and 'prescribing additional therapy for the management of glycaemic control,' there are indicators that could reflect the active performance of the two behaviours. In practices where clinicians are actively trying to tightly control both BP and glycaemic control, it would be reasonable to expect higher rates of patients with lower BP and HbA1c-and there is one QOF indicator for each of these with rates of performance of 80% and 68% respectively.