Logic model development
Following data extraction and quality appraisal, the process of systematically constructing the logic model began. We developed the model column by column, underpinned by the evidence. The model contains five columns detailing the pathway from interventions to short-term outcomes; via moderating and mediating factors; to demand management outcomes; and finally demand management impact.
The first stage in building the model was to develop intervention typology tables from the extracted data, in order to begin the process of grouping and organising the intervention content and processes which would form the first column. This starting point in the pathway details the wide range of interventions which are reported in the literature. It groups these interventions into typologies of: practitioner education; process change; system change; and patient intervention. Within each of these boxes the specific types of interventions in each category have been listed, for example the GP education typology contains interventions targeting training sessions, peer feedback, and provision of guidelines. Process change interventions include electronic referral, direct access to screening and consultation with specialists prior to referral. System change interventions include additional staff in community, gate-keeping and payment systems. We found few examples of patient interventions. The model provides an indication of where the evidence is stronger or weaker. For example in regard to physician education it can be seen that peer review/feedback has stronger evidence underpinning its effectiveness, with the use of guidelines being underpinned by conflicting evidence of effectiveness. For all but two of the interventions, the evidence was for either none or some level of positive outcome on referral management. For the additional staff in primary care and the addition or removal of gatekeeping interventions however there was strong evidence that these could worsen referral management outcomes.
The interventions thus formed the starting point, and first column of the logic model. By developing a typology we were able to group and categorise the data, and begin to explore questions regarding which types of intervention may work, and what characteristics of interventions may be successful in managing patient referral.
The intervention studies used a wide range of outcomes to judge efficacy. A key aim of logic models is to uncover assumptions in the chain of reasoning between interventions and their expected impacts, and to develop a theory of change which sets out these implicit “if…then” pathways. The next stage in development of the model was therefore to begin to unpack these outcomes and assumptions regarding links between interventions and demand management impacts.
The outcomes were divided into those which were considered to be short-term outcomes, long term outcomes or result in broader impact on demand management systems. In order to do this we used “if…then” reasoning to deduce in what order outcomes needed to occur for these to then lead to the intended impact. Short-term outcomes were classified as those that impacted immediately or specifically on individual referrers, patients or referrals. Long term outcomes were categorised as those which had an effect more widely beyond the level of the individual GP, service or patient, and impact factors were those that would determine the effectiveness of referral management across whole health systems.
Outcomes and impacts reported in the intervention studies were identified and grouped by typology, by the stage in the pathway, and by the level of evidence. The outcomes column includes all those outcomes which were reported in the included papers. They encompass: whether or not the adequacy of information provided by the referrer to the specialist was improved; whether there was an improvement to patient waiting time; whether there was an increase in level of GP or patient satisfaction with services, and whether referrals were auctioned more appropriately. These outcomes form an important element of the pathway to the final impacts column and demonstrate the importance of identifying all the links in the chain of reasoning. For example the model outlines that referral information needs to be accurate in order that referrals may be directed to the most appropriate place or person. Interventions need to include evaluation of this interim outcome and not only consider impact measures such as rate of referral if they are to explore how and if an intervention is effective. Also, GP satisfaction with a service will determine where referrals are sent, and patient satisfaction may determine whether a costly appointment with a specialist is attended. Here again many studies we evaluated used only broad impact measures (such as referral rate) to evaluate outcomes rather than explore where the links in the pathway may be breaking down.
The impacts column contains all those impacts that were reported in the included literature. These were: the impact on referral rate/level; whether attendance rate increased; any impact on referrals being considered appropriate; any impact on the appropriateness of the timing of the referral; and the effect on healthcare cost. As can be seen from the model, the relationship between interventions and a wider impact on systems was challenging to demonstrate from the evidence.
Having developed the first and final two columns of the model, attention then turned to the key middle section. This phase of the work required detailed exploration of the change pathway to explore exactly how the interventions would act on participants in order to produce the demand management outcomes and impacts. The second and third columns of the model are core elements of the theory of change within the model.
While a small amount of data for these elements came from the intervention studies, the majority came from analysis and synthesis of the qualitative papers and non-intervention studies. Much of the intervention literature seemed to have a “black box” between the intervention and the long term impacts. This was a key area of the work where we employed iterative additional searching in order to seek evidence for associations, to ensure that the chain of reasoning was complete. For example, the first additional search aimed to explore evidence underpinning the assumption that increasing GP knowledge would lead to improved referral practice. The second additional search aimed to identify evidence underpinning the link between changes in referral systems and changed physician attitudes or behaviour. The search also sought evidence regarding specific outcomes following interventions to change patient knowledge, attitudes or behaviour.
The second column of the model details the short-term outcomes for individual GPs, patients, and GP services that may result from interventions. These are the factors which need to be changed within the referrer or referral, in order that the longer term outcomes and impacts will happen. The short-term outcomes we identified were: physician knowledge; physician beliefs/attitudes; physician behaviour; doctor-patient interaction; patient knowledge, or patient attitudes/beliefs or behaviour. Of note is the weaker evidence of physician knowledge change impacting on referrals, and greater evidence of change to physician attitudes and beliefs, and also doctor-patient interaction having an impact.
The third column (and final element to be completed) is another key part of the theory of change. This section identifies a range of factors which may be associated with or influence whether the short-term outcomes will lead to the intended longer term outcomes and impacts. This column examines the moderating and mediating variables which may act as predictors of whether an intervention will be successful. They can be considered as similar to the barriers and facilitators often described in qualitative studies. The model details a wide range of these moderating and mediating factors relating to: the physician; the patient; and the organisation. Of particular interest here is the conflicting evidence relating to physician and patient demographic factors (the subject of a large number of studies) influencing referral patterns, and the clearer picture regarding the influence of patient clinical and social factors in the referral process.
Having outlined the content of each column, the following provides an example of the flow of reasoning for one particular type of intervention underpinned by elements of the model. Much work in the UK has been directed towards issuing guidelines for GPs regarding who and when to refer, with the assumption that changed knowledge will lead to changed referral practice. However, the model questions these assumptions by indicating that there is conflicting evidence regarding the efficacy of this type of intervention, and also suggesting that there is weak evidence of interventions such as these leading to enhanced knowledge outcomes, Perhaps if the guidelines focused more on elements of the model where evidence is stronger such as addressing GP attitudes and beliefs (for example tolerance of risk) or behaviour (such as the optimal content of referral information) this may lead to more successful immediate outcomes. The model highlights however that the effect of any guideline intervention will also be modified by GP, patient and service factors, for example the complexity of the case, the GP”s emotional response to the patient and GP time pressure. These potential barriers need to be considered in the implementation of guideline interventions. If these elements can be addressed however, use of guidelines by GPs may enhance referrer or patient satisfaction, improve waiting time, or change the content of a referral and thus have a resulting impact at a service wide level.
Evaluation of the model
Following development of the model we sought feedback from stakeholders regarding the clarity of representation of the findings, and potential uses. This consultation was carried out via individual and group discussion with practitioners, patient and the public representatives, commissioners (individuals who have responsibility for purchasing services), and by circulating the model to experts in the field. In total we received input from 44 individuals (15 GPs, five commissioners, seven patient and public representatives, and 17 hospital specialists from a range of clinical areas). Thirty eight of the respondents reported that they clearly understood the model however, four specialists described the model as overly complex and 2 patient representatives reported some confusion understanding it.
GPs in particular gave positive feedback, highlighting that it was a good fit with their experience of the way referrals are managed, and that it successfully conveyed the complexity of general practice. The model was also described positively as identifying the role of both the GPs’ and the patients’ attitudes and beliefs, and the doctor-patient interaction. Also, GPs noted with satisfaction that the model included the physicians’ emotional response to the patient, which resonated with their experiences. Most specialists also reported that the model was a good fit with their experience of factors influencing referral management. Potential uses of the model described were: as a tool for GP trainees and educators; as a teaching aid for undergraduate medical students; for analysing the demand management pathway when commissioning; for comparing what was being commissioned with what was evidence based; and to direct research into poorly evidenced areas.
Some of the feedback from participants concerned factors that had not been identified in the literature. For example the potential role of carers as well as the patient in doctor-patient interactions was highlighted, and the potential influence of being a GP temporarily covering a colleagues’ work. Some amendments were made to the model following this feedback, principally clarifying where there was no evidence versus inconclusive evidence, and editing terminology.