Measuring the quality of MDT working: an observational approach

Each of the 86 ‘characteristics of effective MDTs’ [19] was considered in turn for its relevance and feasibility for independent observational measurement in MDT meetings. Additional consideration was given to whether the characteristics were already measured in national peer review assessments, and priority was given to including characteristics that did not overlap with peer review. In total 18 of the 86 characteristics were felt to be observable. Due to overlap between characteristics these aggregated into 15 aspects of teamworking (Table 1).

Quality criteria for each characteristic were initially informed by relevant literature, observation of MDMs in a range of tumour types, and review of the survey data that had informed the Characteristics [18]. These prototype quality criteria were subsequently reviewed by an independent panel consisting of a consultant radiologist, oncologist and nurse with bowel cancer expertise as well as a senior NHS cancer manager. The panel discussed the potential variation in quality for each characteristic. Optimal ratings were calibrated against the recommendations in the Characteristics. Thresholds for lower quality performance were based on consensual agreement about the added value or detriment to MDT-working of variations to the optimal rating. It was agreed that the variation in quality for most characteristics would best be represented on a four-point scale: very poor (1), poor (2), good (3) or very good (4). Very poor/poor were merged for three characteristics where it was felt that further categorisation would lose value (Table 1).

Video-taped observation was proposed for rating behavioural characteristics such as chairing, teamwork and decision-making, to enable multiple observers to rate identical footage and allow replay. To enhance the reliability of behavioural ratings a case discussion proforma was deve-loped for rating characteristics on a case-by-case basis prior to being aggregated to an overall rating (Additional file 1). Aggregation was a qualitative process for most characteristics: forming global judgments on the rating that best reflected performance based on notes taken for individual cases. The only exception was patient centred care where the rating was based on quantification of objective content. This was because it would be necessary to know the patient in order to value-judge this characteristic of effectiveness. Rating is instead based on the frequency that patient-based information is considered (Table 1). Aspects of team functioning that could not easily be observed on a video-taped recording and could be objectively described were rated on the basis of notes and/or information collected by the researcher at the time of the videotaped meeting. This included attendance, the meeting agenda, meeting venue and technology. The researcher who was present at the meeting verified attendance information from the MDT lead and/or MDT coordinator, obtained an anonymised hard copy of the meeting agenda, and recorded information about the meeting venue and technology on a study-specific proforma (Additional file 2) to ensure systematic recording of the detail required for assessing these criteria.

Ten bowel cancer teams volunteered to participate (ascertained via their participation in another study [20]). One MDM per team was filmed and observed in-vivo. Quality was subsequently assessed using completed case discussion proformas, the completed in-vivo proforma and anonymised agenda (Table 1, Additional files 1 and 2). Short structured interviews were conducted with a range of members from each team (including at minimum the MDT lead, MDT coordinator and clinical nurse specialist) to determine the impact of the observational method on the meeting, and confirm attendance information.

Team members provided written informed consent prior to the meeting being video-recorded and all patient discussions and agendas were anonymised. Ethics approval was granted by the South East Multi-Centre Research Ethics Committee and R & D approval was obtained from the relevant NHS Trust for each MDT.

Feasibility was estimated by collecting data on the time taken to complete observational assessments. Inter-rater reliability was assessed by comparing the ratings of the research team with those of two independent observers: one oncology specialist registrar and one post-doctoral research psychologist. Intraclass correlation coefficients were calculated (two-way mixed models with measures of absolute agreement). 95% confidence intervals are shown. A significance level of p = 0.05 was used. Data were analysed using SPSS v.15.0 for windows.

The quality criteria and methods for assessing each characteristic are summarized in Table 1.

Presence of an observer and videocamera in MDT meetings was acceptable to team members and feasible to implement: it caused no delay or interruption to the usual flow of meetings. Interviews with team members confirmed all meetings had been typical: the presence of the researcher and camera had negligible impact on team member behaviour.

Meetings ranged in size from having 10 to 45 core and extended members present (median 16 members). The largest MDT meeting was a network-wide meeting comprising three local MDTs; all others comprised one local MDT. The average meeting length was 53minutes (range: 25–86 minutes) and teams discussed an average of 13 patients (range: 6–21 patients). Each patient was discussed for an average of 4.5 minutes (range 3–9 minutes).

There was wide diversity in ratings between teams across all characteristics. Total scores (out of a possible 59) ranged from 29 (team 3) to 50 (teams 2 and 9; Table 2). Only one team (Team 3) failed to achieve ‘very good’ criteria for at least one characteristic. The quality of teamwork in this team was particularly poor in relation to chairing (the chair allowed long pauses in discussions and did not explicitly confirm treatment plans before moving onto the next case); prioritisation of the agenda (9/14 cases discussed were rated as having too much time spent on them as key information or results were missing, or because relevant team members, for example the oncologist, were absent); and presentation of clinical information (case presentations were unprepared and unstructured, time was spent searching for relevant information for most cases). In contrast, Teams 2 and 9 met ‘very good’ criteria for 9 out of the 15 characteristics. These teams demonstrated: optimal chairing (by ensuring discussions remained focused and clearly articulating treatment plans); inclusion of all relevant team members in discussions (most other teams, although not always dominated by one or two members, particularly lacked contribution from the clinical nurse specialist or any other nurses); explicit use of the meeting for professional development (a well prepared, informative discussion about a waiting times breach resulting in clear actions); optimal venue and technology (a spacious purpose-built room; core team seated in horseshoe formation enabling sight of imaging/pathology and each other. Picture Archiving and Communications System (PACS) and multiple screens facilitated viewing of current and retrospective pathology and radiological imaging); circulation of a comprehensive agenda prior to meetings (including patient ID and demographics, all previous relevant history, tests undertaken/results, reason for MDT discussion, and space for MDT recommendation to be inserted).

Some internal consistency within teams was evident: MDTs performing well in one domain tended to perform well in other domains (and vice versa, Figure 1). Eight out of the 10 teams had the same rating (either ‘very good/good’ or ‘very poor/poor’) for at least three of the four domains of teamworking. The internal consistency was most evident in teams performing at either end of the spectrum: Team 3 received consistently poor ratings across all characteristics with three ‘very poor’ ratings and no ‘very good’ ratings, and teams 2 and 9 received 9 ‘very good’ ratings and had no ‘very poor’ ratings of quality for any characteristic (Table 2).

Teams were most likely to achieve ‘very good’ ratings for: administrative preparation (a comprehensive agenda), membership/attendance (all core team members or deputies present for whole meeting), and tension/conflict (no tension or conflict in meetings) (Table 2, Figure 2). Teams were least likely to achieve ‘very good’ ratings for: patient-centredness of case discussions and prioritisation of cases on the agenda. No teams were observed to explicitly consider patient-centred factors in all patient discussions: most teams considered patient-based factors in less than half of cases, mostly comprising demographic information rather than holistic needs or preferences. There was no attempt by any team to use the agenda to prioritise the order of case discussions to ensure appropriate time was spent discussing complex cases. In all teams at least some cases were judged as being discussed for too little or too much time based on their complexity.

Completion of assessments required 93 minutes on average (range: 45–160 minutes). There was no consistent difference in time taken by different observers; instead the time taken was associated with meeting length, averaging between 1.25 and 2 times the meeting length. There was a trend for assessments to take longer initially (average 2 times length of meeting) compared to the final few teams observed (average 1.25 times length of meeting).

Acceptable intra-class correlation coefficients of at least 0.50 (and up to 0.92) were achieved for all except two characteristics of effective teamworking: the absence of tension/conflict (ICC = 0.10) and evidence of treatment planning (ICC = 0.32; Table 3).