Summary of results
In the present study the diagnostic agreement between ESPs and OSs was 62% for primary diagnosis, 79% for the same combination of diagnoses and 96% for partial agreement on the diagnoses (the primary diagnosis registered by one of the professionals was also registered as either primary or secondary diagnosis by the other). The OSs had a significant higher agreement with the common decision on the common decision than the ESPs. In the predefined primary analysis, ESPs and OSs agreed on the treatment plan being either possibly invasive, physiotherapy or no intervention in 88% of the cases. Due to a skewed distribution of data (93% allocated to possibly invasive treatment) this result was considered less meaningful than expected. Across treatment categories the agreement between OSs and ESPs varied between 68 and 100%, with OSs having a significant higher agreement with the common decision on the need for diagnostic imaging. In 43% of the cases, ESPs and OSs agreed on the combined treatment plan and in 96% of the cases they partially agreed.
In the evaluation of collaboration between ESPs and OSs, data showed positive statements regarding all seven dimensions of relational coordination. We found close communication, equal and respectful relationship and professional skills to be especially important.
In the following, the quantitative results related to aim 1a (agreement on diagnoses) and 1b (agreement on treatment plan) are discussed integrating some of the qualitative findings in the interpretation of the results.
Agreement on diagnoses
We have only been able to identify two studies, which have reported inter-observer agreement between ESPs and OSs when diagnosing patients with shoulder disorders, thus being eligible for comparison with our results [
15,
23]. Among seven categories, Razmjou et al. [
15] reported a diagnostic agreement between ESPs and OSs on each category varying between 84 and 98%. In their study, they used different diagnostic categories compared to our study and allowed for selecting more than one diagnosis without differentiating between primary/secondary [
15]. This means, that our estimates cannot be directly compared, and the 62% agreement on primary diagnosis was expectedly lower than Razmjou’s estimate of agreement in each category. In a retrospective audit on selected shoulder patients, a fully comparable diagnostic agreement of 65% and partial agreement on further 31% were reported [
23]. Adding it up, it makes their result similar to the 96% of cases with partial diagnostic agreement in our study.
Furthermore, we found 79% agreement on the combination on diagnoses. Considering the fact, that in our setting, it is not clinical practice to distinguish between primary and secondary diagnoses, this estimate is probably the most clinically relevant. To our knowledge, no other studies have reported a similar estimate making comparison with other studies challenging. It is, though, comparable to general results on diagnostic agreement, when looking at studies investigating different musculoskeletal conditions [
12].
To determine if our results reflect acceptable quality several factors should be considered. A 100% agreement cannot be expected, as diagnosing patients with shoulder disorders is a complex procedure, which takes comprehensive clinical reasoning skills in addition to performing specific tests to make a diagnosis. Some degree of subjectivity cannot be eliminated, and intra-professional agreement among OSs or ESPs is not expected to be perfect either. We haven’t been able to identify studies conducted in a setting similar to ours, that verify this, but the assumption is supported by an older study evaluating the agreement between three rheumatologists diagnosing 44 patients with shoulder disorders [
45]. A 46% complete agreement was found, but it should be noted, that it will always be more difficult to get full agreement between three persons than two. In the same study, further 18 patients were examined by all three together where after they discussed and agreed on symptoms and signs, before individually writing the diagnosis and recommended treatment. With this procedure 78% agreement were reported [
45], which underlines the impact of clinical reasoning.
It should be considered also, that there is no gold standard available for comparison. In cases of disagreement, we cannot determine the correct choice. Even in cases with agreement, a previous study has reported the diagnosis could still be incorrect [
22], and although this was an example based on a knee disorder, it is reasonable to expect, this could also be the case among patients with shoulder disorders.
In this study, we used the common decision as the assumed most correct diagnosis(−es). Comparing each of the professions with the common decision, OSs had a significant higher agreement than the ESPs. This could be due to better professional skills, supported by the fact, that the OSs in this study were highly experienced consultants or specialty registrar. This is to some degree supported by the findings in our qualitative study. Although, an equal and respectful relationship were revealed, still, the physiotherapist acknowledged, that the OSs had more experience and another level of education. This consideration could also be part of the explanation why the ESPs registered need for inter-professional consultation in 74% of the cases, compared with 7% registered by the OSs. The difference in proportions was not surprising, since the patient was referred for an orthopaedic specialist evaluation and the ESPs practice under the responsibility of the OSs. It was also expressed in the qualitative study, that ESPs consulted the OSs more often than the other way around. We did not collect data on reasons why the ESPs needed inter-professional consultation. However, during interviews a mutual understanding of indications for inter-professional consultation was expressed, hence, the OSs in our setting presumably agrees, that consultations were relevant.
Looking at further findings from the qualitative study, the ESPs’ high proportion of 74% could also be interpreted, as the ESPs feeling so comfortable with the OSs, that they sought advice whenever needed. Furthermore, even though the OSs only registered need for inter-professional consultation in 7% of the cases, the results reflect the finding of ESPs and OSs having a mutual understanding of having some competences in common but also having some competences specific to each profession.
Returning to the fact, that OSs had a higher level of agreement with the common decision compared to the ESPs, it should also be considered an explanation, that the task is traditionally managed by the OSs and the historical hierarchy of OSs being placed higher than physiotherapists.
Agreement on treatment
We predefined the primary analysis as a comparison of the agreement on treatment plan (three categories). This choice was clinically based, because we considered it highly important to decide if a possibly invasive treatment was a relevant option for the patient. However, the data distribution between categories (93% in the category “possibly invasive”) made it less meaningful to use as an indicator of quality. Although, we showed an 88% agreement between ESPs and OSs, based on Kappa, this was not higher, than what could have been achieved by chance. However, due to the skewed data distribution, this was expected, and when calculating PABAK, it was 0.76. Thus, we still consider the agreement on 88% to be satisfactory. In seven of the eight cases where ESPs and OSs didn’t agree, it concerned whether the patient should be offered a steroid injection in adjunction to physiotherapy or not. This was not surprisingly the issue, as these two treatment types are often prescribed simultaneously, in patients with shoulder disorders caused by subacromial impingement [
46]. In one case only, the physiotherapist had not registered either need for inter-professional discussion or a follow-up visit, thus in usual clinical practice, the disagreement would have been solved this way in all cases but one. This underpin the findings from Study 2, highlighting the importance of close communication, enhanced by using a setting allowing for the opportunity to do immediate communication per need.
The inter-professional agreement across the five treatment categories varied between 68 and 100% with a 93% agreement on indication for surgery. In this study, only three patients were immediate candidates for surgery, and this low proportion inevitable leads to a reduced kappa-value (0.41, indicating moderate agreement). The corresponding PABAK was 0.86, which support, that the low proportion of candidates for surgery is the main reason why our kappa-value is lower than the 0.75 (PABAK = 0.76) reported in the study of Razmjou et al. [
15], in which, around half of the patients required surgical consultation. Our proportional agreement on surgery is nonetheless considered high and similar to the 88% reported by Razmjou et al. [
15] and the 86% of cases in the study of Oakes et al. [
23], where ESPs correctly predicted which patients would undergo surgery.
In our study, 20 participants (29%) were referred to diagnostic imaging (MRI or MR-arthrogram) and the overall inter-rater agreement was 81% (κ = 0.57, PABAK = 0.62). In comparison, Razmjou et al. [
15] reported agreement on each category separately and found 91% agreement on MRI and 97% on MR-arthrogram. However, due to a small proportion of their population referred to these investigations, they reported lower kappa-values, 0.27 (PABAK = 0.86) and 0.38 (PABAK = 0.94) respectively. The differences in analyses and proportion of patients referred to MRI should be taken into consideration when comparing our results, but all together, we find our results to be comparable.
In the explorative analyses, the combined treatment plan was found to be exactly the same for the ESPs and the OSs in 43% of the cases and partially the same (at least one category in common) in 96% of the cases. To our knowledge, there are no other studies reporting similar estimates for comparison. To evaluate, whether these estimates indicate satisfactory quality, different factors should be taken into consideration. First, the total agreement would expectedly be quite low, as there were 16 different possible combinations available – and 13 were actually used. Second, the high partial agreement could be affected by the professionals’ option to choose as many categories as considered relevant, thus increasing the chance of reaching agreement on at least one treatment. However, this is not considered a great concern, as both professions chose no more than one or two treatments in about 90% of the cases.
In lack of a gold standard, we used the common decision as the assumed best choice. When comparing the treatment plans suggested by ESPs and OSs with the common decision, it is important to note, that there was no statistical difference between the professions on indication for surgery. ESPs differed from the common decision in three cases and OSs in one case. This exemplifies, that both professions could change their view based on inter-professional consultation, even when it comes to indication for surgery. This result supports the qualitative finding of an equal and respectful relationship, and their mutual trust in each other’s professional skills.
The absence of a statistical difference between the professions on indication for surgery is positive, when comparing with Razmjou et al., where the ESPs tended to recommend surgery more often than the OSs [
15]. But, compared to the common decision, the OSs in our study had a significantly higher agreement than ESPs with respect to treatment (three categories), diagnostic imaging and total agreement on the combined treatment plan. Reasons for this difference are similar to those discussed for diagnostic agreement.
In summary, when comparing with other study results and considering factors challenging a perfect agreement, we believe that our results of both diagnostic and treatment agreement indicate a satisfactory level of quality when evaluating ESPs and OSs sharing the task of diagnosing patients with shoulder disorders. In the interpretation of the quantitative results, it is important to note, that close communication, equal and respectful relationship and professional skills, seems to contribute to being able to successfully share the task.
Collaboration
The findings of our study indicate a high degree of relational coordination between ESPs and OSs highlighting close communication, equal and respectful relationship and professional skills. Based on the indication of a high relational coordination, the overall interpretation of the qualitative data was, that inter-professional collaboration was good. Our findings indicate, that our setting and the collaboration between ESPs and OSs, meets a great deal of the needs and recommendations mentioned in other studies [
19,
30] as well as The World Confederation for Physical Therapy (WCPT) Policy Statement: Advanced Physical Therapy Practice [
47]. In the study of Weatherley et al. it was described, that to provide patients with the best care, the referrals should be appropriate and the physiotherapist should receive support from the surgeon [
19]. Our data shows, the support of the OSs is undoubtedly present, and we also found indications of a well-functioning referral. In our setting, based on the referral from the patient’s general practitioner, a selected group of patients are deemed equally eligible to be examined and diagnosed by an ESP or an OS (Table
1). A criterion for this is the patient not being a clear candidate of surgery. This is highly fulfilled in our study population, where three patients only, were immediately recommended surgery. Even though a higher proportion of surgery candidates are expected later in the process (e.g. after diagnostic imaging), the referral is considered highly adequate. In the study of Dawson and Ghazi, it is concluded, that ensuring a good relationship between ESPs and the medical team as well as providing adequate ongoing training and support, could minimize many of the difficulties encountered by the ESPs [
30]. Furthermore, in the same study, one of the recommendations for the future were that ESP positions should be set up at the request of, or with full back-up from the orthopaedic team [
30]. This description fits perfectly the way our setting evolved, and the findings in our study revealed, that the ESPs experienced full back-up from the OSs. Some of the other recommendations were: adequate time allowed for shadowing; basic training in requesting and reading X-rays, blood results and understand common pharmacology; and responsibilities and expectations defined at onset [
30]. To a great extent, these recommendations are met by the ESP education program established at our hospital. The provision of appropriate education is also advocated in the World Confederation for Physical Therapy policy statement [
47], thus underlining the importance of having adequate professional skills to manage the post of being an ESP.
To our knowledge, no previous studies have investigated the experience of ESPs in the light of professional quality. In this study, we found indications of both a high level of relational coordination between ESPs and OSs as well as a satisfactory level of agreement on diagnoses and treatment plan, thus an association may be present. This is in accordance with studies in other field of health care, showing a positive association between relational coordination and quality of patient care [
26,
27]. It was beyond the scope of this study to investigate and describe how to best establish a new position for an ESP and how to achieve a good collaboration. However, we have thoroughly described our setting and how it evolved, as well as our education program (Table
2). Along with specific recommendations from other studies [
30,
47], this description may be used as an inspiration.
Study strengths and limitations
The strengths of the inter-rater agreement study are the examinations being performed in a randomized sequence, thus avoiding systematic bias from potential changes of patient history or reaction to tests based on the previous examination. Furthermore, the ESPs and OSs performed the examinations fully independent, blinded to each other’s findings and on the same day. Also, the patients were blinded to the diagnosis and treatment plan suggested by the ESP and OS, respectively, and to further avoid referral of information between examinations, participants were instructed not to talk to the second assessor about what was said and done at the first examination. Altogether, the risk of information bias is considered low and expectedly reduced compared to the previous studies investigating inter-rater agreement between ESPs and OSs examining patients with shoulder disorders [
15,
23]. Finally, all patients included in the study were included in the analyses, thus no attrition bias is present. However, our results can only be generalized to the group of patients with shoulder disorders not being clear candidates of surgery, as this was part of the eligibility criteria. Still, in our setting, this group accounts for about 70% of all patients referred to The Shoulder Clinic, thus making it relevant for the majority of patients.
The registration chart for diagnosis and treatment were self-developed, thus not scientifically validated. This was chosen to reflect clinical practice and also because, due to our knowledge, there is no consensus on how to group diagnoses in the most appropriate manner. However, we minimized the risk of misclassification by using ICD-10 codes in each category and increased content validity by pilot-testing the chart to ensure that all diagnoses were covered. Based on this, we consider the risk of misclassification to be low.
In the qualitative study, we aimed to ensure trustworthiness as described by Miles, Huberman and Saldana [
48]. Some of the study’s strengths and limitations of importance for trustworthiness are discussed below. Two separate focus-group interviews were performed to prevent bias (presumably in the positive direction) when describing the collaboration. We also consider it a strength, that the findings were confirmed by three researchers and considered accurate, when being presented to the original participants. Furthermore, as all ESPs and OSs at The Shoulder Clinic participated, data was collected across the full range of respondents and participants were fully representative for our setting. However, although being fully representative for our setting, the results can only be generalized to highly experienced and educated clinicians (both ESPs and OSs) in similar settings.
The use of relational coordination as theoretical framework and semi-structured interviews were chosen to evaluate dimensions of collaboration previously shown to be of importance for quality. However, this framework may not be sufficient to fully describe the inter-professional collaboration. First, it does not comprise power relations, which could also have an impact on the collaboration [
49,
50]. Second, in our study, both the setting (including physical framework) and professional skills were contributors to the good collaboration, and neither of these factors is explicitly comprised in the seven dimensions of relational coordination. Thus, using semi-structured interviews primarily based on relational coordination implies the risk of missing some important areas. We tried to counter for this, by also asking the participants for overall factors necessary for a good collaboration, but still, the data produced cannot be interpreted as exhaustive.
A potential cognitive bias could be a positive attitude towards ESPs sharing the task of diagnosing patients with OSs, since several authors are employed at The Shoulder Clinic (TMK, CLL, SMRV, SJD and BEL) and others are Physical Therapists (MNM, LRM and JT). To counter for this, the main part of the analyses was performed by an author without a health professional education (MLK) and not employed in The Shoulder Clinic.
Finally, based on our results, we cannot evaluate overall quality of ESPs and OSs sharing the task of examining patients with shoulder disorders. We have investigated – and shown positive results on - some of the indicators of professional and organizational quality, but it would be important to know the outcome of treatment for patients, the cost-effectiveness and the patient-perceived quality as well to fully evaluate the quality. Presumably, ESPs and OSs sharing the task of diagnosing patients with shoulder disorders has the potential to reduce costs, optimize use of specialist consultant competences, and increase quality – especially by using inter-professional competences in patients with unclear clinical pictures. However, to establish scientific evidence of quality in this broad range of areas, further research – especially based on high quality randomized controlled trials - are needed.