A mixed methods evaluation of the Paediatric Musculoskeletal Matters (PMM) online portfolio

Musculoskeletal (MSK) presentations in children and young people are common with a reported prevalence of 1 in 8 [1] and a frequent cause of health care consultations increasing with age (6% of 7 year olds to 16% of 22 year olds in a cohort study from the UK) [2]. Delay in diagnosis and access to specialist care is a priority to address amongst parents [3]; unfortunately delay is often reported in conditions that present with MSK features [4‐9] with adverse impact on clinical outcomes [4, 5, 10]. The reasons for delay are multifactorial [9, 11] including complex care pathways involving community and various hospital care specialties (such as general paediatrics or emergency care) [8, 11] where self-rated lack of confidence in paediatric MSK clinical skills is reported [12]. Clinicians working in the community play a crucial role to suspect MSK disease and instigate specialist referral [2, 7, 8] and family medicine practitioners are often the ‘gatekeepers’ to specialist services [13]. Unfortunately many training schemes for family medicine do not include paediatrics [14], or MSK medicine [15] despite learning needs being known [16]. Other clinicians including nurses and allied health professionals (AHP) may encounter children in their clinical practice in the community, are important in the initial care pathways and yet also have unmet needs about MSK problems [17, 18].

With these challenges in mind our group developed a portfolio of e-resources (called the PMM Portfolio), to aid learning and give support to all clinicians who play an integral role in the early recognition, diagnosis and initial care of children with MSK conditions. The target audiences of the PMM Portfolio therefore include a spectrum of clinical learners who are not ‘paediatric MSK experts’; ranging from students in medicine and nursing, trainees in family medicine and paediatrics, through to practitioners in general paediatrics, family medicine, nursing and allied health. Paediatric rheumatologists are key to the teaching of others [19‐21] and especially teaching clinicians to whom children may initially present; such teaching is important to raise awareness, facilitate diagnosis and referral to specialist care. Paediatric rheumatologists are often the drivers of teaching paediatric MSK medicine at faculty and institutional level [19, 22, 23]; our evaluation aimed to support the PMM Portfolio as a valid tool to support teaching practice.

The PMM Portfolio encompasses the Paediatric Musculoskeletal Matters (PMM) website (launched 2014), the paediatric Gait, Arms, Legs and Spine (pGALS) app (added 2015) and e-learning modules (ELM) (added 2017) to aid signposting through the website. Figures. 1, 2 and 3 – Screenshots from the PMM Portfolio – PMM Website (Fig. 1), pGALS app (Fig. 2), ELM (Fig. 3).

The PMM website (www.​pmmonline.​org) is a free and open, evidence based, peer reviewed e-resource with user engagement to inform the content and website design functionality [24]. PMM Nursing (2017) is a further version of PMM to address the needs of the wider nursing community [17]. The PMM website includes an open, anonymised online survey and since 2014, 84 responses from 19 countries (data not shown but available on request), has informed PMM Portfolio development with requests for more global content and signposting. PMM International was hence developed and replaced the original PMM website in September 2018 following collaboration with paediatric rheumatologists in 11 countries (within Asia, Africa, Americas, Australasia and Europe) who developed additional content reflecting case mix and the health care in their country [25]. New content focused on infections and infection-related disease with MSK features, or differential diagnoses of rheumatic disease with further cases and images to reflect ethnic diversity. All contributions were subject to editorial review to ensure consistency of language and compliance with our governance framework [24, 25].

Paediatric Gait, Arms, Legs and Spine (pGALS) [26] is a simple MSK examination schedule useful in clinical practice [27‐33] and is widely taught [34, 35]. The free pGALS app (launched in 2015) was developed with medical students at Newcastle University UK to inform format and content with exam revision notes and links to PMM website key pages [36, 37]. Language translations of pGALS produced with our PMM International collaborators (to date 20 languages) and a version for telehealth (V-pGALS) are freely available on the pGALS app and PMM website (http://​www.​pmmonline.​org/​doctor/​approach-to-clinical-assessment/​examination).

The PMM E-Learning Modules (ELM) aid navigation through the PMM website and were developed with our multi-professional team (including family medicine doctors, paediatricians, medical students) [35, 37]. At the time of our evaluation there were three e-modules (https://​cpd.​ncl.​ac.​uk/​courses):

This study focuses on evaluation of the PMM Portfolio to describe and understand reach and impact on learning and clinical practice. Furthermore the evaluation aimed to inform future development of the PMM Portfolio and strategy to optimise impact.

We adopted a mixed methods approach comprising e-resource analytics, an online survey (including self-rated confidence before and after access to the e-resources) and telephone interviews to explore themes raised by the survey.

Google Analytics described access to the PMM website (site hits, page hits, accessing countries) and resource analytics described pGALS App Store data (country and number of downloads) and ELM data (country, number of users, clinician role).

An online survey using©Survey Monkey (see Additional file 1) explored how the PMM Portfolio e-resources impact on learning and clinical practice. The survey included sections about each of the e-resources and respondents were asked to provide feedback on all the resources (including ‘non-users’ – defined as those who had accessed some but not all the available PMM Portfolio e-resources to give insight into barriers to access). The survey was piloted and included Likert Scale questions (on perceived usefulness and impact on confidence on MSK knowledge and skills), free text options and open-ended questions to encourage comment.

Use of various recruitment sources was important to include a range of target audiences and to enable recruitment of users and non-users of the PMM Portfolio. A random selection of PMM website registered users (n = 450) and all the users registered for the ELM at the time of survey recruitment (n = 142, July 2019) were invited to complete the online survey in addition to purposive sampling amongst user groups within the UK and through our PMM global partners. Selected 1–1 telephone interviews followed the online survey to explore the findings in more detail. Interview participants comprised medical students (n = 2), clinical lecturer paediatricians (n = 2) and family medicine doctors involved in teaching alongside their clinical practice (n = 2); 4/6 were from the UK.

Written consent was obtained from interview participants and survey respondents consented to participation through an online response. All participant information was anonymised. Interviews were audio-recorded and transcripts anonymised before data analysis. E-resource analytic and survey data was analysed using descriptive statistics, with free-text comments and interview transcripts analysed following standard procedures for qualitative analysis, including open and focused coding, constant comparison and deviant case analysis [38]. Reflexivity was maintained throughout the analysis and writing, by recording, discussing and challenging established assumptions. A Paired T Test (©Minitab) compared self-rated confidence scores (Range: 1 not very confident – 5 very confident) before and after use of the e-resources. The study had ethical approval from Newcastle University, UK.

The PMM Portfolio was developed to address the reality that many children will present to a myriad of clinicians and in most health care systems, not directly to a paediatric MSK specialist. Many of the clinicians involved in the early stages of the care pathway have only minimal, if any, experience or training in paediatric MSK medicine [12, 14, 15, 40]. As such, they may not easily recognise which cases are appropriate for prompt referral to a paediatric specialist. This delay in referral, which results in a delay in confirmatory diagnosis and initiation of effective treatment, can significantly impact clinical outcomes. With the aim of improving access to the “right care”, the PMM portfolio improves awareness, knowledge and clinical skills of a broad target audience, which includes medical and nursing students, medical trainees, nurses, AHPs and family medicine clinicians. The inclusion of medical and nursing students is important to instill essential skills and knowledge early in their career path [18, 34, 41].

Our evaluation demonstrates that we are reaching our target audiences and the most popular pages of the PMM website are clinical assessment, limping child, MSK infections and indicators of when to be concerned and refer to a paediatric specialist. These knowledge themes reflect essential learning in paediatric MSK medicine for medical students [42] and family medicine [16] and are key to raising awareness, facilitating early diagnosis and referral.

The mixed methods described impact with quantitative analytics (e.g. the numbers of hits, number of users, most accessed pages). Qualitative methods enabled us to explore the experiences of the users; e.g. how they use the PMM Portfolio, the impact on learning and to understand what they felt was useful and why. Our recruitment method allowed us to reach users who had used some but not all of the resources in the PMM Portfolio (e.g. those who had used the website but not the app or e-modules). Such feedback was helpful to explain why some resources had not been used and how we can further increase our reach. The methods also allowed feedback, which was used to inform iterative development of the e-resources to optimise content and format to meet the needs of the users.

Many PMM Portfolio users reported positive impact on their learning, their clinical practice and teaching of others. There was a significant increase in self-rated confidence in clinical skills and knowledge following access to all e-resources. Positive comments related to ease of use, open access and the content being at an appropriate level (either for their own learning needs or those that they teach). There was variation in access to the e-resources across user groups; e.g. students and trainees primarily using the PMM website and pGALS app for clinical skills guidance and academic examination revision, AHPs reporting use of the ELM for clinical practice and clinicians (including paediatric rheumatologists) using the PMM website and pGALS app for their teaching of others. Many users accessed the e-resources for CME/CPD.

The international uptake of the e-resources reflects the wide stakeholder engagement in the PMM Portfolio design and development [17]. Most users of the PMM website were initially from the UK and US but over time, uptake has markedly increased around the world, especially amongst trainees and AHPs. Our global partners ensure relevance of content to the target audiences and provide pGALS language translations. The global partners facilitated dissemination with the countries of several PMM global partners ranking highly in those accessing the PMM Portfolio. Dissemination has also been facilitated through endorsement by professional societies (e.g. Paediatric Rheumatology European Society (PReS), Royal College of Paediatrics and Child Health (RCPCH), Royal College of Nursing (RCN), India Paediatric Society and the National Institute for Health and Care Excellence (NICE)). The PMM Portfolio is embedded in PReS Basic Courses for paediatric rheumatology around the world, postgraduate paediatric rheumatology training programmes for paediatricians (e.g. India and Kenya), NICE Clinical Knowledge Summaries for family medicine about paediatric MSK development, RCPCH guidance for postgraduate paediatric examinations, the RCN Competency Framework for nurses and the ‘Call to Action’ strategy of the Paediatric Task Force for Global Musculoskeletal Health [43].

We firmly believe that user engagement is integral to iterative development of e-resources and optimising impact as a means of knowledge transfer [44]. User engagement provided ideas to further increase reach and these included; 1) Increasing awareness of the PMM Portfolio amongst students, trainees and AHPs; notably lack of awareness was the most cited reason amongst non-users. 2) Integration of the PMM Portfolio in training programmes through links with training bodies and increasing their exposure at CME/CPD events. 3) More ELM with topics relevant for the global context - most ELM registered users reside in UK/Ireland suggesting more work is needed to promote these internationally. 4) Offering additional e-formats to enable offline access would facilitate further their uptake. 5) Maintaining open and free access; all the e-resources are free of charge other than one ELM (‘The paediatricians approach to a child with fever’), which had higher development costs and the charge may have contributed to the lower uptake. With these suggestions in mind, work is underway to develop a PMM app to enable offline access to the PMM website content, the ELM portfolio now includes a module targeting physiotherapists (focus on gait) and a further module for school teachers is planned. To date all author contributions have been forthcoming without financial reimbursement and we gratefully acknowledge the valuable input from all our PMM partners. Funding is a major barrier to future PMM Portfolio development and we are actively working to secure sustainability and growth whilst maintaining the ethos of PMM being free and open to all.

The PMM Portfolio is fulfilling an important educational role reaching many target user groups across the world. The PMM Portfolio continues to grow and engagement with users will facilitate future iterations maintaining relevance for the global context.