Development of a benchmarking toolkit for adolescent and young adult rheumatology services (BeTAR)

A systematic literature search was performed to establish any published standards or assessments for YP-friendly rheumatology services. All publications until June 2018 from four popular databases (PubMed, CINAHL, EMBase, and Web of Science) were searched. Initial search terms were obtained from MeSH (Medical Subject Heading) nomenclature and then mapped to appropriate database-specific terminology from selected publications (Table 1). Papers matching any combination of terms from the four concept areas in title, subject, keyword, and full text were retrieved for review. In addition, we searched websites of respected professional organisations for published guidelines and recommendations for YP’s healthcare to screen for additional standards. These included the following: Arthritis and Musculoskeletal Alliance (ARMA), British Society for Rheumatology (BSR/BSPAR), European League Against Rheumatism (EULAR), Paediatric Rheumatology European Society (PReS), American College of Rheumatology (ACR), Royal Australian College of General Practitioners (RACGP), Canadian Rheumatology Association (CRA), National Institute for Health and Care Excellence (NICE) and Department of Health and Social Care (DHSC).

Studies looking at quality of care for YP-friendly rheumatology services written in English were identified (Fig. 2). Studies were excluded if they were not generic and focused specifically on chronic conditions that are not related to rheumatology. From these data, quality standards for YP-friendly services that are relevant for rheumatology care were reviewed by the core study team (RAC, DE, RT and YI). This helped define themes relating to what constitutes good healthcare provision for YP.

YP (12–24 years old) with chronic inflammatory rheumatic diseases cared for at a pediatric, adolescent and young adult rheumatology hospital unit in London were approached to take part in the study for a mixture of FGs and one-to-one interviews. All sessions were conducted, taped, and transcribed by the same researcher (RAC), who was not involved in participants’ clinical care. YP’s thoughts on these previously published standards of care were explored. They were asked what criteria need to be met by a service to provide the best possible care, and whether they felt any new standards or areas of care needed to be considered. Lastly, YP were asked to generate specific questions and response items for each criterion. As a result of this process, the toolkit developed comprised of two versions, one to be completed by HCPs and one to be completed by YP (Fig. 3). This procedure was repeated until saturation was reached and no new information and further criteria were suggested from YP [27]. The structure of the FGs and in-depth interviews are shown in Additional file 1. YP were then asked to prioritise all criteria based on their order of importance using a diamond ranking exercise, and each criterion was assigned a rank-order based on their ranked position [28]. YP also rated each criteria on a 5-point scale (1 = not important at all, 5 = very important).

A multidisciplinary group of rheumatology HCPs reviewed the standards and rank-order assigned by YP. The process was completed through online surveys, followed by a face to face consensus meeting. Key areas covered were comprehensiveness, clarity, relevance, and feasibility of use for the proposed HCP toolkit. If ≥80% of HCPs endorsed a proposed standard or a way of combining multiple criteria, then this standard was included in the toolkit. If a criterion reached < 80% consensus it was discussed in the next phase with YP. These thresholds were also used to evaluate the question and response items, as well as the rank-order of each criterion. Lastly, HCPs were asked about the challenges of using the toolkit in practice.

Phase 4 aimed to refine the toolkit by accounting for any multi-site variation, and to develop a quantitative scoring system. YP and their parents/carers were recruited from rheumatology outpatient’s clinics in Sheffield, Newcastle, Northampton, and London to participate in either FGs or one-to-one interviews. We chose these places to maximize variations in participants’ experiences by involving health services from larger, more ethnically diverse cities where less than 60% of the population are from the White ethnic group (London), as well as from smaller, less diverse cities with a more than 80% (Newcastle and Northampton) and 90% (Sheffield) of the population from the White ethnic group.

The 1000minds software [29], which is a decision-making program using the Potentially All Pairwise RanKings of all possible Alternatives (PAPRIKA) method [30], was used to develop the scoring system for both versions of the toolkit. Participants were first asked to review the toolkit’s content and were then given repeated comparisons between two service criteria through 1000minds. They were asked to choose which of the two service criteria is more important or whether they are equally important (Fig. 4). Each criterion received a ‘preference value’ according to the PAPRIKA method; criteria with greater importance received higher preference values and were thus assigned more points than criteria with lower preference values.

During this process, the rank-order of each criterion proposed by YP in phase 2 and by HCPs in phase 3 were also discussed to encourage YP to reassess their judgements in light of the comments and feedback provided by other participants [31]. Relative weights of the criteria were used to develop a scoring system such that when all item responses are considered together, the maximum score possible for each version of the toolkit is 100 and the minimum score is 0.

Phase 5 aimed to evaluate face validity and test the usability of the toolkit through individual cognitive interviews (CI). Sample size used for cognitive interviews are normally small, and five participants can be sufficient [32‐35]. Purposeful sampling [36] was used to ensure HCPs from various rheumatology centres and YP with diverse disease characteristics and demographics were included. Participants were interviewed in person or over the telephone. A mixture of think-aloud and verbal probing techniques were used in order to gain maximal information and to encourage participants to talk about any words or concepts that they found troublesome (see Additional file 2) [34, 35]. For any item that participants judged as ‘medium’ or ‘hard’ to understand, they were asked to suggest ways to reword the item to improve comprehension. The data gathered were used to modify and improve question items and response choices.

Qualitative responses from FGs and one-to-one interviews were taped, transcribed, and analysed using content analysis to identify key themes and categories [37]. To reduce the chance of bias, transcripts were reviewed by two researchers (RAC and DE) to identify major themes [38]. Differences were resolved by discussions with two more researchers (YI and HC) until consensus was reached. Data analysis was carried out using NVivo software [39] for coding data with similar contents into meaningful categories and overarching themes. In addition, content analysis was performed according to centre and age group (12–15 years and 16–24 years).

Out of 95 YP who were approached, eight did not participate either because they did not have time (n = 3) or were not interested (n = 5). A total of 87 YP (21 males) aged 12–24 with rheumatic diseases and five parents/carers of YP participated in the study (Table 2). The median age of diagnosis was 10.5 (range 1–16) and median disease duration was 9 years (range 1–22). Out of 29 rheumatology HCPs who were approached for the study, 26 agreed to take part. The median number of experience in adolescent rheumatology was 11 years (range 1–34).

The literature search identified 1953 articles; after screening for full text and duplicates, 285 articles remained. In total, 48 distinct criteria pertaining to quality of healthcare services for YP in rheumatology services were extracted from these articles and were grouped into 6 themes: provision of information or education, preparation for transition to adulthood, staff expertise/support, YP involvement, service efficiency, and service accessibility. All these themes were discussed with YP in the following phases.

Two FGs (n₁ = 5, n₂ = 3) and 30 one-to-one interviews were conducted with YP. During this phase, YP suggested 17 new criteria that included: providing specific pain-management information; self-injection of methotrexate taught by a rheumatology nurse; and sharing and explaining test/assessment results (e.g., blood tests). All 17 of these new criteria proposed by YP, as well as all the criteria and standards derived from previous literature, are listed in Additional file 3. The five most important criteria according to YP were: HCPs’ expertise/knowledge in adolescent rheumatology, accessing effective therapies and treatments; timely access to treatments; monitoring symptoms, and accessing urgent consultations. All 65 criteria were included in the preliminary version of the toolkit. YP also generated a list of questions and response items to define and assess each criterion. Moreover, YP suggested a YP version of the toolkit to assess the quality of care from the patient’s perspective.

Fourteen HCPs participated in an online survey and eight HCPs attended the consensus meeting. The group agreed on 18 of the original criteria to be included in the toolkit and 42 original criteria were combined into 13 new criteria. For example, providing out-of-hour appointments and phone/skype consultations were combined into one inclusive criterion for “convenient appointments”. Five criteria were excluded as they were ranked low in importance by YP and > 80% of the HCPs voted in favour of excluding them. These were: providing information on nutrition, providing information on alcohol and drugs, providing information on sexual health, accessibility for patients with physical disability, and easy access to the hospital by public transport. HCPs also suggested higher rank-orders for developmentally appropriate care (e.g., providing transitional care plan) and for providing easy ways to contact the service with timely responses, which were ranked relatively low by YP.

Moreover, HCPs agreed to remove 18 out of the 105 question items that are likely to generate the same responses from all services, and hence are not discriminatory. These include whether or not services provide blood monitoring for anti-inflammatory treatments and whether or not services send out clinic letters to patients. Forty-seven items were kept without revision and 40 items were either clarified to improve understanding or reworded to better identify challenges and issues with care provision. HCPs also discussed differences between adult and adolescent/paediatric care, and how the final item pool needs to account for these variations. As a result, two additional response items were added for services seeing YP under the age of 16, which are whether services provide educational information and resources to parents/carers and whether service policies (e.g., discharge policy) are explained to parents/carers. The revised toolkit at this stage contained 30 criteria with 89 question items for paediatric services seeing YP under 16 years old, and 87 response items for services seeing YP who are 16 years old or older (Fig. 5). Out of these 30 criteria, YP selected 11 criteria to be included in the YP version of the toolkit (see Additional file 4). These modifications were discussed with YP during phase 4.

Lastly, HCPs agreed with the need for a YP facing version of the toolkit in order to address problems with care identified from both patients’ and providers’ perspectives. Both versions have similar structures with three levels of assessment. Level one includes overarching domains made up of individual criteria (level 2), which are assessed using different question items (level 3). The overarching domains from level 1 are: assessment and treatment, information and involvement, accessibility and environment, and continuity of care.

Seven FGs with 27 YP (n₁ = 5, n₂ = 3, n₃ = 3, n₄ = 4, n₅ = 4, n₆ = 5, n₇ = 3) from four cities across the UK and 5 one-to-one interviews with parents/carers of YP were conducted. Participants from this phase discussed the final criteria and question items, and agreed with almost all adaptations made by HCPs in phase 3. YP were also asked to complete the 1000minds programme collaboratively as a group during each FG. They were presented with an average of 156 pairwise-ranking for the HCP version, and an average of 78 pairwise-ranking for the YP version.

The mean preference values (derived using 1000minds) from all 7 FGs ranged from 0.3 to 6.2 for the HCP version in which 0.3 reflected the lowest and 6.2 the highest importance (see Additional file 5). For the YP version, the mean preference values ranged from 3.7 to 13.9 (see Additional file 4). Intra-class correlation estimates of these preference values based on mean-rating of the FGs (k = 7), absolute agreement, and 2-way random-effects model were high (0.89), indicating good agreement between YP regarding the priority weighting given to these items. A preliminary scoring system based on the mean preference values was developed. This scoring approach across multiple domains allows for a good score in one domain to compensate for a poor score in another domain.