Background
Palliative care (PC), a relatively young specialty, is growing rapidly and will continue to do so over the next decades [
1,
2]. The values of PC, such as adequately controlling symptoms, alleviating the burden of patients and informal caregivers, and preventing unnecessary hospitalisations [
3,
4] have been presented in an increasing number of scientific publications [
5‐
7]. Clinical practice is preferably guided by a sufficient body of high quality evidence from research in combination with clinical expertise and patients’ preferences [
8]. To inform evidence-based guidelines and protocols, the need for literature reviews in PC is pressing. Literature reviews summarise and appraise the best available evidence on a topic and are considered the highest quality of evidence for evidence-based medicine [
9,
10].
Widely used methods for literature reviews are developed primarily for intervention studies and have been applied to other fields, including PC. However, there is a need for literature reviews in PC beyond those that seek to pool evidence from intervention studies. The methods used for reviews concerning evaluation of interventions may not be transferable to literature reviews on less clearly defined topics that involve different challenges [
11]. One of the challenges in PC is to build review questions based on the four parts of the PICO framework (Patient-Intervention-Control-Outcome). The challenge for PC is characterised by the wide range of domains due to variations in patient characteristics, disease trajectories, stages of illness, management of treatments, and involved stakeholders, which leads to a variety of topics, such as symptom management, psychosocial care, decision-making, and health services [
1,
6,
7,
12]. A developing discipline such as PC often uses explorative review questions to gain a better understanding of the topic of interest, for example: ‘How do patients with chronic heart failure experience an exercise programme to reduce illness related fatigue?’. The heterogeneity in practice and the explorative nature of the questions have hampered the use of PICO, which should be considered by a researcher when developing the review plan. Different frameworks have been developed to handle this variation, such as SPICE (Setting, Perspective, Intervention, Comparison, Evaluation) or SPIDER (Sample, Phenomenon of Interest, Design, Evaluation, Research type), but the aforementioned challenges remain [
13‐
16].
After formulating a review question, the next stage of study identification has its own challenges. A young discipline such as PC often suffers from concepts and terms that are heterogeneous, poorly defined, indexed, or standardised, making term-based searching difficult. This is not unique for PC, as similar problems have been encountered in social sciences [
11,
12,
17]. Consequently, indexing systems such as MeSH (Medical Subjects Headings, the controlled vocabulary thesaurus of MEDLINE) do not cover many key concepts within PC. Furthermore, most general bibliographical databases only publish the author written abstracts together with independently annotated indexing terms. However, relevant information for PC review questions is not always part of the original study objective or is only presented as a subtopic and not reflected in the abstract. In these cases, a perfect match search based on the elements of the review question will not be sufficient to retrieve relevant studies. Therefore, a different approach for identifying key representational features within abstracts to discover these articles needs to be employed. Taken together, poor indexing, and the heterogeneous use of terminology will result in an unbalance between specificity and sensitivity. To specify, either ineffective searches missing many relevant articles or inefficient search strategies resulting in very high numbers of search results, tens of thousands, that must be screened manually. To narrow down results in intervention studies, a component on study methodology is added to the search query. However, most research within PC cannot be answered by randomised controlled trials, [
18‐
20] rather, it relies heavily on alternative study designs such as mixed methods and qualitative studies [
12,
21]. Since the preferred study design is not always clear at the start and most research papers poorly report the applied methodology, the use of methodological search filters has been contested [
22]. Although some success using filters has been reported, the broad terms used will yield low-precision results and, therefore, a high number of needed-to-screen (NNS) [
22]. This phenomenon has also been seen in fields such as diagnostic accuracy [
23].
Although the Boolean search query is most widely used in literature reviews, it is not the only way of retrieving studies or finding information. Other retrieval methods, including berry picking (Table
1), pearl growing (Table
1), and snowballing, have their own strengths and weaknesses. Berry picking is difficult to reproduce and lacks transparency, but has the advantage of gaining knowledge and identifying knowledge gaps with each item (berry) found. Pearl growing can help in identifying the relevant phrases and indexing terms used within the field, but is highly dependent on the composition of the initial set. Using the knowledge of peers regarding the relevance of studies, can reveal information not available in the abstract, but runs the risk of bias towards the predominant view within the field. For literature reviews, transparency and reproducibility are key features and, therefore, the Boolean logic query is so popular, as it is transparent in what it does, all elements are visible, and it is reproducible.
Table 1
Search techniques and analytic tools
Berry Picking | Berry Picking is a retrieval model where obtaining evidence is not a linear path, but an iterative process where each newly identified piece of information can result in a modification of the information base required. Various techniques are used to identify the next piece of relevant information such as footnote chasing, journal browsing or database searching. Where it differs, is that information is not returned as a complete set, but in bits and pieces (the berries) informing the information base as one goes along [ 24]. |
Pearl growing | In the process of pearl growing, relevant articles to the topic of interest are identified and they enable researchers to isolate keywords and index terms on which the researchers can base their search. By using these identified keywords and index terms to build the search, the corpus of relevant articles will grow. This process is repeated for all initial papers and newly identified relevant papers for either a predetermined number of times or until no new relevant papers are identified [ 10, 25]. |
Citation tracking | For citation tracking, researchers search for all articles which were cited by relevant articles (backward citation tracking) and for all articles which cite the relevant articles (forward citation tracking). Every found reference has been deemed relevant after careful consideration by the researchers. As such, researchers make use of the ‘knowledgeable crowd’. That is, a corpus can grow through citation tracking based on the knowledge present within the literature by peers based on their knowledge and judgement of the content of the full article [ 11, 22]. |
‘Golden bullets’ | ‘Golden bullets’ are articles that align with the inclusion criteria for the systematic literature review and, therefore, undoubtedly should be part of the review. The ‘golden bullets’ are used for feature extraction to inform the Boolean search strategy. Furthermore, the ‘golden bullets’ are used in the validation test of the search. During the validation, the reviewer is checking whether the ‘golden bullets’ are included in the outcome of the search, ensuring a suitable search strategy to identify relevant studies. |
Software | During the iterative method, some text analysis tools can be used. For instance, during the analysis of the ‘golden bullets’ the analysis tools present in Eppi reviewer [ 33] can be used. A possible tool for word frequency is the TF*IDF option, which helps to identify relevant terms and PubReMiner ( http://hgserver2.amc.nl/cgi-bin/miner/miner2.cgi). PubReMiner is an online resource to which PubMed search queries can be submitted to produce a list and frequency counts for all keywords (subheadings, title-words etc.) and MeSH-terms associated with the articles in that query. Swift-review is an interactive workbench that provides numerous tools to assist with literature prioritization. The software utilizes recently developed statistical modelling and machine learning methods that allow users to identify over-represented topics within the literature corpus and to rank-order titles and abstracts for manual screening [ 34]. To identify multi-word phrases, n-grams, the Termine tool can be used [ 35]. For identifying concepts within the ‘golden bullet’ set, it can be helpful to use cluster analysis [ 36] within Eppi reviewer, which is an application of the Lingo3 engine. Results of the search can be loaded in Endnote X7 (or any other suitable program for managing references) for deduplication. In the absence of Eppi reviewer a plethora of tools is available on the web like voyant-tools ( https://voyant-tools.org) for term frequency analysis, termine on the web for n-grams ( http://www.nactem.ac.uk/software/termine/) and vos-viewer for cluster analysis ( http://www.vosviewer.com). For more information see http://systematicreviewtools.com. |
To address the aforementioned issues, there is a need to combine several of the existing retrieval methods in a logical way to ensure transparency and provide guidance for researchers. To reflect the more iterative nature of searching for PC studies, we developed a pragmatic framework, Palliative cAre Literature rEview iTeraTive mEthod (PALETTE), to guide the fine-tuning of the review question, performing a literature search, and applying screening eligibility criteria. By introducing intermediate validation steps, the reasoning for going from one phase to the next within the framework becomes visible which increases the transparency. It is the combination of these iterative steps, the use of multiple retrieval methods, and the validation on evaluated suitable studies that will boost confidence by the researchers that all relevant studies are captured. The structured iterative manner also facilitates a better ability to trace-back decisions for re-evaluation in light of new discoveries and adjust when or where necessary.
In this paper, we assess the usability and performance of PALETTE on two literature reviews in PC. Furthermore, with a detailed description, we provide guidance on how to apply PALETTE for literature reviews in PC.
Discussion
Constructing relevant, focussed review questions in PC is a daunting task and requires an intricate knowledge of this field and all its actors. The same applies to the terminology used and the ability to identify all relevant studies. To address these issues and the shortcomings of the current literature review methodology, mainly developed for intervention studies, we present PALETTE as a pragmatic framework, which encompasses multiple retrieval methods applied in an iterative transparent way. Although the different techniques used within PALETTE have been around for some time, we provide a framework to use these in a transparent and coherent way with a clear decisional tree. As such, we provide guidance for researchers in the field of PC as well as in other specialties challenged by explorative questions, heterogeneity, and poorly defined keywords and concepts when conducting a review. Not every single technique will lead to a proportional number of relevant articles in every review; however, using PALETTE ensures a high likelihood of retrieving relevant articles with confidence.
The introduced iterative method results in four main positive aspects. Firstly, because of the more qualitative nature and the poorly defined concepts, review questions in PC need preliminary exploration. If not, researchers run the risk of missing a related concept not envisioned at the beginning. When applying the more iterative approaches such as berry picking and pearl growing solely, [
24,
25] it is difficult to maintain transparency concerning relevant article identification and introduces the possibility of bias. By having a clear framework, such as PALETTE with the precise reporting of each step, we overcome this problem and provide the researchers with an opportunity to evaluate the process. This is in line with the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines, which underline the importance of transparent reporting [
26]. In addition, the PRISMA flowchart can be complementary to PALETTE. To illustrate, once the final search string has been developed, the steps in PALETTE (phase 4) are comparable with PRISMA and can be reported according to the PRISMA flowchart.
Secondly, as compared with the search building methods in intervention studies, PALETTE enables the research team to provide input on opinions and views, which in-turn enables them to explain what works for whom, in what contexts, and why in a transparent manner [
27,
28]. This is necessary for an in-depth understanding of the content of the topic in the still poorly defined field of PC [
29].
Thirdly, the total body of evidence in an article on PC is not well captured in terms. Therefore, validation is required on an article level. By checking the ‘golden bullets’, PALETTE grants this opportunity and validates the literature search on content and not just on the presence of keywords. This technique within PALETTE results in a representative set of articles.
Lastly, PALETTE might offer greater proportionality between the efforts of the researchers and the results of the literature search. When using a Boolean logic search query based on the initial review question and using every conceivable terminology on its own, some of which are quite ambiguous, huge amounts of results (10s of thousands) have to be screened manually and highly relevant citations are still missed [
30]. The literature search in PALETTE is guided by the keywords and the content of studies that undoubtedly should be part of the review (‘golden bullets’) to find an optimal balance between specificity and sensitivity to keep the NNS manageable. This became apparent in the comparison between the recommended search method for reviews of intervention studies and PALETTE for LR1 and LR2 in which the NNS decreased for both LRs whereas the number of relevant articles increased with the application of PALETTE. Additionally, the kind of evidence researchers are often looking for when performing a review in PC aims to discover the variety of experiences or all opinions. Therefore, it is less critical in comparison with studies about a specific intervention when not all studies are identified. A view does not necessarily gain importance with the number of studies found [
22].
Four limitations of PALETTE should be considered when applying PALETTE. Firstly, regular feedback within the research team is necessary to fine-tune the review question and to keep focussed on the aim of the review. Secondly, care should be taken when compiling the ‘golden bullets’. The ‘golden bullets’ should reflect the topic well from multiple angles so as to not introduce a skewed data set. By combining wisely chosen experts with the initial literature search and the expansion of articles, the risk of a skewed data set can be avoided. Thirdly, the benefit of the involvement of experts was limited in our examples. In the literature, different opinions regarding the involvement of experts are evident [
22,
29]. We argue that although time-consuming, the involvement of experts should remain a component of PALETTE. Especially because the involvement of experts could be valuable due to the experts’ intricate knowledge of their topic and their ability to identify key articles (potential ‘golden bullets’). The value of the involvement of experts could however depend on the content of the review. Finally, to ensure the quality of the iterative literature search, researchers should preferably collaborate with an information specialist. In such a collaboration, researchers can provide the information and specialist experience of clinical practice to explain concepts whereas the information specialist can contribute to the literature search with his/her knowledge about the most optimal way of retrieving data from the sources, including which software to use to optimise the literature search (Table
1). Therefore, the collaboration provides the ultimate opportunity to combine knowledge of practice and knowledge of software and techniques used during the literature search, as also stated by Beverly et al. [
31].
Some strengths and limitations should be taken into account. PALETTE is a new approach that can be helpful in performing literature reviews in PC. However, we still have limited experience with the application of PALETTE and compared minimal results between PALETTE and the recommended search method. We, for instance, did not measure the costs in terms of time needed for each phase of PALETTE. Regarding the time needed, we know from previous research that an experienced reviewer can screen an average of two abstracts per minute, but abstracts for complex topics may take several minutes each to evaluate [
32]. Given the decrease of NNS when using PALETTE, we hypothesise, that a significant amount of time will be saved in the sub-phase of ‘identification eligible articles’. Knowing these strengths and limitations of this study, we encourage researchers to use PALETTE and to evaluate the time needed for and the value of this method.