Introduction
Qualitative studies allow for a deeper understanding of people’s experiences, beliefs, attitudes or behaviours. These studies usually focus on
why participants think or act in a certain way, using open ended data gathering methods such as interviews, focus groups or observations [
1,
2]. They can be regarded as hypothesis generating research, and while research methods fundamentally differ when compared to quantitative research, they are not necessarily incompatible nor mutually exclusive. Both methods can complement each other, for example hypotheses that originated from qualitative research may be statistically tested in quantitative research, or findings from quantitative research can be explained by qualitative research [
3,
4]. As in all fields of research, poorly designed, conducted or reported qualitative studies can lead to inappropriate findings [
5].
In 2007, the COREQ (Consolidated criteria for reporting qualitative research) checklist was developed to assess the reporting quality of qualitative studies [
6]. Realizing that, in contrast to most other research fields, no widely used comprehensive checklist, nor uniform and accepted requirements for publication of qualitative research existed, the authors aimed to “…
promote complete and transparent reporting among researchers and indirectly improve the rigor, comprehensiveness and credibility of interview and focus-group studies.” [
6] Items from 22 published checklists were compiled into a single 32-item checklist and grouped into three domains (
research team and reflexivity,
study design and
data analysis and reporting), thus creating a comprehensive checklist covering the main aspects of qualitative research.
Though aimed at researchers conducting an interview- or focus group study, the COREQ also became frequently used in reviews on qualitative studies to assess the reporting quality of the included studies in the absence of a checklist specifically developed for this purpose. Qualitative reviews, a novel study design, aims to systematically synthesize the included qualitative studies instead of generating original data to achieve abstraction and transferability at a higher level beyond the included original studies [
7,
8]. While in 2007, when the COREQ was published, the number of qualitative reviews was relatively limited, in 2012 this number had increased substantially. Thus, using a similar approach as the COREQ, in 2012 members from the same research team and international experts developed the ENTREQ (Enhancing transparency in reporting the synthesis of qualitative research) checklist, for reviews as opposed to original studies [
9]. This 21-item checklist covers five domains (
introduction, methods and methodology, literature search and selection, appraisal, and
synthesis of findings) and aims to
“… develop a framework for reporting the synthesis of qualitative health research.” [
9]
Since the publication of both checklists, a large number of reviews of qualitative studies have been published on a wide array of topics. Though it has been argued that reporting checklists for qualitative research would not necessarily result in
better research [
10], and neither checklists were developed following the now accepted methods for developing reporting standards [
11], both the COREQ and the ENTREQ are now included in the EQUATOR network [
12], and are required by many clinical journals for submission; the high number of citations (respectively over 5.600 and 700 in Web of Science) indeed indicate usage. To this date however, no studies have been conducted to explore the uptake of the COREQ and the ENTREQ in reviews, or the effect on the reporting quality, which for guidelines in other research methods has been the case [
13‐
18]. Therefore, the aim of this meta-review is twofold: 1) to investigate the uptake of the COREQ and ENTREQ checklists in reviews of primary qualitative studies, and 2) to compare the quality of reporting of the original qualitative studies included in these reviews prior- and post-publication of the COREQ.
Discussion
In this meta-review, we explored the uptake of the COREQ- and ENTREQ-checklists in qualitative reviews, and compared the reporting quality of original qualitive studies prior- and post COREQ publication. Though reviews of qualitative research are a novel methodology to achieve abstraction beyond the original qualitative studies, we demonstrated an exponential publication trend over the past twenty years. By including 1.695 reviews, that in turn included 49.281 studies, we were able to present an in-depth overview of current qualitative research – both at the level of reviews, as well as the level of individual studies included within these reviews. Answering the first research question, we found that the COREQ, published in 2007 to score the quality of reporting of original qualitive studies, was used in 17% of the reviews to appraise the reporting quality of their included studies. The ENTREQ, published in 2012 specifically for systematic reviews, showed a better uptake with 28% of the reviews using the checklist. Finally, using the COREQ-scores of 2.775 studies within these reviews, we demonstrated a positive trend in reporting quality since the publication of the COREQ, with 13 out of the 32 signalling questions showing improvement.
The uptake of the COREQ in qualitative reviews may be explained by the original aim of the COREQ, namely to improve quality of reporting in original interview- or focus-group studies [
6]. In the absence of a comprehensive checklist for reporting the quality of qualitative reviews, the usage of the COREQ to appraise the reporting quality of studies within reviews may have followed naturally with the increasing numbers of qualitative reviews since its publication. The ENTREQ, specifically designed for reviews, showed a higher uptake [
9]. Yet, appraising qualitative studies remains a debated topic. While some argue that adhering to checklists improves transparency and validity of findings, others feel endorsement as a limitation, arguing that a ‘one size fits all’ -set of criteria cannot encompass the broadness of qualitative research as a whole [
5,
26‐
29]. In our study, this unresolved debate is clearly illustrated by the large number of reviews that adapted the COREQ for their purposes: more than half of the studies assessed their included studies with a selection of COREQ-items, or combined it with other checklists, both designed for reporting- or overall quality assessment, such as the CASP [
30], QualSyst [
31], GRADE-CREQual [
32], MMAT [
33], amongst others. The incomplete reporting, or the limited uptake of the COREQ and ENTREQ is not unique for qualitative research. For example, impact-studies on guidelines used for quantitative reviews [
19], clinical trials [
13,
34], observational studies [
15,
16], prediction- or prognostic studies [
14,
17], show that, even with endorsement of journals, the completeness of reporting remains suboptimal although for some, reporting quality improved.
By extracting the COREQ-scores of 2.775 appraisals included in these reviews, we were able to observe changes in the quality of reporting over time. On average, the total score, one of the three domains, and nearly half of the 32 signalling questions showed improvement when comparing studies published prior- versus post-publication of the COREQ. Though causal inferences cannot be made, this improvement, especially viewed in combination with the exponential trend of qualitative review publications, reflects the maturation and increasing acceptance of qualitative research. Although the overall quality of reporting improved, the scores of some items remained remarkably low: 16 out of the 32 signalling questions scored lower than an average score of 0.5. For example, in the first domain (“
research team and reflexivity”), the items “
experience and training”, “
relationship established” and “
participant knowledge of the interviewer” were reported poorly and did not improve markedly, with an average score of 0.25, 0.18 and 0.16, meaning that only 25, 18 and 16% of the articles reported these items, respectively. For the second domain (“
study design”), most items were reported better than in the first domain, and improvements were even stronger. Nearly all items improved, and almost half remained significant after Bonferroni correction for multiple testing. The third domain (“
analysis and findings”) showed good reporting on nearly all items, except for “
software” and “
participant checking”, though the first showed the largest improvement of all 32 items of the COREQ. These findings are in line with the two other studies that graded qualitative studies for the same purpose: Al-Moghrabi et al graded 100 qualitative studies, and demonstrated poor quality of reporting for most signalling questions [
31]. In the second study, Godinho et al confirms this poor completeness of reporting in 246 Indian qualitative studies [
24,
25]. When plotting the results over time, completeness of reporting remained modest, but increased over time, possibly facilitated by the publication of the COREQ and subsequent endorsement of journals [
30].
The strengths of this study are the large sample size and comprehensive search methods. We conducted our study on reviews of qualitative studies (i.e. a meta-review). This method allowed for exploration of checklist usage in the same study type, namely reviews. Furthermore, the original qualitative studies included in these reviews are independently assessed for reporting quality by the authors of these reviews, assuring independent quality assessment and allowing for a large number of study appraisals to be included. We aimed to include as many studies as possible, tin order to present a comprehensive overview of all qualitative reviews. However, because of this large sample size, we did not perform complete cross-checking at two levels: title selection and data-extraction. We did cross-check the abstract- and full-texts for inclusion, showing excellent agreement (Cohen’s kappa coefficient for inter-rater reliability of 0.86 and 1.00 respectively). Data-extraction was cross-checked for 10 reviews, showing no errors. Furthermore, nearly all COREQ-studies could be extracted directly by recoding the COREQ-tables to our format, instead of typing the scores in our datasystem, thus reducing the risk of errors. Next, though misclassification of study type could be a more serious issue (e.g. misclassify a qualitative study design as mixed methods), all authors used the same methodology to classify the study types, as detailed in the supplement. Another limitation related to the COREQ-score is selection bias: studies of higher quality may have been easier to find in database-searches than those that are of lower quality (e.g. because of the use of identifiable terms as ‘thematic synthesis’ or ‘grounded theory’), possibly resulting in overestimation of the average COREQ scores. Furthermore, some review authors might have excluded studies based on their COREQ-score, which will result in an overestimation of the COREQ scores. Since the publication of the COREQ and ENTREQ, various new checklists have been published, both for appraising the reporting- and the overall study quality (e.g. the CASP in 2013 [
30], the SRQR checklist in 2014, the eMERGe in 2019), underlining the developments in this research field since these guidelines. The use of these guidelines might partly explain the limited uptake of the COREQ and the ENTREQ, however we believe this to be to a limited extent since most reviews that did not use the COREQ or ENTREQ did not use any other checklist. Another explanation of the limited uptake may be improved retrievability of the post-COREQ and ENTREQ studies: including terms as ‘adhering to’, ‘appraising’, or naming these checklists likely increased the likelihood of inclusion in our review, compared to studies published prior these guidelines. Because of this, we based our search on previous studies [
22,
23], designed our queries together with an experienced medical librarian, and conducted iterative search methods, and we thus believe this effect to be minimal. Lastly, it cannot be inferred that differences prior- and post-publication of the COREQ and ENTREQ are causally related to the publication of these checklists.
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.