Applying GRADE-CERQual to qualitative evidence synthesis findings—paper 3: how to assess methodological limitations

The methodological approaches used in a primary study may have consequences for how much we can trust the findings from that study. Where there are concerns regarding the appropriateness of these approaches (e.g. data collection or analysis methods), or how the studies were conducted, study findings may be produced that are not an adequate representation of the phenomenon of interest. For example, we may have less trust in findings from a study where participants were recruited in a manner that did not fully address the aims of the research or where the data analysis methods were not appropriate for the study design.

One or more studies contribute data to each review finding in a qualitative evidence synthesis, and these data make up the body of data for a review finding. The methodological limitations of the body of data supporting a review finding are assessed as a whole to identify whether or not any methodological weaknesses within individual studies impact our confidence in a review finding. The methodological limitations for each review finding must be assessed separately since different studies contribute varying amounts of data to each review finding, and methodological quality issues may have varying impacts on different review findings. For example, the same set of studies may contribute data to many review findings. However, individual features of study design may have implications for some of those review findings, but not necessarily other review findings. Methodological limitations of the body of data may weaken our overall assessment of confidence in the review findings to which these studies contribute. See Table 1 for the examples of review findings with concerns regarding methodological limitations.

The extent to which it is possible or appropriate to critically appraise the methodological quality of qualitative research is contested among researchers in the field [22‐25]. However, the starting point for the CERQual approach is that there is a need for ‘clear evaluative criteria that are responsive to the unique nature of qualitative inquiry’ ([26] p. 113).

Despite the existence of more than 50 guidelines for assessing the quality of qualitative research [27], there is no agreement on the best approach for assessing the methodological quality of primary qualitative studies. Even where essential criteria for assessing methodological quality have been agreed upon, there are challenges related to the definitions underlying these criteria, and how much importance should be given to them within a critical appraisal tool [27] (p. 151).

The methodological limitations component of the CERQual approach requires some systematic and transparent approach for identifying methodological weaknesses of individual studies. Critical appraisal tools can help us to identify such weaknesses. We are, however, dependent on the quality and completeness of the individual study reports. The Cochrane Qualitative and Implementation Methods Group recommends that review authors consider a number of issues when choosing a tool to assess methodological strengths and limitations of qualitative studies, including the type of designs or methods selected to address the review question and the included primary studies [28]. In addition, the group advises using tools that privilege assessment of methodological strengths and limitations over the quality of reporting [29]. In the context of CERQual, we take a similar approach, and currently recommend that review authors use an approach that fits their review question and synthesis methods, with which they are familiar and comfortable using, and that focuses on methodological strengths and limitations. Regardless of the approach chosen, the review authors should provide a detailed and transparent assessment for each element of the tool. In line with Cochrane Qualitative and Implementation Methods Group guidance, we do not recommend using reporting guidelines as proxies for quality appraisal tools (see Table 2). To date, review authors using the CERQual approach have primarily used the Critical Appraisal Skills Programme (CASP) checklist or an adapted version of it [30]. Research is underway to examine which elements of critical appraisal are key for assessing the quality of research in the context of qualitative evidence synthesis and for use in the CERQual approach. See Table 2 for an outline of areas where further work is needed with respect to critical appraisal tools for qualitative research.

To assess methodological limitations of the body of data contributing to a review finding, you first need to choose an appropriate critical appraisal tool to assess the methodological strengths and limitations of the primary studies contributing data to the review finding. Regardless of the chosen tool, you will need to collect detailed information regarding the methods of data collection and analysis used in each study, as well as other aspects covered by the critical appraisal tool that you have chosen. The level of detail reported on the conduct of the included studies may vary greatly depending on the study design, the topic/field, type of publication or journal specifications.

When applying CERQual to a review that you have conducted, you will normally have gathered this information during data extraction, as this is a standard part of the review process. When doing so, you should present and explain in detail the assessments of each criterion within the critical appraisal tool for each primary study. Some review teams choose to present this as a matrix of methodological limitations of included studies. However, if you are applying CERQual to findings from somebody else’s review, you will need to have access to their critical appraisal assessments of the included studies, which are often published as part of the review. Where these assessments are not available, you may need to go directly to the included primary studies and assess the methodological strengths and limitations for each included study yourself. For more information on applying CERQual to findings from someone else’s review, see [7].

Once you have carried out your critical appraisal of the included studies, you can start to assess whether you have concerns regarding any methodological limitations of the body of data supporting each review finding.

Once you have assessed methodological limitations for each review finding, you should categorise any concerns that you have identified as either of the following:

You should begin with the assumption that there are no concerns regarding methodological limitations for the body of data contributing to each review finding. In practice, minor concerns will not lower our confidence in the review finding, while serious concerns will lower our confidence. Moderate concerns may lead us to consider lowering our confidence in our final assessment of all four CERQual components.

Where you have concerns about methodological limitations, describe these concerns in the CERQual Evidence Profile in sufficient detail to allow users of the review findings to understand the reasons for the assessments made. The Evidence Profile presents each review finding along with the assessments for each CERQual component, the overall CERQual assessment for that finding and an explanation of this overall assessment. For more information, see the second paper in this series [7].

In Table 1, we give examples of how methodological limitations can be assessed for a selection of review findings. These examples illustrate how different, and differing degrees of, methodological weaknesses can affect the overall assessment of methodological limitations for a review finding.

Concerns about methodological limitations may not only have implications for our confidence in a review finding but can also point to ways of improving future research. Firstly, where serious methodological limitations have been identified, this may indicate the need for future primary researchers to use more appropriate methods or to report their methods more clearly. You should also consider updating the review once this research is available.