Background
Aim
How CERQual was developed
Assessing data adequacy
Data adequacy in the context of qualitative research
Data adequacy in the context of a review finding
A related concept to data adequacy is the concept of data saturation. In primary research, “data saturation” is often used to refer to the point in data collection and analysis when “no new themes, findings, concepts or problems were evident in the data” [29]. When used in this way, the concept of data saturation is clearly different from the concept of data adequacy as the former focuses on identifying new themes while the latter concept focuses on the extent to which an individual theme or finding is adequately supported by the data. Within grounded theory, the concept of data saturation is more ambitious, however, and “relates not merely to “no new ideas coming out of the data” but to the notion of a conceptually dense theoretical account of a field of interest in which all categories are fully accounted for, the variations within them explained, and all relationships between the categories established, tested and validated for a range of settings” [25]. This second use of the concept is closer to the concept of data adequacy as both focus on the extent to which the data has allowed us to explore the topic in sufficient depth. But there are also differences between these concepts. Researchers applying the concept of data saturation in the context of primary research use this concept as an ideal or goal when collecting and analysing data, and strive to collect new data until saturation has been met. When applying the concept of data adequacy in the context of a qualitative evidence synthesis, on the other hand, researchers assess data that has already been collected, and focus on identifying concerns with this data. As the process of data saturation is potentially limitless; and determining the point at which “saturation” has happened is difficult, if not impossible [26]; the concept of data adequacy may be a more pragmatic and feasible approach. |
Guidance on how to assess data adequacy in the context of a review finding
Step 1: collect and consider the necessary information related to adequacy (Fig. 2)
-
An overview of the data upon which each review finding was based
-
An overview of the number of studies from which this data originated, and where possible, the number of participants or observations. Information about the number of participants or observations supporting each finding may be difficult to gain from the individual studies. While most studies describe the number of participants they included in their study overall or give some indication of the extent of their observations, they may be less clear about how well represented participants are in different themes and categories. You can contact study authors for additional information, but they may not be able to readily provide this level of detail. In these cases, this lack of information should be noted, and your assessment of data adequacy will have to be made based on the information available.
Step 2: assess the body of data that contributes to each review finding and decide whether you have concerns about data adequacy
Example 1: minor concerns
A qualitative evidence synthesis explored factors affecting the implementation of lay or community health worker programmes for maternal and child health [12]. One of the review findings was relatively complex and explanatory in that it made claims about programme recipients’ attitudes towards the lay health workers and suggested factors that appear to influence these attitudes: “Programme recipients were generally very positive to lay health workers. Reasons for this included the respect, kindness and concern shown by lay health workers, and their non-dogmatic approach. Recipients also appreciated the similarities they saw between themselves and the lay health workers, either because they came from the same community or because they shared similar social backgrounds.” Twenty-five studies contributed to this finding. Ten of these studies described how recipients were generally positive to the lay health workers, but offered little or superficial information about the factors that appeared to influence these attitudes. However, nine of the studies gave more detailed and specific information about these factors. Based on an overall assessment of the richness of the data and the quantity of the data, we concluded that we had only minor concerns about data adequacy.
Example 2: serious concerns
Another finding from the same qualitative evidence synthesis made the following claim: “Recipients who lived near town and therefore had short distances to doctors preferred doctors to lay health workers” This finding was also relatively complex and explanatory as it suggested an association between where people live and their preferences regarding different groups of healthcare workers. However, the data upon which this finding was based offered very little information about this phenomenon, and it was not possible to properly explore or understand why doctors were preferred, and what role the distance to doctors played in people’s preferences. The finding was also only based on two studies. Based on an overall assessment of the richness of the data and the quantity of the data, we concluded that we had serious concerns about data adequacy.
Example 3: serious concerns
A second qualitative evidence synthesis explored the mistreatment of women during childbirth in health facilities [10]. One of the review findings described a relatively unexplored phenomenon as well as making a claim that was unexpected: “Studies from Benin and Sierra Leone suggest that either the mother or baby may be detained in the health facility, unable to leave until they pay their hospital bills.” Two studies contributed to this finding and the data that this finding was based on were superficial. While the finding was relatively narrow in scope, the small number of studies was of concern as the finding was unexpected and we were unsure of the extent to which studies undertaken in other settings or groups would have reported similar issues. The lack of rich data was also of concern as we were unable to properly understand this unexplored phenomenon. For instance, it was unclear from the studies whether women and babies were commonly detained, how long women and babies were detained for, and how they experienced this phenomenon. We therefore concluded that we had serious concerns about data adequacy.
Example 4: No or very minor concerns
A third qualitative evidence synthesis explored parents’ views and experiences of communication about child vaccination [7], and included the following review finding: “Parents generally found the amount of vaccination information they received to be inadequate.” Seventeen studies contributed to this finding. The data that this finding was based on were often relatively superficial. However, as the finding was a relatively simple, primarily descriptive finding, we concluded that we had no or very minor concerns about data adequacy.
Example 5: moderate concerns
The same qualitative evidence synthesis [7] included the following finding: “Parents want vaccination information resources to be available at a wider range of health services and community and online settings, for instance through schools, pharmacies, clinics and libraries.” Only four studies contributed to this finding and both had relatively thin data, which did give us some concern. However, we judged this to be a relatively simple and descriptive finding. We therefore concluded that we had moderate concerns about data adequacy. |
Assessing data richness
Assessing data quantity
Step 3: make a judgement about the seriousness of your concerns and justify this judgement
-
No or very minor concerns
-
Minor concerns
-
Moderate concerns
-
Serious concerns