Specify and calibrate condition sets and outcome set
Because one of our study aims was to assess the suitability of using QCA in a systematic review context, we used a completed review to determine whether data typically abstracted during a review would be acceptable to use with QCA. Thus, our initial approach was to rely on the review’s completed data abstraction files and published evidence tables. However, we adjusted our approach during the course of the analyses to verify and supplement previously abstracted data as we needed additional information not collected during the original review process.
Set calibration refers to the process of assigning a numeric value between 0 and 1 based on data collected from or about the case for each condition set and outcome set included in an analysis. These values are referred to as set membership values and represent the degree to which the case belongs to each of the sets in the analysis. Researchers typically define the rubric that determines what set membership value to assign based on existing theory or information external to the cases at hand. Qualitative and/or quantitative data collected from a case is evaluated against the calibration rubric to determine the specific set membership value that should be assigned to the case. In a crisp-set (cf, binary) calibration scheme, cases are either assigned values of “1” (fully in the set) or “0” (fully out of the set). For example, when trying to establish whether an adherence intervention belongs to the set of studies that are “theory-based,” one could examine whether the intervention designers described and cited specific behavioral theories that were used to develop the intervention; if so, the study would be assigned a 1, and if not, the study would be assigned a 0. Non-binary calibration schemes are also possible and are described in more detail in the online supplementary material (Additional file
1).
Studies in the completed review used a variety of medication adherence outcomes measured at various time points based on self-report, prescription fills, or medication event monitoring systems (“smart” medication bottles). Some studies used more than one measure of adherence. We reviewed abstracted data and original studies and determined that we would consider studies to be fully in the set of studies with improved adherence if at least one measure of adherence demonstrated a statistically significant improvement as compared to a usual-care comparison group. We chose this calibration rubric because of the lack of a common adherence measure across studies. We considered using a fuzzy-set calibration rubric, which allows for set membership values between 0 and 1; but, the panoply of adherence measures used both within and across studies and the lack of external standards for defining differences in degree of adherence (e.g., “very much improved adherence” from “slightly improved adherence” from “slightly not improved adherence”) proved too challenging.
Condition sets used in each analysis are summarized in Table
1. The abstracted data and evidence tables that described the BCTs and implementation features used in studies generally provided inadequate information to enable us to calibrate condition sets; thus, we went back to original study publications to obtain more detail and to clarify ambiguous data abstraction entries for nearly all studies.
Table 1
Conditions sets used in two qualitative comparative analyses (QCA) within an existing systematic review of medication adherence interventions
Analysis 1 Behavior change techniques useda
|
Increasing knowledge—provision of general information about behavior-related health consequences, use of individualized information, increase in understanding/memory enhancement |
Increasing awareness—risk communication, self-monitoring, reflective listening, behavioral feedback |
Providing facilitation—ongoing professional support, dealing with adverse effects, individualizing/simplifying regimen (fewer pills, fewer medications, less frequent dosing, timing of dosing to fit individual schedule), reducing environmental barriers |
Increasing self-efficacy—modeling, practice/skills training, verbal persuasion, coping response, graded tasks, reattribution of success/failure |
Supporting intention formation—general intention, medication schedule, goals, behavioral contract |
Increasing use of action control techniques—cues/reminders, self-persuasion, social support |
Changing attitudes—targeting attitudes toward adherence behaviors |
Supporting behavior maintenance—setting maintenance goals, relapse prevention |
Using motivational interviewing—client-centered yet directive counseling style that facilitates behavior change through helping clients resolve ambivalence |
Analysis 2 Implementation features |
Intervention agent—the entity interacting with the intervention target to provide the intervention, for example health care professional, research assistant, automated computer or phone agent |
Intervention target—the entity receiving the intervention, for example patient, provider, health care system, or combination |
Span—the total length of time (in weeks) over which the intervention was provided |
Mode of delivery—the mechanism through which the intervention was provided, for example in-person, over the phone, or virtually (online, text message, email, chat room, etc.) |
Exposure—the total dose of the intervention (in minutes) |
The BCTs abstracted during the completed review were determined and defined a priori by the review team and derived from a previous meta-analysis of medication adherence interventions and a published taxonomy of BCTs [
24,
25]. One study reviewer captured a study’s use of each BCT as “yes” or “no” or “unclear” based on information available in the published intervention description, and this was confirmed by a second reviewer. Thus, studies could be identified as using multiple BCTs. To studies that used a BCT, we assigned a set membership value of 1 for that BCT, and we assigned studies that did not use a BCT, or for which use of the BCT was unclear, a set membership value of 0. We also conducted sensitivity analyses with an alternate rubric that calibrated “unclear” as BCT use.
A challenge we encountered for the first analysis was the large number (12) of BCTs identified during abstraction in the completed review. With this many conditions, we were concerned about limited diversity that would result by including too many condition sets for the fixed number of studies (60). We winnowed the number of included condition sets to nine by eliminating three BCTs that were used by fewer than three studies. We attempted to further reduce the number of BCTs included in analysis by combining two BCTs to create a macrocondition, a typical strategy in QCA to reduce the number of included condition sets. However, we found the BCTs too conceptually distinct to combine into a single macrocondition. Thus, we could not implement a QCA standard of good practice with respect to keeping the number of condition sets relative to the number of cases at a reasonable level [
21].
For the second analysis, which evaluated implementation features, we specified condition set-based implementation features that the completed review authors determined a priori and captured during study abstraction. These features, listed in Table
1, included intervention
agent,
target,
span of intervention over time,
mode of delivery, and intervention
exposure. Information about these characteristics was captured by the review team using unstructured abstraction fields. For three of the condition sets, target, agent, and mode, the review team collapsed abstracted data into multivalue, mutually exclusive, categories for descriptive reporting of intervention characteristics.
We evaluated whether the multivalue categorical groupings for target, agent, and mode could be further collapsed into dichotomous categories for a crisp-set calibration rubric. For target, the review team used information from the published description to assign each study to one of three categories: patient-only, combination of patient and provider, combination of patient and provider and system. For our analysis, we decided that the inclusion of a provider or system target, in addition to targeting the patient, was a key distinction as provider and system interventions would require additional training, infrastructure, and expense. Thus, we considered a study as “fully in” for the target condition set if the intervention targeted a provider or system in addition to a patient. Studies targeting only patients were considered “fully out” of the set. Similarly for mode, we first evaluated the completed review’s categorical groupings before deciding that a key design feature relevant to policy-makers and practitioners would be whether the intervention was delivered in-person versus some other mode (e.g., telephone, virtual, automated) because of secular trends in virtual care, convenience to patients, and perhaps lower costs. We developed two alternatives to accommodate interventions with mixed modes, where some of the intervention was delivered in person and some delivered by phone or virtually. For calibration of the agent condition set, we considered studies that used licensed health care professionals (e.g., nurse, physician, pharmacist) as fully in, and studies that used agents described as research assistants, health coaches, or other non-licensed types of staff as fully out.
The calibration of the final two condition sets in the second analysis, time
span of intervention and intensity of
exposure, exemplified the iterative back and forth between theory and empirical information from the cases at hand that is a QCA standard of good practice [
21]. Study abstractors captured raw data about these two condition sets in an unstructured format during the review. We first transformed the raw data into standardized numeric values such that time span was represented in “weeks” from beginning to end of the intervention and the total time spent exposed to the intervention was represented in “minutes.” Because exposure information in some studies lacked detail, we made assumptions regarding average length of a clinic visit, telephone contact, or time spent exposed to an automated intervention when it was not specifically provided. For simplicity in interpretation, we chose to calibrate
span and
exposure with crisp sets. We contemplated various thresholds guided by the following considerations:
1)
Select the calibration threshold with some knowledge of the range of values represented within our studies to avoid setting it too high or too low such that most studies would be in or out of the set.
2)
Incorporate our substantive experience with behavioral interventions regarding what would be considered a threshold for a longer span or a higher exposure, but convey the condition sets using their numeric threshold value rather than terms such as low or high to mitigate concerns over the inherent arbitrariness of wherever we placed the threshold (e.g., span >12 weeks is “in,” rather than “long span” is “in”).
3)
Test alternative thresholds in sensitivity analyses to assess the robustness of our findings with respect to the placement of the calibration threshold.
Ultimately, our main analysis used a calibration threshold of greater than or equal to 12 weeks as fully in the span condition set and a threshold of greater than or equal to 120 min as fully in the exposure condition set. In sensitivity analyses, we evaluated a span threshold of 6 weeks and two exposure thresholds, 60 and 240 min. We identified some differences in findings, and all supplemental analyses were made available as appendices to the main substantive analysis to support transparency and demonstrate the sensitivity of findings to changes in calibration thresholds.
Construct and analyze the truth table
For each analysis, we transformed the raw data matrix of set membership values into a truth table, which places studies with the exact same configuration of set membership values for condition sets into the same truth table row. The number of logically possible truth table rows in an analysis is equal to 2
k
, where
k is equal to the number of included condition sets; thus, the truth table for the first analysis contained 512 (i.e., 2
9) rows and the table for the second analysis contained 32 rows (i.e., 2
5). In both analyses, some of the truth table’s logically possible configurations were not present in any studies so these rows are “empty” of any empiric cases and are called logical remainders. The truth table is the analytic device in QCA for determining which configurations of condition sets consistently demonstrate the outcome. If all studies within a truth table row demonstrate improved adherence, then that row is coded as fully in or 1 with a consistency of 100 %. Rarely do real-world phenomena exhibit perfect consistency. In QCA, rows with a consistency of less than 100 % (also referred to as contradictory rows) can still be coded as 1 and included in sufficiency analyses if row consistency is above a prespecified level. Different thresholds for consistency can be used based on the nature of the research question, data quality, and number of cases, but typical thresholds are between 75 and 90 % [
21].
Using the truth table created for each analysis, we identified set relationships between condition sets and configurations of condition sets and the outcome set. As described in the supplemental online materials (Additional file
1), superset relationships between condition sets and an outcome set can be interpreted as indicating necessary conditions. Similarly subset relationships between condition sets and an outcome set can be interpreted as indicating sufficient conditions. We used Stata Version 13 (StataCorp, College Station, TX) to create 2 × 2 contingency tables using set membership values for each condition set and the outcome set. Data from these tables are interpreted through a set-theoretic lens, meaning that the proportions produced by the table are interpreted as the consistency of each condition as a necessary condition for the outcome (% of cases in the outcome set that are also in the condition set) or as a sufficient condition for the outcome (% of cases in the condition set that are also in the outcome set). In the first analysis, we identified one BCT (techniques that increase knowledge) as individually necessary and one BCT (techniques that increase self-efficacy) as individually sufficient; in the second analysis, we did not identify any individually necessary or sufficient conditions.
Though an assessment of individually necessary or sufficient conditions is the initial analytic step, it is the evaluation of configurations of condition sets that allows QCA to offer powerful insights into complex causal patterns. For a configuration of condition sets to be necessary, it would need to be consistently present among all studies with the outcome of “improved medication adherence.” We did not identify two or more individual necessary condition sets in either analysis, and because formal logic prescribes that no configuration can be considered necessary unless each individual component condition set is necessary, we quickly discerned that we would not need an assessment of necessary configurations.
We used fsQCA version 2.5 to conduct sufficiency analyses for configurations [
26]. In crisp-set QCA, the configuration of set membership values in each row of the truth table where the outcome set is 1 represents as expression of sufficiency. In other words, if the outcome is consistently present among cases within the row, then that unique combination of condition sets (i.e., presence or absence of conditions in a crisp-set scheme) is a sufficient pathway to the outcome. If multiple truth table rows consistently demonstrate the outcome, then multiple sufficient pathways are present (i.e., an equifinal solution). The most complex expressions of sufficiency can be taken directly from truth table rows; however, these statements are often unwieldy in the number of conditions and operator terms (ANDs, ORs, NOTs), which makes them difficult to interpret. These expressions can be logically minimized to simpler expressions with fewer terms and operators that are still logically consistent with the more complex expression, but easier to interpret.
The fsQCA software uses the Quine-McCluskey algorithm to perform this minimization procedure. The basis of this minimization procedure is that if two truth table rows with the same outcome differ in set membership value of only one condition set, then that condition set is irrelevant for producing the outcome in that row and can be eliminated. The two rows can be merged resulting in a simpler expression of sufficiency. This algorithm is repeated such that all truth table rows are compared and reduced until no further simplification is possible. In actuality, three variants of the minimization procedure are used to produce three variants of a solution, the conservative, the intermediate, and the parsimonious solutions. These three solutions are all logically consistent with each other but represent different degrees of parsimony and differ with respect to whether logical remainders are used as part of the minimization procedure.
Ultimately, we identified seven sufficient configurations in the intermediate solution for the first analysis and four sufficient configurations for the second analysis. A summary of these results is in Tables
2 and
3. We computed parameters of fit to describe how well the set relationships we identified deviate from a perfect set relationship (i.e., consistency) and how well the solutions identified explain the outcome across all empiric cases included (i.e., coverage). See the online supplementary materials (Additional file
1) for additional information regarding parameters of fit.
Table 2
Summary of findings from analysis 1 evaluating combinations of behavior change techniques used by effective adherence interventions
Combination 1 | Increasing knowledge AND enhancing self-efficacy | 100 | 17 cases (50/44) |
Combination 2 | Using motivational interviewing AND not using facilitation | 100 | 4 cases (12/6) |
Combination 3 | Enhancing self-efficacy AND using intention formation AND improving attitudes AND not increasing awareness | 100 | 2 cases (6/0) |
Combination 4 | Using action control AND not increasing knowledge AND not using facilitation AND not using maintenance strategies | 100 | 1 case (3/3) |
Combination 5 | Enhancing self-efficacy AND using intention formation AND improving attitudes AND not using facilitation AND not using maintenance strategies | 100 | 2 cases (6/0) |
Combination 6 | Increasing knowledge AND using facilitation AND increasing awareness AND using intention formation AND using action control AND not using maintenance strategies | 100 | 2 cases (6/6) |
Combination 7 | Increasing knowledge AND using facilitation AND improving attitude AND not increasing awareness | 100 | 3 cases (9/9) |
Solutiona (%) | – | 100 | 76 |
Table 3
Summary of findings from analysis 2 evaluating combinations of implementation features used by effective adherence interventions
Combination 1 | Agent: uses staff other than licensed health care professionals AND Target: patients plus provider and/or systems | 100 | 6 cases (18/12) |
Combination 2 | Agent: uses staff other than licensed health care professionals AND Exposure: less than 120 minutes AND Mode: no face to face component AND Time span: more than 12 weeks | 88 | 8 cases (21/15) |
Combination 3 | Agent: licensed health care professionals AND Exposure: more than 120 minutes AND Mode: includes a face to face component AND Time span: less than 12 weeks | 100 | 4 cases (12/12) |
Combination 4 | Agent: licensed health care professionals AND Exposure: more than 120 minutes AND Time span: more than 12 weeks AND Target: patients only | 100 | 4 cases (12/12) |
Solutiona (%) | | 95 | 56 |