Health Information Technologies—Academic and Commercial Evaluation (HIT-ACE) methodology: description and application to clinical feedback systems

This article presents (1) a theory-informed, structured methodology—Health Information Technologies—Academic and Commercial Evaluation (HIT-ACE)—to support compilation, empirical comparative evaluation, and synthesis of available information for HIT, as well as (2) preliminary results from a case example application to MFS. In brief, this methodology integrates information from the scientific literature, as well as publicly available commercial sources, and draws on existing theories and frameworks to conduct a competitive analysis. The competitive analysis method is drawn from marketing and strategic business management [24] to structure and prioritize results. Competitive analysis drives innovation by identifying the strengths and weaknesses of existing products within a particular domain, but has not yet been utilized in implementation science.

The current example application of HIT-ACE is specific to MFS used to support ROM in behavioral health. Consistent with the literature in the USA, where this project was carried out, we use the term “behavioral health” as an overarching term for mental health and substance abuse services [25]. This scope allows for a broad (in that it applies to multiple settings in which MFS have been developed and implemented) yet specific exemplar of the HIT-ACE methodology. The methodology is intended to be generalizable to a broad range of extant HIT (e.g., electronic health records, personal health records, mobile health apps, patient/disease registries, clinical-decision support tools) to advance their theoretical and empirical basis and, ultimately, their capacity to advance implementation science and practice.

The first phase of HIT-ACE is focused on developing a theory-driven, objective coding scheme for the capabilities of a given technology. A capability is defined as the ability to perform or achieve certain actions or outcomes through a set of controllable and measurable faculties, features, functions, processes, or services. A characteristic is defined as distinguishing trait, quality or property. Phase 1 is intended to address descriptive research questions surrounding the nature of a particular type of technology (e.g., How many different products have been developed that may be classified as this type of technology? What are the most common capabilities of the identified technologies?). Although Phase 1 identifies the frequency with which different capabilities and characteristics are present within a given class of technology, no specific weighting is given to more or less commonly occurring aspects. This is because the presence of a capability in a system may be influenced by a multitude of factors (e.g., when and by whom it was designed) and is not necessarily reflective of the importance or effectiveness of that capability. Further, while it is practically useful to gather information about system characteristics (e.g., price, languages available), it is critically important to gather capabilities which are then ultimately linked to putative mechanisms of practice change (phases 3 and 4). The HIT-ACE methodology draws capabilities from multiple sources—including theory, expert opinion, and review of existing systems—with the goal of identifying a broad set of capabilities for subsequent empirical evaluation.

The creation of the coding scheme is integral to Phase 1 of HIT-ACE. To ensure its coherence, generalizability, alignment with existing literature, and potential for aiding the identification of core capabilities of HIT and associated implementation mechanisms, code development should be guided by theory representing at least three levels: (1) theory specific to the intended functions of a given class of HIT, namely its intended mechanisms of influence on user behavior in the service of optimizing health outcomes; (2) general frameworks for developing usable, effective, and compelling technology products; and (3) implementation models, focused on the ways in which a technology and its associated supports can facilitate initial adoption and sustained use within service systems. As an example, we present the theoretical foundations for the HIT-ACE methodology applied to MFS below. However, the frameworks identified are not intended to be prescriptive, as specific technologies may be best evaluated using different models.

Phase 2 of HIT-ACE is intended to confirm the results of phase 1 coding and gather more detailed information about system development, capabilities, and implementation processes than is often possible from the publicly available sources. Phase 2 also expands on Phase 1 with additional descriptive research questions (e.g., Through what processes—and with what types of stakeholder input—were these technologies developed?). Most consumers are unlikely to move beyond the type of information gathering reflected in phase 1 (albeit in a less systematic way), making phase 2 data especially important to aid stakeholders in making informed adoption decisions. It is likely that developers will have additional information that may contribute to the most accurate depiction of a particular HIT landscape. Phase 2 involves a semi-structured interview—based on identified theories—which gathers specific information about the spread of the identified HIT product. Prior to conducting phase 2 interviews, summaries of phase 1 coding are distributed to respondents for review in order to facilitate clarification and correction of the information collected. Additional interview content in phase 2 aligns with the three domains and frameworks described above (i.e., those specific to the intended functions of the HIT; those focused on design and usability; and those emphasizing successful implementation).

In the third phase of the HIT-ACE methodology, specific system capabilities are linked to implementation mechanisms to support future empirical testing (in phase 4). Identification and evaluation of mechanisms of change has become increasingly common in intervention studies [26], but comparable efforts within implementation science are practically non-existent. Mechanisms explain why or how implementation strategies, such as HIT, exert their effects and are critically important for understanding their heterogeneity and contextual dependence [27]. Phase 3 is designed to address exploratory questions surrounding the putative mechanisms through which aspects of a technology affect implementation outcomes (e.g., What mechanisms might be responsible for the anticipated—or previously measured—impact of the technology? Which system capabilities are likely to influence which mechanisms most directly?). Capabilities that are commonly occurring and those present in only a few systems should both be considered during phase 3.

Recent studies have used expert consensus to improve the specificity with which implementation strategies are articulated [15, 28], but very little empirical evidence about the mechanisms through which they operate is available to drive selection or prioritization [29]. For these reasons, phase 3 reflects a Discovery and Linking Process with the goal of (a) identifying the mechanisms through which a specific class of HIT exerts its effects on service provider behavior to optimize service recipient outcomes and (b) linking those mechanisms to the capabilities identified in phase 1 and confirmed in phase 2. Candidate mechanisms should be drawn from existing, multilevel frameworks for mechanism and implementation research (e.g., [30‐32]) and confirmed via structured input from experts in implementation and the specific class of HIT via a systematic Dephi process [33] to link mechanisms to capabilities (e.g., EMR warnings [capability] to aid in memory recall [mechanism]), and achieve consensus rank-ordered capability-mechanism pairings for phase 4 evaluation.

Phase 4 is designed to address more causal research questions about the impact of specific capabilities on mechanisms (e.g., does an increase in the intensity or dosage of capability X result in a corresponding increase in mechanism Y?). To accomplish this, Phase 4 involves conducting a series of microtrials—rapid and brief tests of the effects of circumscribed environmental or behavioral manipulations on proximal outcomes or mechanisms of change [34]—to evaluate the connection between HIT capabilities and mechanisms identified in phase 3. Recently, microtrials have been used as a feasible method for testing individual parenting techniques (e.g., praise) outside of the context of full treatment packages to determine their discrete merit [35]. Early in a development process, microtrials have the potential to support the collection of “proof of concept” evidence for specific techniques or HIT capabilities using within-subjects, case study designs. Although microtrials represent a feasible approach to engaging in rapid, small-scale prototyping to evaluate components of complex psychosocial processes [36], they have not yet been applied to the analysis of HIT. In HIT-ACE, each microtrial can test the effects of a single, experimentally manipulated capability of the technology on identified mechanisms and provider practice changes expected to result from the technology (e.g., EMR embedded warnings [capability] to remind [mechanism] physician’s about evidence-based prescribing algorithms). It is hypothesized that each capability will have stronger effects when, based on theory, it targets the putative mechanisms of action per phase 3. Overall, phases 3 and 4 aim to utilize and build upon the preliminary findings of phases 1 and 2 while focusing in an increasingly experimental manner on implementation success. In phases 3 and 4, evaluation of which capabilities impact which mechanisms is intended to allow for the identification of a smaller set of core capabilities and, ultimately, more parsimonious and pragmatic technologies.

MFS were defined as digital technologies that (1) include, or provide the ability to input into the system, quantitative measures that are administered regularly throughout treatment to collect ongoing information about the process and progress of the intervention, as well as (2) provide an automated presentation of this information and, in doing so, supply timely and clinically useful feedback to mental health providers about their patients and caseloads. As indicated above, we defined the scope of the example application to focus on behavioral health MFS. The behavioral health literature has seen considerable MFS advancement in recent years, including a growing number of MFS publications [47], special issues [48, 49], theories outlining intended use [38], and methods of MFS development [50, 51].

Given the likelihood of information gaps between the commercial and academic sectors, identification of MFS—and of HIT more generally—requires a multi-method process. We engaged in a systematic process to identify MFS in each sector including: (a) Google searches using the search strings below, (b) database searches (i.e., Web of Science, PsycINFO, PubMed), (c) soliciting systems from known experts (i.e., researchers who have published in the area), and (d) email related professional listservs (e.g., Society for Implementation Research Collaboration and the Association of Behavioral and Cognitive Therapies’ Dissemination and Implementation Science Special Interest Group and Technology and Behavior Change Special Interest Group). Search strings used to uncover relevant MFS included “measurement feedback system,” “measurement-based feedback system,” “clinical measurement feedback system,” “outcome monitoring system,” “routine outcome monitoring”, and “system to track outcomes.” Subsequent to MFS identification, all available materials was collected including websites and relevant literature. Identified MFS had to fit the MFS definition described above and had to report facilitating ROM in behavioral healthcare. Identified systems were excluded if it was not possible to locate a website or literature describing the system. If two systems from the same development team had different names or branding but clearly contained the same set of capabilities, the more recent of the two systems was included. In total, as of December 31, 2014, the final list included 49 systems for review.

Although the HIT-ACE coding scheme—composed of capabilities and characteristics—is primarily deductive in nature, the initial coding scheme was created to evaluate MFS via both inductive and deductive processes [52]. First, system capabilities were extracted from the literature to establish an initial framework. This stage of development incorporated aspects of Feedback Intervention Theory [37], including capabilities related to “feedback timing” (i.e., along what schedule does the MFS provide feedback) and “standard-gap feedback” (i.e., does the MFS provide feedback relative to a norm or standard). In addition, relevant literature on electronic health records and other HIT led to the inclusion of additional characteristics, such as the existence of “patient portals” as well as characteristics such as whether the systems were compliant with the US Health Insurance Portability and Accountability Act (HIPAA) [53‐55]. Next, two representative MFS were reviewed to ensure comprehensiveness of the coding system and identify additional characteristics or capabilities. This process led to the inclusion of characteristics related to system marketing (e.g., the availability of a promotional demonstration) and specific data elements tracked within the systems (e.g., service recipient critical events). Finally, stakeholders (e.g., agency administrators either using or desiring to use MFS) and experts (e.g., researchers developing or publishing on MFS) reviewed the list and provided additional characteristics or capabilities to be included.

The final step of coding scheme development required the concrete operationalization of each MFS capability or characteristic in order to support their consistent application, including positive and negative examples. For example, the capability, “Service Recipient Portal to View Outcomes,” was defined as, “A treatment view tailored specifically for service recipient. This must be intended for the recipient, rather than a common portal that can be shared, with a separate login from that of the service provider.” Definitions and examples were refined during the pilot phase of the coding process. Complete tables of the capabilities and characteristics, including definitions, are provided in Tables 1 and 2, respectively. When combined, Tables 1 and 2 represent the complete phase 1 coding scheme.

Prior to the formal coding process, the coding system was piloted to test its feasibility and accuracy and to drive refinements. This process was completed by six of the seven authors (A.R.L., C.C.L., M.B., A.M., F.F.L., and N.J.), who independently applied the coding scheme with representative systems and the primary MFS information source (typically a commercial website). After completion of pilot coding for one system, coders met to compare codes, discuss questions, identify possible new codes, and resolve conflicting ratings. Next, revisions were made to the coding scheme to accommodate additional relevant capabilities and to consolidate similar or redundant capabilities. For example, the capability “Alerts to Providers,” defined as “critical information brought to the attention of the mental health provider in a way that differs from how information is usually presented in the system (e.g., flags/highlighting, emails, pop-ups, etc.),” was complemented with a new code, “Alerts to Others,” in order to capture system alerts directed to other recipients (e.g., supervisors).

As the next component of phase 1, the coding scheme was applied to evaluate identified technologies. In the case of these MFS, all systems were coded by two independent research assistants using the same information sources, including system websites and empirical articles (see Table 3 for a full list of systems and primary coded source). As needed, the two raters consulted with the investigative team to resolve discrepancies. To facilitate coding feasibility, efficiency, and consistency, a decision was made by the investigative team to code the most information-rich source in an attempt to maximize accuracy in the information accessed. The primary information source was then reviewed in full, including all videos, links, and appendices where relevant, by two independent raters.

A consensus coding process was used in which raters reviewed materials independently and then met to arrive at consensus judgments through open dialogue [56, 57]. All capabilities and characteristics were assigned a “0” or a “1” to reflect its absence or presence, respectively. Coding was intended to reflect the viewpoint of a potential consumer in that, if the presence of a capability was unclear or not mentioned, a consumer would likely assume that the system lacked this capability. Therefore, a “0” was given if the information about a capability/characteristic was too ambiguous, the capability was not mentioned at all, or if it was stated explicitly that the system did not have this capability (although few systems explicitly mentioned not having a particular capability). A “1” was given if the MFS clearly had the capability. See Figs. 2 and 3 for a visual representation of this process. The dichotomous ratings for each of the capabilities were analyzed to determine the frequency with which various capabilities were represented across the systems. In addition to dichotomous coding, additional qualitative information was collected to further describe the capabilities and characteristics identified as present or absent. For example, specific information was collected about the assessment measures contained within each MFS, types of standard-gap feedback provided, and the extent to which provider interventions/practices were tracked.

We also examined the number of published articles referring to each system. Relevant articles were obtained by searching the full system name—and acronym if applicable—in quotation marks in Web of Science and PsycInfo. Articles were included if the research described, used, developed, tested or reviewed the system, but not included if the article only cited or made reference to the existence of a system. This search method was intended to provide a snapshot of the frequency with which the system was discussed in the literature, but not necessarily an indicator of the scientific rigor of the system. Additionally, locating these articles made it possible to systematically track the emergence of a subset of these systems over time. Exact data related to the inception of these systems were difficult to locate due to the fact that public information often does not include creation or development history. However, all relevant articles for each system were reviewed through the process described above and the earliest publication date was treated as a proxy for the “inception date,” as it is likely that the publication date of a relevant article is correlated with actual system development.