1. Implementation-specific issues
In this section, we present IRB challenges specific to implementation research. Implementation research uses a variety of methodologies at different stages and often requires collaboration among disciplines [
24]. Some implementation research issues may not fit within the traditional human subjects paradigm used to assess clinical trials. Risks in implementation research are generally related to loss of confidentiality, rather than risks associated with drug treatments, as in traditional clinical trials. The implementation-specific challenges introduced below are: external validity considerations, Plan-Do-Study-Act cycles, risk-benefit issues, the multiple roles of researchers and subjects, and system-level unit of analysis.
a. External validity considerations
Most IRBs tend to be much more familiar with the rationales for biomedical research procedures than with those for implementation research. Two conceptual models can be helpful in providing clear explanations for specific aspects of implementation research that promote external validity, and also bear on human subjects protection issues – the Stetler Model of Research Utilization [
25,
26] and the RE-AIM framework [
27‐
29]. These models can be incorporated in IRB applications by researchers, as appropriate, to inform their interactions with IRBs, as they describe external validity considerations. While these frameworks came to our attention after we had put much thought into our approach to IRB issues, they may be helpful to other researchers. These two models are presented briefly in Additional file
1: Implementation Evaluation Theory: Stetler Model of Research Utilization and RE-AIM.
b. Plan – Do – Study – Act (PDSA) intervention adaptation cycles
PDSA cycles are often a cornerstone of implementation research. The nature of PDSA cycles means that implementation researchers may not be able to specify all aspects of the project ahead of time for the IRB to judge risk-benefit issues in the initial application. To change a care system in a sustainable fashion, an "intervention" must be both individualized to the resources and goals of each site, and dynamic enough to take into account the inevitability of changing conditions. PDSA cycles within a particular study may incorporate multiple levels of formative evaluation [
4]. When formative evaluation results are applied to the project, the equivalent of Plan Do Study Act cycles [
30] result in changes that may be unforeseen at the beginning of the project.
We utilized two different approaches to address IRB concerns resulting from PDSA cycles, depending upon the preferences of each IRB. At the majority of our sites, we submitted an initial protocol that was as complete as possible, given our site knowledge and expectations, and then submitted modifications as needed. In situations where the protocol was considered exempt, we consulted with our IRB representative to ensure that the proposed modification would not alter the IRB's original decision that the protocol was exempt. However after extensive discussion and collaboration, other IRBs requested dividing the initial protocol into key component submissions, each individually well specified.
Under this approach, we submitted the first study component along with an overview of the entire project. We then submitted additional project activities for individual review in the context of the project overview. This approach allowed IRB reviewers to judge project activities in context without having to judge all components at once. Reviewers had complete information for each component at the time of submission. It allowed the level of IRB review (i.e., exempt versus full) to be appropriate to a specific component. At one study site, four consecutive components were submitted individually and received different IRB approvals.
Component 1 involved carrying out an expert panel to decide how to best administer a depression collaborative care project and was exempted. Component 2, staff qualitative interviews, was also exempted. Component 3, a survey of site clinician opinions, was not exempted but received a standard review and Waiver of Documentation of Consent. The final component, a clinical decision support software tool developed by study participants (not researchers), was acknowledged by the IRB as having been received, but not reviewable, i.e., not research.
Over time, this component solution proved to be administratively burdensome to both the IRB and the research team due to the need to track each component as a separate application, complicating modifications and renewals. We anticipate that future IRB developments will identify the best methods for introducing the unique features of individual components to the IRB and, thereby, will decrease the burden on both researchers and the IRB [
9].
c. Risk-benefit issues
Implementation research often involves both evidence-based interventions carried out by clinical partners and evaluation activities carried out by researchers. In QUERI implementation research, the best practices to be implemented are not experimental as earlier QUERI research phases have already documented the need for clinical improvement and provided an evidence base (Table
1). Unless carefully explained and supported, we have found that it may not be obvious to IRBs that the "research" issue is not the efficacy of the best practice, but the ability of a quality improvement strategy to introduce or improve a system's ability to apply the evidence-based practice. IRBs may not realize the extent to which quality improvement principles and requirements guide clinical activities carried out in implementation projects and, therefore, risk-benefit issues might be most appropriately considered in a QI framework. Currently, the determination as to whether an activity is reviewable research or non-reviewable QI is made by the IRB. Unfortunately there is not yet a standard method for assisting an IRB to make this decision.
In preparing our applications, to assist the IRB, we sought to use clear layperson terminology to explain our project. For example:
There are two aspects to this project that could be described as "intervention:" 1) the system-level evidence-based quality improvement process that has the goal of changing healthcare provider behavior, and 2) the depression care that the providers deliver. This project studies the former. The latter, described as collaborative care for depression, has been extensively studied and is accepted as best practice care. No experimental treatments are employed in this project.
We have had some success in using an opinion letter process as an initial step in the application process. For example, one IRB (Table
4, Site I) considered a protocol to fall under quality improvement and be classified as "non-human" research, thus not requiring IRB review. However, for the same protocol another site required a full submission. Given the societal health benefits likely to be associated with researcher involvement with clinical systems and practices, and the known risks of poor quality care, the ethical issues of adding burden to the process of improving care should be considered in these determinations [
31,
32].
Table 4
TIDES/RETIDES IRB initial application information
Site*
|
IRB type
|
IRB required changes?
|
Approval Type† |
Time Elapsed‡
|
A | VA in-house | No | Exempt | 32 |
B | VA in-house | No | Exempt | 67 |
C | University | Yes1
| Expedited | 75 |
D | University | No | Expedited | 9 |
E | VA in-house | Yes2
| Expedited | 5 |
F | University | Yes3
| Full | 7 |
G§ | VA in-house | No | Full | 82 |
H¶ | University | Yes4
| Expedited | 38 |
I | University | No | Expedited | 5 |
| | | | Average = 35.55 |
ReTIDES Study
|
F | University | Yes5
| Full | 22 |
I | University | N/A | Non-human research | 5 |
J | University | No | Exempt | 3 |
K | University | No | Exempt | 20 |
H¶ | University | Yes6
| Expedited | 30 |
L | VA in-house | Yes7
| Expedited | 14 |
A | VA in-house | No | Exempt | 6 |
M | VA in-house | No | Exempt | 14 |
| | | | Average = 14.25 |
Risks in implementation research are often elusive, and the regulatory definition of "minimal risk" is imprecise. There often are no physiological risks in implementation research and, in the case of our studies, minimal psychosocial risk, including psychological stress from completing staff interviews or surveys and evaluation discomfort. Thus, implementation research might often be categorized as minimal risk. However, breach of confidentiality due to a security breach or human error remains a risk even in minimal risk studies. Researchers should accurately assess the associated risks of the research, including the evidence-based practice.
Some systems provide expedited review for research judged to be minimal risk. For example, our protocols include evaluation through surveys, medical record reviews, and stakeholder interviews. However, these multiple activities may result in complex IRB decision-making and risk to the implementation activity timeline, if after initial review the IRB decides that the study is inappropriate for expedited review, and the study must be transferred to the full-review mechanism. For example, one site (Table
4, IRB C) stated, "If the survey was the only thing that came to the IRB board, it could have been exempt or if the chart data had come to the IRB separately, it too, could have been expedited. But because it came all together, it would be hard to expedite it as a whole with all the different components presented."
Finally, if an IRB judged the initial proposal to be exempt, it became difficult to review a later modification of the proposal. For example, this occurred when a staff survey planned as an anonymous QI activity could not be feasibly carried out without participant identifiability. It then required extra effort to return to IRBs that had judged the survey as exempt, in order to notify them of the new circumstances.
d. Multiple roles of researchers/facilitators & multiple subject/participant classes
Sometimes it is not the implementation research practice itself that is the risk, but ill-defined relationships among researchers and the system with which they are working that may add risk. Implementation researchers may best be understood as facilitators or consultants within their research projects, with no direct decision-making authority in the intervention. Within our projects, researchers were available as consultants to local site champions who provided leadership for customizing and adapting the intervention to the sites. Researchers also functioned as consultants at the national level to encourage national rollout of improved depression care. These different roles (and risks) need to be clearly communicated with administrators and IRBs. For example, because our projects involved depression treatment, it was important to establish suicidality risk-response protocols specifying activities of researchers and clinicians of which all parties were aware.
Subjects, more appropriately called "participants," in implementation research have different relationships with researchers depending on their relationship to the health care system in which the implementation takes place. Participants may include providers, regional and/or medical center leadership, and/or patients. Patients, for example, may not encounter researchers, but have specific aspects of their medical records examined for evidence that is relevant to the implementation. Providers are crucial in the success of clinical improvement efforts, however their role as a "subject" must be carefully considered. If they see themselves as subjects from whom consent is required for their participation, their buy-in and interest in sustaining the improvement after the investigative phase is over is likely to be vitiated. All classes of participants, or their representatives, may be involved as decision-makers in the specifics of the implementation. For example, our project start-up meetings often included nurse depression care managers, primary care physicians, mental health practitioners, leadership/management personnel, and patient representatives. Clearly defining these different roles and the risk appropriate to each will enable the IRB to make an informed decision.
e. System-level unit of analysis
In these projects, like similar implementation efforts, patients, providers and/or sites can be units of analysis to measure/support the external validity of the implementation effort, as well as to monitor its clinical and organizational outcomes. The impact of implementation research is often assessed on the patient level through patient administrative records; at the provider level through survey results and aggregated, de-identified data; and at the site level though cost analyses, site-level performance measurements, interview results, and project records (i.e., conference call minutes, emails, etc.) analyses. IRBs that are familiar with evaluating clinical trials may not understand the necessity of collecting information at each of these levels. Each of these levels has its own set of concerns and ethical issues. For example, whereas breach of confidentiality may be a major risk for patients, concern that results of a survey may be accessible to supervisors and used for evaluative purposes may be an issue for staff. It is important to clearly delineate these various roles, the risk appropriate to each, and the justification for each when communicating with the IRB. Using clear layperson language and/or tables to present relationships may be useful.
2. Multiple health care systems/sites
This section focuses on concerns surrounding working with multiple sites or health care systems, namely the ethical responsibilities of researchers, preserving accountability across sites, and relationships among IRBs and other organizational entities.
a. Transparency of system/researcher ethical responsibilities
As consultants, researchers often function at the behest of the organizational entity. Organizational participants may hold considerably more power than the researchers by virtue of their decision-making authority. Clinicians may encounter implementation researchers as consultants, purveyors of Continuing Medical Education and/or as gatherers of information regarding professional opinions and knowledge that may be relevant to the implementation project. IRBs are appropriately concerned when researcher/clinician roles are complex, in that this can raise issues of possible coercion in recruitment of patients and staff, and/or confidentiality problems in keeping clinical, job performance, and research data separate.
We developed an Implementation Charter (see Additional file
2: Project Implementation Charter and Site Memorandum of Agreement) with participating VA regional networks that specified the division of responsibilities between researchers and site clinical staff. For example, the Charter indicated that we were responsible for training depression care managers within the clinics, but that the care managers were hired and supervised by appropriate clinical personnel. It also specified our adherence to clinical privacy rules for aspects of our project that were quality improvement, and the need for site participants who wished to utilize any QI data from the project for research purposes to seek IRB approval. This process aid was successful in clarifying our roles and expectations with participating sites.
b. Relationships among IRBs and participating organizations
Implementation researchers often need to satisfy multiple sets of organizational reviews and requirements before beginning a project. For example, within the VA system, a research study conducted by VA personnel and/or involving VA patients cannot begin without both IRB approval and approval by the appropriate local VA Research & Development (R&D) Committee. While the IRB reviews the protocol to ensure adequate protection of human subjects, the local R&D committee provides the primary executive review for their site, assessing the overall scientific merit of the protocol. R&D application procedures and protocols are not uniform across sites.
We researched a new study site's local IRB and R&D through web sites or personal contacts. To confirm that we had the correct IRB forms for a particular submission, we informally "interviewed" the IRB administrative contact using our IRB Contact Questionnaire (see Additional file
3: IRB Contact Questionnaire). Through this process we obtained necessary information about the particular IRB process, including their relationship with the local R&D Committee. This process enabled us to effectively submit our applications with minimal problems and demonstrated our willingness to cooperate with local administrators and procedures.
c. Maintaining accountability across systems/sites
As a site was brought onto a particular project, responsibility for that site was assigned to specific research personnel within the project. This created continuity within the submission process and in interactions between the IRB and the project as a whole. Record retention has been centralized for the projects. Study personnel forward electronic and/or hard copies of all IRB paperwork and communications to the administrative site in addition to keeping copies at their respective sites. As the number of sites increased across projects, it became clear that a relational database was necessary to efficiently track all the paperwork and deadlines.
Using Microsoft ACCESS, information gathered from the IRB Contact Questionnaire is entered into the database (see Additional file
4: IRB Relational Database). As applications are submitted and approvals received, additional key information is entered. The structure allows monitoring expiration deadlines, duration between submission and IRB approval, and time elapsed since submission. Information for all sites or specific to project, site, or region can be extracted and presented as reports. The database is especially useful to track information at sites where the component submission approach is utilized.
3. Multiple local site PIs
Enthusiastic, informed and active Site PIs are crucial to the success of a multi-site study. The study staffs' ability to finesse individual Site PI situations, accommodate the desired level of Site PI involvement, and, generally, limited local support staff availability is crucial for the project to be successful. As would be expected, we experienced several challenges collaborating with multiple Site PIs, such as recruitment, varying levels of experience and limited time, inconsistent ethics certification requirements, and varying levels of commitment to the study.
a. Recruitment
We have used a variety of strategies when identifying and recruiting local Site PIs. Through our contacts as researchers and established Site PI contacts, we were able to obtain information on a potential Site PIs' availability and overall interest. The initial conversation between the project PI and potential Site PI covered background information on the project, noting other clinicians who may be participating in the project, and a brief overview of responsibilities. An assigned member of the research staff followed up with the Site PI to further address questions. Communicating Site PI responsibilities, requirements, and level of commitment clearly and accurately is necessary for success. We used the Implementation Charter mentioned earlier to help clarify the role of the Site PI in relation to the researchers (See Additional file
2: Project Implementation Charter and Site Memorandum of Agreement). It is critical that the involvement of the site PI is supported by their organizational supervisors and relevant to their own career goals.
b. Varying levels of experience & limited time
After recruitment, the research team must assess the experience level of Site PIs with regard to their local IRB, the IRB process and responsibilities, and our project(s). One consistent factor among our Site PIs was their busy schedules and limited availability. As a result, we developed an efficient, concise protocol to gather this information through our Site PI Orientation Checklist (see Additional file
5: Site PI Orientation Checklist). Upon completion of the Checklist, roles and responsibilities of the research staff and Site PI were clearly delineated, including who would handle the majority of IRB paperwork. At some sites, research staff effectively became the research coordinator for the local Site PI. A Site PI's willingness to assist in problem-solving proved to be critical to the timeliness with which each site progressed through the IRB process. For example, some IRBs do not include remotely-based research staff on correspondence relating to the project. Site PIs are encouraged to proactively forward those documents to their research staff contact person to ensure optimal project responsiveness to IRB requirements and to minimize project delays.
d. Varying ethics certification requirements
All researchers are responsible for fulfilling any necessary ethics certification and training requirements prior to the start of any research. However, different sites may require renewals of these trainings at different times and may require additional certification. It is in the project's interest to monitor these requirements to avoid any project delay due to a lapse in training certification. The relational database mentioned above can be used for this purpose (see Additional file
4: IRB Relational Database).
d. Varying levels of involvement
Site PIs demonstrate varying levels of involvement to their role within the project. Some take an active role in communicating new ideas and suggestions regarding implementation, or independently promoting and presenting the project at conferences or meetings with clinicians. To help provide incentive, we try to involve interested Site PIs in project-related activities, such as workgroup meetings. Several of these working groups have developed and published manuscripts.
4. Multiple IRBs
For researchers who have the need, the logistics of working with multiple IRBs are discussed below. These include: lack of standardized procedures and forms, communication with IRB administrators, multiple competing renewal deadlines, variable adverse event reporting requirements and definitions, and issues with coordination and consistency of IRB-related tasks.
a. Lack of standardized procedures & forms
Our experience is that IRB procedures, forms, deadlines and meeting schedules vary from one IRB to the next. Areas of variability include: whether the IRB will communicate directly with staff other than the Site PI, structure and organization of application forms, deadlines and meeting schedules, and reporting requirements for adverse events. This diversity increases the burden on research staff. We developed a consistent approach to submissions. One senior member of our research staff composed the "template" text that was subsequently distributed to other research staff members, who tailored the text to best fit a particular IRB.
b. Communications with IRB and R&D administrators
The multi-site nature of our projects means that in the majority of cases, the project PI, Site PI, and research staff are in different geographical locations. Communicating this to respective administrators enables them to understand the constraints of the situation and, in some cases, resulted in their exceptional assistance in the submission process. For instance, one R&D Administrator offered to collect the necessary Site PI signatures for submissions, hand deliver the paperwork to the IRB, and inform the research team via e-mail.
At the start of the projects in 2001, many IRBs did not have websites, or if they did, all necessary forms and instructions were not available on the site. This required the assistance of the site's IRB staff in order to obtain all forms and any additional relevant information on the submission process. Today, even with many forms and instructions available online, it is crucial to communicate with IRB staff to ensure the correct forms are used to fit a particular activity (see Additional file
3: IRB Contact Checklist). Considering IRB and R&D staff as integral members of a QI effort helps minimize communication difficulties and facilitate the sometimes cumbersome IRB process.
c. Multiple competing renewal deadlines
Carrying out projects involving multiple IRBs creates a revolving door of renewal deadlines. This requires that the research team monitor the approval status of each site. Renewal forms change over time as IRBs revise old questions and add new ones. It is important to take this into consideration when budgeting the amount of time needed for IRB activities.
All of our IRB submissions are tracked through the relational database, and bi-weekly submission deadline reports are created that are shared with other coordinating administrative sites (see Additional file
4: IRB Relational Database) using group emails and IRB-specific conference calls.
d. Variable adverse event definitions & reporting requirements
Implementation projects involve the possibility of adverse events that, although not directly connected with study activities, may need to be reported to involved IRBs. Since our projects work with a depressed population, we prepared for the eventuality of a serious or successful suicide attempt. We discussed this potential risk in IRB applications and collected different IRB definitions, reporting requirements, and related items for Adverse Events (AEs) and Serious Adverse Events (SAEs) (see Additional file
3: IRB Contact Questionnaire).
There is a significant degree of variation in AE/SAE definitions and reporting requirements illustrated by the following hypothetical situation of a completed suicide by a study patient. One IRB would consider this a reportable SAE, because a subject died. A second IRB would consider it a reportable SAE because suicide is an expected risk among this study population. A third IRB would not consider it a reportable event because it was believed not to be related to the study procedures, since all established medical center and research policies and procedures were appropriately followed.
e. Coordination & consistency in IRB-related tasks
All the process aids and approaches mentioned above can not be fully realized without coordination, consistency, advanced planning and consultation. For example, while developing the IRB portion of our grant applications, we consulted with several IRB experts. IRB work for implementation projects is a significant resource allocation and budget issue. It may be cost-effective to budget for an IRB specialist to assist research staff. This specialist should have extensive knowledge of Human Subjects protection issues and the IRB system in which the study takes place (see Additional file
6: IRB Specialist Position Description). In our case, other project staff devoted approximately 1.5 FTEE (Full time employment equivalent) to IRB-related tasks over all study years (6 years, at present), with significant involvement both by investigators and research coordinators.
Regular conference calls were established to anticipate and discuss IRB issues within the projects. These calls ensure that research staff can discuss and resolve IRB concerns (see Additional file
7: IRB Call Minutes). While this effort is substantial, similar projects should anticipate relevant IRB challenges and budget appropriately.