Evaluating efforts to diversify the biomedical workforce: the role and function of the Coordination and Evaluation Center of the Diversity Program Consortium

The Coordination and Evaluation Center (CEC) serves as a critical link in the activity and vision of the Diversity Program Consortium (DPC) effort. As an entity separate from the funding agency – the National Institutes of Health (NIH) - and the programs implementing targeted activities – the Building Infrastructure Leading to Diversity (BUILD) and National Research Mentoring Network (NRMN) initiatives - the CEC has distinct and unique functions. The main role of the CEC is to conduct an evaluation of the DPC initiatives across programs, with a focus on long-term outcome assessment. In contrast, BUILD and NRMN were designed to develop and test novel and innovative programs to increase the number of well-trained NIH-funded investigators from underrepresented groups. The resulting scope of the DPC is broad, spanning multiple career stages that represent critical years of investment in training and education for biomedical scientists. Shared identification of existing measures and development of new measures leverages the power of the Consortium and contributes to local program evaluation and the scholarship on training for biomedical research careers. To frame the expansive evaluation of the DPC, the CEC worked with a committee of representatives from each initiative awardee and the NIH program office to identify the important indicators of transition through career stages that are being addressed across the consortium, or DPC Hallmarks of Success. Using these Hallmarks, the CEC has designed a multidimensional, comprehensive assessment of the impact of the DPC utilizing common measures within novel multi- methods (qualitative and quantitative).

Connecting the BUILD and NRMN initiatives in a comprehensive assessment approach enables the NIH to discern and promote dissemination of generalizable knowledge, including best practices that optimize institutional commitment to diversity, faculty and underrepresented student’s career success in biomedical science. Critical to accomplishing these goals are partnerships with a variety of stakeholders: students and faculty at a wide variety of institutions, fellow grantees and investigators, NIH leadership and staff, institutional leaders at DPC-funded institutions, professional societies, and biomedical training programs with related goals and objectives. The CEC is charged with coordinating activities and communication across the Consortium. Dissemination of tools, approaches, and findings will extend the investment in the Consortium and the CEC serves a central role in such activities.

Thus, the integrated DPC evaluation is particularly significant because it is one of the few national, longitudinal evaluations of training programs for the National Institute of General Medical Sciences. The Consortium-Wide Evaluation Plan is the largest longitudinal, hypothesis-driven diversity pipeline evaluation ever funded by the NIH. The findings of the initiative, including the processes for data collection and management, will serve as a national model for biomedical training and assessment.

A key challenge to the evaluation of the DPC initiatives is that the trajectory for undergraduate students (~age 20) to NIH R01-awarded researcher (~age 42) is complex and involves a longer period than the time frame of the federal funding provided. There are many transition points along that trajectory where students may be more vulnerable. For instance, underrepresented minorities intending to major in science, technology, engineering and mathematics (STEM) are twice as likely as white students to switch to a non-STEM field before graduation from college [1]. Thus, the CEC developed an “Arc of Success” to delineate stages along this trajectory (Figure 1). Participants in DPC initiatives reflect individuals from across this arc, with some entering as undergraduate STEM students while others may become involved as they transition from graduate degree to post-doctoral positions or first faculty positions. Still others may be in faculty positions but not yet have had an NIH grant funded. Thus, a range of longitudinal and cross-sectional data collection modalities and strategies need to be utilized for assessing the variety of experiences and accomplishments that predict a high likelihood of future success in biomedical research, academia, and NIH funding along the career trajectory.

Accordingly, the DPC developed a set of consortium-wide Hallmarks, or potential predictors of successful transitions across the Arc of Success – essentially, key career transition points. Some Hallmarks are both outcomes at early stages and predictors for later stages of the Arc of Success. By defining Hallmarks, the DPC can measure and track the path for individuals in a way that can be summarized and analyzed for patterns that promote and inhibit progress [2].

Traditionally, it has been difficult to assess relatively matched cohorts of trainees using such hallmarks. One barrier has been the paucity of consistent and reliable evaluative data [3]. Often, programs are not provided with direction on what type of evaluation data to collect so that common measures can be employed across programs targeting different points along the career trajectory [4]. Participating stakeholders may not understand the importance of providing all the information requested or there may not have been clear communication about a data reporting schedule. In many cases, the funding for collection of evaluation data may be limited or not included in the project budget.

Another barrier has been bias in student participation and program reporting of interventions. For instance, programs may work to maximize limited resources by narrowing the selection criteria for participating students. While quasi-experimental approaches can be used to define appropriate comparisons, funding agencies may not emphasize the importance of such designs or may not have adequate resources to support such designs [4].

There is also lack of agreement on how confidential data that could be used for such evaluations should be disaggregated across disciplines, racial groups, and gender [4]. It is important to maintain the confidentiality of program participants. This may become difficult if data are examined by multiple strata, perhaps resulting in making it possible to identify the five women biochemists who are participating in a grant-writing workshop. Data structures and reporting templates can be designed to minimize the risks of identification, but such tools need to be explicitly designed and incorporated into the evaluation.

Finally, collecting longitudinal data to generate and test causal models linked to successful participation in STEM for diverse student populations has been challenging due to evolving appreciation for privacy and confidentiality issues [4]. Over the last several years, programs have worked with their Institutional Review Boards to determine when it is necessary to incorporate language about consent to contact participants for follow-up. Funding for longitudinal contacts can also be difficult to obtain, as downstream effects may need to be measured several years after the original program.

The CEC worked with the DPC Executive Steering Committee (composed of PIs and investigators from the funded initiatives and NIH) in a collaborative effort to establish the consortium-wide Hallmarks of Success. First, the CEC conducted an extensive literature search including several NIH reports recommending goals of diversity training programs [5].

From the literature review and NIH model, the DPC Executive Steering Committee completed an iterative review process to establish the Hallmarks. The guiding principle was that the Hallmarks needed to be relevant to a majority of the DPC programs and measurable in a consistent manner across DPC program sites for a broad array of activities and institutional interventions. The DPC Hallmarks of Success include traditional, non-traditional, and potentially novel hallmarks based on prior experience and planned activities and interventions of the DPC program awardees. Figures 2, 3 and 4 present a summary of the Hallmarks that received collective approval across the Consortium. For organizational purposes, the Hallmarks of Success were divided into three domains: Student/Mentee, Faculty/Mentor, and Institutional. Within each domain, Hallmarks are clustered conceptually. Hallmarks relevant for individual mentees may be classified in student or faculty domains, depending on the career level of the individual. Student and faculty Hallmarks can also be viewed as defining the beginning, intermediate, and advanced career levels and may often overlap.

The DPC Hallmarks of Success are designed to be comprehensive, encompassing various levels of student and faculty activities – each known or hypothesized to have a positive influence on the probability that the individual will pursue a successful career in biomedical research. Each of these Hallmarks is not essential. Some researchers, for example, may seek graduate training in health sciences after completing undergraduate degrees in other fields. Others may experience rewarding research activity without attending a summer enrichment program. The Hallmarks can be viewed as a set of key indicators of the likelihood of success in pursuing a career in biomedical research. The cumulative effect of achieving Hallmarks is one of the scientific questions the DPC will provide data to address. Generally, it is hypothesized that the more of these Hallmarks achieved, the higher the likelihood of a successful biomedical research career.

These Hallmarks provide a framework within which to assess factors contributing to different career paths. In the case of the BUILD and NRMN initiatives, the Hallmarks can be monitored longitudinally and compared within and across institutions to understand program impact and collective progress in unique institutional settings. Such longitudinal study may suggest that certain Hallmarks are more critical experiences or accomplishments for certain groups of individuals. For instance, participating in research as an undergraduate appears to be more critical as an experience in predicting pursuit of biomedical graduate school training for first-generation college students compared with students whose family members received college degrees [6]. Thus, the robust analysis of Hallmarks will allow tailored recommendations to advance national strategies to increase diversity in high quality students committed to careers in the biomedical sciences. A brief overview of the Hallmarks is described below.

Student/mentee Hallmarks are a mix of psychosocial factors and educational and professional accomplishments and experiences (Figure 2). Some individual factors are specific to scientific and biomedical careers, such as development of science identity, scientific self-efficacy, and aspirations for scientific research [7, 8]. Other factors are more general, such as academic self-efficacy, social integration and sense of belonging in the university setting [1, 9‐11]. Increases for undergraduate students on these factors are considered hallmarks that represent early predictors of successful longer term outcomes such as completing graduate work in biomedical research and becoming an independent scientist (either as research faculty or a senior scientist in the private sector).

Accomplishments and experiences noted as student/mentee Hallmarks are focused on retention and persistence in biomedical research training [6, 8, 10, 12]. Key Hallmarks specifically for undergraduates are participation in biomedical research activities that include presentations at scientific conferences [5, 13, 14] and matriculation to graduate biomedical programs [5]. Matriculation is viewed as both a successful outcome of the undergraduate experience and an important predictor of future career success in the biomedical sciences.

Faculty/mentor-level Hallmarks include program-relevant activities and behaviors among faculty, including increased participation in professional development activities for improving teaching and in mentorship activities and training, as well as improvement in the quality of mentoring based on identified behaviors and attitudes [15‐17] (Figure 3). Hallmarks that are accomplishments are increased research productivity in publications, grant submissions and awards. Finally, psychosocial Hallmarks for faculty and mentors include change or increase in self-efficacy as instructor, mentor or researcher [17‐19].

Institutional-level Hallmarks include the availability of resources and opportunities for students and faculty that are associated with their development (Figure 4). For example, the availability of pre-professional and departmental clubs for students, supportive and collaborative science environments, and viable research opportunities for financial assistance have been shown to enhance student success [9, 12, 20]. Compared to other institutions, successful institutions can document increased inter-institutional collaborations, improved undergraduate retention rates of students in BUILD relevant programs, increased participation in program-relevant mentoring activities (students and faculty) [21], institutional commitment to sustaining initiatives evidenced by maintenance of key elements of program interventions after grant period, the availability of undergraduate research [22], and an increase in the number of faculty seeking and participating in mentor training.

In sum, the DPC Hallmarks are a comprehensive set of training and career exposures and outcomes which evidence indicate are associated with biomedical research career success. These hallmarks are being assessed longitudinally in students and faculty and will be used to evaluate the interventions implemented by the BUILD and NRMN programs.

The Consortium brings a range of resources, approaches, and expertise and will produce evaluation and pedagogical tools, scientific findings, and novel approaches to structure and process for BUILD/NRMN interventions. To leverage the resources and maximize dissemination of products, the CEC creates collaborative environments for the Consortium, supports the shared governance of the Consortium, and manages communication within the Consortium and more broadly to various communities such as other undergraduate institutions, professional societies, and funding agencies.

The CEC formed by pairing established leaders of health-related training programs with national experts in higher education research on faculty, undergraduate student experiences and STEM pathways. Various authors have described the importance of gathering diverse talent across disciplines and expertise when approaching complex problems [25]. Campbell’s fish scale model of omniscience also describes how science as a broad concept advances when overlapping narrow disciplines combine evidence [26]. Similarly, team science has been proposed as an approach to bring content and methodological expertise together from multiple perspectives in order to study multi-faceted problems [27]. As the goals and activities of the project developed, the CEC expanded to include those specializing in team science, evaluation, qualitative and quantitative methods, robust data systems, and translational science. This blend provides a cross-disciplinary approach as well as representation across career level to effectively inform decisions, essentially creating a new network from existing resources and expertise at the University of California, Los Angeles.

A significant resource for the CEC is the Higher Education Research Institute at UCLA (HERI), a group focused on improving outcomes for college students in the United States through longitudinal research. HERI is home of the Cooperative Institutional Research Program, a 50-year study of American college students. HERI manages administration of the surveys of the program. Collaboration with HERI enables the CEC to leverage expertise in longitudinal surveys with college students, infrastructure for such administration, a reporting structure, and an extensive bank of survey items with robust psychometric properties.

In addition to the expertise at HERI, the Director of the UCLA Computing and Technology Resource Laboratory is a CEC co-investigator. This group provides advanced technology for data collection and management as well as document sharing and storage platforms important for CEC coordination across the Consortium. The NIH-funded UCLA Clinical Translation Science Institute serves as a resource for the evaluation of clinical science training programs. CEC investigators and staff also work with this group, allowing shared resource and expertise. Finally, one of the senior investigators is principal investigator for the NIH-funded Resources Centers for Minority Aging Research Coordinating Center, which organizes and implements an Annual Investigators Meeting and provides the framework for reporting of uniform data and RCMAR Scholar productivity and achievements for junior faculty across six national sites, several of which have been active for 10–15 years. The CEC benefits from these experiences with uniform reporting and the tracking of scholars during key career transitions.

In summary, a consortium provides an incredible opportunity for team learning, growth and ultimately the generation of a deep and lasting impact on important problems of achieving diversity in biomedical education and training. Vital to securing the long-term success of the consortium, the CEC consults with the Executive Steering Committee on how best to facilitate the leveraging of consortium capacities and sharing lessons learned. This has led to the creation of several consortium-based workgroups led by the CEC using synchronous (e.g., webinars, conference calls) and asynchronous distance interactions (e.g., webinar recordings, email). The primary avenue for structured in-person interactions is at the DPC Annual Meeting, attended by key personnel (e.g., Principal Investigators, Program Managers, Directors, Support Staff, Evaluators) from the consortium and serves as a way to share scientific progress, engage in professional learning, and make plans for programmatic and consortium success.

In addition, the CEC approach to capacity building is founded on years of successful experiences developing, leading, and facilitating networks and partnered activities for a diverse group of biomedical and health science related programs, institutions, community-based organizations, and foundations. A systems approach is blended with systematic inquiry for learning in an effort to more fully develop the DPC capacity [2, 28‐34]. Thus, the CEC has worked to harness the vast amount of skills and expertise that lie within the Consortium to support, sustain, and publicize the work. Some examples of this are best practices in outreach and recruitment, and structuring authentic research experiences that merit further evaluation to establish its evidence base. Ultimately, the CEC is responsible for the collection and rigorous assessment of DPC activities to understand and disseminate the evidence for how to support the increasing number of dedicated faculty, institutions, and strategic networks committed to making sure every trainee has opportunity to succeed in the biomedical sciences and to enhance the collective scientific brain trust to accelerate advances that will improve the nation’s health.