Relations Between Program and System Variables and Fidelity to the Wraparound Process for Children and Families

Meeting the mental health needs of children and their families has been declared a national priority ( 1 , 2 ). To respond to this challenge, advocates and leading researchers have called for improving the effectiveness of services by implementing evidence-based practices ( 3 , 4 , 5 ) while improving families' access to and engagement in services by developing systems of care and implementing the wraparound process ( 6 , 7 , 8 ).

Wraparound is not a specific treatment but rather a process through which providers collaborate with families to develop an integrated and creative service plan tailored to the strengths and specific needs of the child and family. At the core of this process is a team comprising the family, identified supports (teachers, clinicians, and friends), and a facilitator to coordinate the process. The team, with family members as lead decision makers, works together to develop, implement, and adjust the plan over time ( 9 ). Services are "wrapped around" the child and family in their natural contexts (home, school, and community) rather than being forced upon them in less accessible or more restrictive environments (such as the provider's office or a residential treatment center). Because of its intensity, the wraparound process is typically implemented with children diagnosed as having serious emotional and behavioral disorders whose needs cannot be adequately met by a single intervention or who are involved with multiple child-serving agencies. Details on the model are presented elsewhere and cover phases and activities ( 10 , 11 ) and components for implementation ( 9 , 12 ).

The wraparound process has been cited as a promising approach in reports from the Surgeon General on both mental health and youth violence and as a means for more effectively delivering evidence-based practices ( 1 , 8 ). Wraparound programs in community mental health organizations have been adopted widely across the country. Over 200,000 youths with serious emotional and behavioral disorders are estimated to be served annually via the wraparound process in the United States ( 13 ). The model is also promoted by the children's services program of the Substance Abuse and Mental Health Services Administration ( 14 ).

Despite its promise, the wraparound process has not been the subject of adequate research and evaluation ( 11 ). Although results are positive, drawing conclusions from the small number of experimental studies ( 15 , 16 ) and quasi-experimental studies ( 17 , 18 ) is difficult because of lack of fidelity control and measurement. To address this concern, fidelity tools have been developed that measure the extent to which a family is receiving services consistent with the principles of wraparound ( 19 , 20 ). Studies using these fidelity tools have demonstrated a relationship between model adherence and favorable child and family outcomes ( 21 , 22 ). Thus, while results of ongoing controlled studies are being awaited, many communities that offer wraparound programs have focused on ensuring that a family's wraparound team can implement the wraparound process with high fidelity.

Organizational and system factors and fidelity

Both the system-of-care model ( 6 ) and the research-based framework of Walker and colleagues ( 23 ) emphasize that implementation of the wraparound model is more successful when certain necessary conditions are in place. Such conditions can be characterized at the host organizational level (such as staffing patterns, training and supervision, and implementation regulations) and broader service system level (such as degree of collaboration among agencies and availability of funding). Evidence from the broader literature on children's mental health has also underscored the importance of organization- and system-level factors in providing support for implementation. A study by Glisson and Hemmelgarn ( 24 ) demonstrated significant associations between organizational climate and cultural variables with outcomes achieved for children and adolescents. In addition, research on the transportability of multisystemic therapy has demonstrated relationships between treatment, provider, and organizational and system variables ( 25 , 26 ).

In research on the wraparound process, Walker and colleagues ( 23 ) conducted a series of studies that yielded a framework of conditions necessary for full wraparound implementation. The framework consists of major categories of supports (such as capacity building and staffing, interagency collaboration, and accountability), each with several subcategories. However, there has not yet been an attempt to empirically test the relationship between such variables and actual wraparound implementation.

The purpose of this study was to provide an initial empirical test of the importance of organization- and system-level supports, such as those proposed by Walker and colleagues ( 23 ). The organizational and system factors were assessed through interviews with administrators at community mental health agencies that provide services via the wraparound process. The Wraparound Fidelity Index (WFI) ( 19 ) was used to measure children's, caregivers', and resource facilitators' reports on the quality of wraparound implementation. The specific objectives of this study were to examine variability across different community mental health agencies in organization- and system-level supports and to assess associations between the number of these supports that are in place and family- and provider-reported adherence to the wraparound principles.

Methods

Procedure

Data for this study came from a larger validation trial of the WFI ( 19 ). Administrators at agencies or sites implementing wraparound who were interested in using the WFI were recruited from our Web site, national conferences, and meetings on children's mental health and by word of mouth. To participate in the validation trial, site administrators completed a formal request, providing information on their programs, populations served, evaluation methods used, and procedures to be used to administer the WFI. These requests were reviewed by our research team and accepted if proposals met the following criteria: the site showed evidence it was implementing wraparound, the goals for collecting WFI data were not solely for performance review, site administrators planned to recruit a representative sample of families who participated in their wraparound program, the site had obtained local institutional review board approval or agreed to adhere to the protocols submitted to the University of Vermont Human Subjects Board, and WFI interviews would be administered as stated in the manual by trained interviewers who were not involved in service delivery.

Once approved, collaborating sites recruited families to participate, obtained informed consent, trained interviewers on the basis of the research team's guidelines and the WFI user's manual ( 27 ), administered interviews, entered data into a database provided by the research team, and forwarded data (with identifying information deleted) to the research team. In turn, the research team provided collaborators with summary fidelity reports.

To conduct the study, researchers approached all nine national collaborating sites that had collected WFI data for more than 20 families. Each site agreed to participate. A structured interview, the WFI-Program Administrator form (WFI-PA), was administered by phone to the individual identified by the site as the most knowledgeable about the local organizational and system context, typically the program administrator. One administrator was ultimately unable to be reached, yielding a study sample of eight sites. All WFI data were collected between June 2001 and October 2002. Administrator interviews were conducted between August 2002 and October 2002.

Participating sites. Eight programs administering a wraparound process for families with children experiencing serious emotional disorders participated in the study. The agencies included two from California and one each from Arizona, Indiana, Missouri, Nebraska, North Carolina, and Vermont. Participating sites had been in operation between 2.5 and seven years (mean±SD=3.90±1.45) and had served between 41 and 497 families (mean=223±164). Two sites reported serving children and youths ranging from infancy to 18 years old, and six reported serving youths aged six to 18 years. Two sites were primarily aimed at youths in state custody, three programs explicitly served youths eligible for or at risk of placement in high-level residential care, and one program was specifically tailored to serve youths transitioning from group homes or residential treatment facilities.

Participating youths and families. The sample consisted of 289 families, with WFI interviews completed by 221 caregivers, 158 youths, and 193 resource facilitators. If a family had multiple interviews for the same respondent type (for example, two caregiver interviews), one was chosen at random for inclusion in the study. For each family all interviews were completed within one month of each other. For approximately half of the families, interviews were available from all three informants (84 families, or 29 percent) or from caregivers and resource facilitators (58 families, or 20 percent). For the remaining families, interviews were available from resource facilitators and youths without caregivers (25 families, or 9 percent), caregivers and youths without resource facilitators (31 families, or 11 percent), resource facilitators alone (26 families, or 9 percent), caregivers alone (48 families, or 17 percent), or youths alone (17 families, or 6 percent).

When the WFI interviews were administered, families had been engaged in wraparound for 13.40±9.04 months. Of the youths receiving services 165 (64 percent) were male; no gender was reported for 35 youths. Ages ranged from one to 19 years (mean=13.10±3.89). One hundred fifty-nine (70 percent) youths were identified as Caucasian, 54 (24 percent) as African American, two (1 percent) as Native American, two (1 percent) as Asian or Pacific Islander, four (2 percent) as biracial, and nine (4 percent) as "other race or ethnicity" (race or ethnicity was not identified for the remaining 65 youths). Eleven youths (5 percent) were identified as Hispanic or Latino.

Measures

Wraparound adherence. Adherence to the wraparound process was assessed with caregiver, youth (11 years and older), and resource facilitator responses on version 2 of the WFI. The WFI has demonstrated adequate test-retest reliability and internal consistency ( 19 ), construct validity ( 22 , 28 ), and criterion-related validity ( 21 , 29 ). Administered as a structured interview, the WFI assesses adherence to 11 core principles of wraparound: family voice and choice, team based, community based, culturally competent, individualized, strengths based, natural supports, continuation of care, collaboration, flexible resources, and outcome based. For this study, total scores for each of the respondents were calculated and ranged from 0, for low fidelity, to 8, for high fidelity. Total scores were used because of past research that has shown the greatest construct validity for total scores and poor reliability for some individual principle scores ( 19 , 21 ).

Program and system supports. The WFI-PA was designed to assess the presence of organizational and system supports needed to implement a high-quality wraparound process. The WFI-PA was administered as a semistructured telephone interview to one or more key informants from a community mental health agency, such as a program's lead administrator. WFI-PA items were developed on the basis of the framework proposed by Walker and colleagues ( 23 ) and then organized to provide information on ten indicators of favorable conditions for high-quality wraparound: program duration, staff turnover, resource facilitator caseload, interagency coordination and funding, family centeredness, community-based services, natural supports, strengths based, individualized services, and accountability. Several domains are parallel to WFI elements; however, WFI-PA domains inquire about the program- and system-level expression of these domains rather than the team or family level. For example, family centeredness on the WFI-PA refers to whether family members participate on state and local decision-making bodies, whereas the WFI items for this domain assess whether family members receiving services can influence the nature of services that their family receives.

For some of the WFI-PA domains, a single item is used to achieve a score. For example, "program duration" is simply the reported number of years the program had delivered services via the wraparound process. Scores for other domains are calculated from figures given for two items. For example, "resource facilitator turnover" is calculated by dividing the number of resource facilitators in the past 12 months by the total number employed at the site. Finally, for several domains, scores are calculated by combining responses to two or more items that are assessed via a Likert format (0, no; 1, somewhat; and 2, yes).

For each WFI-PA domain, criteria were determined for characterizing each site's degree of program and system support for wraparound. Such criteria were set via empirical cutoffs, usually a median split. For each domain, sites were assigned either a score of 1, indicating a relatively high level of program and system support, or 0, indicating a relatively low level of program and system support. For the purposes of the study, scores for the ten domains were also summed to create a total WFI-PA score for each site that could range from 0 to 10. (Readers interested in reviewing the WFI-PA or a description of scoring procedures may contact the first author.)

Results

Fidelity

Table 1 displays mean WFI scores for all sites in the study, as well as site-level means. Across sites, mean scores were 6.59±.27 for the resource facilitator form, 6.22±.38 for the caregiver form, and 6.19±.46 for the youth form. Based on results of a benchmarking exercise using normative data from a more recent version of the WFI ( 30 ), the range and mean of site-level WFI scores were typical of other sites nationwide using the WFI and could be characterized as having fidelity ranging from borderline to above average. One-way analysis of variance (ANOVA) demonstrated significant differences in total fidelity scores across study sites for resource facilitators (F=2.78, df=6 and 186, p=.013, partial η² =.08), caregivers (F=2.52, df=7 and 213, p=.016, partial η² =.08), and youths (F=2.94, df=6 and 151, p=.01, partial η² =.11).

Enlarge table

One-way ANOVA also demonstrated significant differences between respondents (F=11.28, df=2 and 556, p<.001), with post-hoc Scheffé analysis demonstrating that caregivers' and youths' scores were lower than resource facilitators' total scores. These results are consistent with previous studies showing that resource facilitator interviews produce higher total WFI scores than for caregivers and youths ( 19 , 30 ).

Organizational and system supports

To examine the variability across agencies on the organization- and system-level supports, descriptive procedures were used to examine the variability across the eight agencies on the ten domains of the WFI-PA. As shown in Table 2 , caseloads of care managers varied widely, from six to 15 across the eight sites. Care manager turnover ranged from .13 to .50 across sites, with a mean turnover rate of .34±.12 over the previous 12 months. Only six of the eight sites (75 percent) reported having a community collaborative team overseeing the project. Of interest, although sites tended to serve youths across all four major agencies, only two of eight sites (25 percent) reported blending or "braiding" of funds from multiple agencies.

Enlarge table

As shown in Table 2 , among those sites with community collaborative teams, the percentage of team members described as consumers, family members, and family advocates ranged from 0 to 55 percent. Four of the six sites with teams (67 percent) reported having no nonagency community representation (such as clergy, business owners, or politicians), whereas the other two sites (33 percent) reported only 5 and 10 percent representation from this constituency.

Several other characteristics of study sites are worth noting. All sites served a majority of youths in community residential settings (range= 65-100 percent). Most sites reported specific mandates for inclusion of natural supports on wraparound teams. In addition, most respondents reported program mandates for conducting team planning activities using family strengths. Also, most reported using outcomes and satisfaction data to aid individual team processes as well as to improve programs. All sites mandated the recording of goals and action steps during the team planning process, and most reported having flexible funding policies and procedures in place that encourage individualization of treatment.

Association between variables and wraparound adherence

Three standard one-tailed regression analyses were performed between number of organizational and system supports as the independent variable and wraparound adherence scores as the dependent variables. Analyses were performed with SPSS for Windows 11.0 ( 31 ). Results indicated that the number of WFI-PA domains was a significant predictor of fidelity ratings by caregivers (F=6.03, df=1 and 6, p=.025). The unstandardized beta coefficient for this relationship was found to be .125 (SE=.51), meaning that each additional support condition in place at a site improved the mean WFI caregiver scores for the site by an average of .125. A relationship in the hypothesized direction was also found for fidelity ratings by youths ( β =.58; F=2.478, df=1 and 6, p=.088). The unstandardized beta coefficient for this relationship was found to be .117 (SE=.74). The number of WFI-PA domains was not found to be a significant predictor of ratings by resource facilitators on the WFI.

Figure 1 shows mean WFI scores (separately for caregivers, youths, and resource facilitators) for each of the eight study sites, by the number of organizational and system supports for which each site met criteria. As shown, sites met criteria for between two and eight of the WFI-PA domains. Regression lines provided in Figure 1 illustrate an observable trend, whereby sites with greater levels of organizational and system supports demonstrated higher mean WFI scores. This relationship was strongest for caregiver reports of wraparound adherence, followed by youth reports and resource facilitator reports.

Discussion

Findings from this study demonstrate wide variation in the types of supports in place to implement the wraparound process in the host organizations and broader systems of care in which they are embedded. In keeping with the speculations of many researchers and practitioners, the results of our study also showed evidence of an association between such program and system supports and an agency's ability to conform to standards of high-quality wraparound, at least as reported by the recipients of services—parents, caregivers, and youths.

Limitations

This study had several important limitations. The convenience sample of eight sites chose to implement the WFI, which may represent a recruitment bias and certainly restricts generalization of the results. In addition, although study sites proposed to recruit a representative sample of participating families, recruitment was conducted by the sites themselves. This may represent an additional recruitment bias. Power to detect a significant effect was reduced because of the small number of sites sampled. And, as with any correlational design, it is impossible to determine causality from this study. The organizational and system supports may directly influence team-level wraparound implementation, or wraparound implementation may alter administrators' perceptions of supports available.

Another limitation is that criteria for greater or lesser conformance to the ten domains were based on median splits and frequency distributions from the study sample rather than on external criteria. It also should be noted that, although the two instruments used in the study assess different levels of practice (program and system levels for the WFI-PA compared with the team and family levels for the WFI), there is content overlap between the WFI-PA and WFI. This is somewhat unavoidable, given that the study was testing a central hypothesis that program and system factors influence practice at the individual level. Nonetheless, future studies should rely on standardized and validated measures of program and system context for implementation.

Finally, a practical limitation of the study is that it was not designed to be able to determine which specific supports are most important to achieving fidelity. Exploratory analyses suggest that having wraparound-specific supports in place, such as interagency collaboration, flexible funding, and mandates for engaging natural supports, may be particularly strong predictors of wraparound adherence ( 32 ). On the basis of the results of this preliminary study, research is now under way that will include a greater number of sites as well as assessment of outcomes, allowing for more definitive conclusions about relationships between specific contextual conditions, implementation fidelity, and outcomes.

Implications for practice

With these limitations noted, this study provides preliminary support for the model formally proposed by Walker and colleagues ( 23 )—and noted by trainers, providers, and advocates for years—that presence or absence of necessary support conditions will likely influence the quality of service planning and implementation for children and families who receive such services. This is an important finding given growing evidence that greater fidelity will yield more positive outcomes ( 21 , 22 ). These results thus provide an additional link in the logic chain that leads from program and system structures to implementation quality and improved outcomes.

Results of the WFI-PA interviews indicated that even within a sample of communities likely to be well informed about the philosophy and activities of the wraparound process, many did not have the support conditions in place that were considered necessary to implement a high-quality wraparound process ( 23 , 33 ). Although some communities featured supports specific to implementing wraparound, such as community collaborative teams, availability of flexible funds, and processes for ensuring presence of natural supports or nonprofessionals on teams, others did not.

Finding such variation in program and system conditions in a sample of wraparound programs is not surprising. Such results have been observed in both wraparound programs and systems of care ( 23 , 34 ). Such variation is also characteristic of the "science-to-service" gap, whereby implementation of promising models is hindered by conditions in the "real world" that are less than ideal.

The system support conditions that we assessed span several different categories proposed by Walker and colleagues ( 23 ), such as reinforcing components of the practice model, ensuring support to staff, building collaboration and partnerships, and sustaining accountability systems. These categories span both the organizational and system levels.

Given the support for the hypothesis that the cumulative number of facilitative conditions is associated with greater fidelity, results suggest that communities that wish to implement a high-quality wraparound process for individuals with complex needs and their families should carefully examine their organizational contexts. For example, can the lead organization provide adequate training and supervision to facilitators? Will caseloads be low enough to permit full implementation of the model? Will facilitators be able to travel to homes and schools for meetings and have the work flexibility to conduct team meetings in the evening hours? Similarly, there are multiple questions to ask of the child-serving system overall about acceptance and understanding of the wraparound process across partner agencies, the presence of flexible funds to support creative wraparound plans, and many additional indicators of program and system support ( 23 ).

A final implication of this study is that measuring relevant indicators of a program's organizational and system context may help achieve higher-quality practice. Such measures have been developed for use in national studies on mental health service system initiatives for both adults ( 35 ) and children ( 36 ). Such measures are also used in quality assurance packages developed for evidence-based interventions ( 25 ). To support high-quality wraparound implementation, measures such as the WFI-PA or other instruments can be used to assess the "fit" between existing community and program structures and high-quality practice.

Conclusions

In addition to continuing to conduct effectiveness studies, those in the field of mental health service delivery need to better understand how characteristics of organizations and service systems influence service delivery ( 37 ). This exploratory study found support for the proposed relationship between program and system variables and wraparound implementation, contributing additional evidence to this emerging line of research. The study also points to areas in which administrators, researchers, and family members may collaborate to assess the "fit" between community and program structures and high-quality practice.

Such assessments could ultimately guide modifications to treatments to fit the realities of everyday practice. On the other hand, such information may be used to advocate for changes in the policy and funding environments within which interventions are embedded. Although potentially more difficult, the latter option may be more fruitful. Achieving a policy and funding context that supports high-quality practice will be more likely to facilitate positive change than merely tinkering with established treatment approaches.

Acknowledgments

This research received support from several sources, including the Center for Mental Health Services, Substance Abuse and Mental Health Services Administration; the Child, Adolescent, and Family Unit of the Vermont Department of Developmental and Mental Health Services; and ORC Macro, Inc.