Background
In North America, one out of every five children experiences a mental health concern, such as depression, anxiety, attention-deficit hyperactivity disorder, or conduct problems [
1,
2]. The stability, persistence, and aversive long-term outcomes of childhood illness are evident across the lifespan [
3]. Over 50–75% of adult mental health issues have their onset in childhood or adolescence [
4], and children who experience mental health issues are at increased risk for academic underachievement [
5,
6], underemployment [
7], criminal activity [
8,
9], and risk for suicide [
10]. Given the compounding effects of mental health issues over time, as well as the staggering economic burden of mental health costs in North America [
1,
11], ensuring children have access to timely, responsive, and integrated services is critically important; however, the children’s mental health care system is underfunded and fragmented, often referred to as the “orphan’s orphan” of health care [
12]. Indeed, over 75% of children who experience mental health issues, do not receive access to appropriate treatment [
13].
For over the past two decades, the systems-of-care philosophy in mental health has recognized the need to respond to a fragmented service system through greater integration and coordination [
14]. In response to the need for a coordinated approach to the delivery of mental health care in Canada, there have been significant advances within the adult mental health sector. More specifically, the call to develop an effective information system that provides a comprehensive, scientifically rigorous approach to mental health assessments led to the Resident Assessment Instrument-Mental Health (RAI-MH). This instrument was developed by interRAI (i.e., an international not-for-profit collective of researchers and clinicians from over thirty countries committed to improving the care of vulnerable persons through a seamless approach to assessment across a variety of service sectors) in collaboration with the Ontario Joint Policy and Planning Committee [
15]. In 2006, the Ontario Ministry of Health and Long-Term Care mandated the use of the interRAI RAI-MH, given its ability to provide a comprehensive assessment of adult mental health needs, and support evidenced-based care planning across service settings. Moreover, the assessment has been used for the purposes of outcome measurement, quality improvement, and case-mix, and compliments interRAI’s existing suite of instruments for use across a variety of service sectors (e.g., home care, acute care, long-term care, etc.).
Despite the advances in adult mental health care in Canada, the children’s mental health care system has continued to lack a comprehensive and psychometrically sound tool for integrated use across a variety of service sectors (e.g., inpatient/outpatient mental health, hospitals, youth justice sites, schools, etc.). Although numerous assessments have been developed to assess child functioning, these assessments are often narrowly focused (e.g., assesses symptoms for a given mental health problem, such as depression). As a result, multiple assessments are often required to obtain a comprehensive understanding of the child/youth’s mental health needs, resulting in redundancies in information collected, additional clinical time and resources, as well as increased assessor burden on children and their families [
16,
17]. Moreover, previous assessments have yet to facilitate an integrated and coordinated approach to children’s mental health across various service sectors, or facilitate a lifespan approach to the delivery of mental health care (i.e., from childhood through to late adulthood). Given the strong need for a coordinated approach, the interRAI Child and Youth Mental Health assessment (ChYMH) was developed to facilitate a comprehensive, standardized, and integrated approach to the delivery of mental health services in Canada to further support a life course approach to assessment.
The development of the ChYMH was a rigorous and arduous process that included the involvement of over 100 clinical experts from nine over countries (e.g., Canada, United States, Poland, Belgium, Finland, Sweden, Netherlands, Australia and Czech Republic). The development process included several stages, to ensure a peer-reviewed high quality end product with the potential for widespread uptake not only in Canada, but around the globe. First, a core research team in Canada conducted extensive literature reviews on all available existing children and youth mental health assessments, the etiology of mental health issues among children and youth ages 4–18 years of age, as well as the best-practice guidelines for children’s mental health care. As part of this literature review, domains requiring assessment were identified (e.g., mental health, physical health, cognitive functioning, strengths and resilience, stress and trauma, etc.), and relevant items pertinent to children and youth were adapted from other interRAI instruments (e.g., RAI-MH). After the literature review was conducted, panels of expert working groups were established (including clinicians, researchers, front line mental health workers, policy makers, etc.). These groups specifically focused on ChYMH item creation to ensure that identified domains were well-represented, items were clinically useful and relevant, and that the instrument had strong face validity. Once items were finalized by these working groups, the items were then presented to two international groups, the interRAI Network of Excellence in Mental Health (iNMH) and the interRAI Instrument and Survey Development Committee (ISD). These committees collectively included over 40 internationally renowned experts in mental health, survey development, measurement and content. The expert committees provided feedback on each item, which was then incorporated by the working groups into the ChYMH. The built-in collaborative action plans (which provide real-time evidence-informed recommendations for care planning around areas of risk), were developed through a similar process.
The interRAI Child and Youth Mental Health assessment has had widespread uptake in Ontario as a standard of care at over 60 mental health agencies in the province. Moreover, the ChYMH has led to the development of complimentary child and youth assessments [e.g., the Child and Youth Mental Health Assessment for youth with developmental disabilities (ChYMH-DD)] that are designed to act in combination as an integrated children’s mental health suite. Despite the widespread uptake of the ChYMH, however, published research on the psychometric properties of the instrument is lacking. In contrast, there have been numerous reliability and validity studies conducted across the family of interRAI instruments for adults. Indeed, the adult interRAI assessments have been shown to demonstrate good psychometric properties, including strong inter-rater reliability, convergent validity, and inter-item reliability [
15,
18‐
21]. Although the ChYMH was adapted from the existing interRAI assessments where possible, additional research is still needed to specifically examine the psychometric properties of the ChYMH among clinically referred children and youth.
Present study
Although the ChYMH has already had widespread uptake in Ontario because of its strong clinical utility and multiple applications (e.g., care planning, program evaluation, case-mix), research on the psychometric properties of the instrument has yet to be published. In the present study, we provide an empirical investigation of the psychometric properties of the ChYMH. More specifically, data collected from several mental health agencies in Ontario was used to assess the inter-item reliability of several embedded scales on the ChYMH (i.e., Aggressive/Disruptive Behavior Scale Anhedonia Rating Scale, Anxiety Scale, Caregiver Distress Scale, Communication Scale, Cognitive Functioning Scale, Depressive Symptoms Scale, Distractibility/Hyperactivity Scale, Peer Conflict Scale, Sleep Difficulties Scale). In addition, participants of the larger sample agreed to complete additional existing child and youth mental health measures with relevant domains, to assess scale criterion validity.
Discussion
The ChYMH was developed by interRAI in collaboration with leading experts in children’s mental health, to facilitate a comprehensive, standardized and integrated (cross-sector) approach to the assessment of children’s mental health needs. Moreover, the ChYMH was designed to foster continuity in assessment across the lifespan (from early childhood through to late adulthood) within the interRAI suite of instruments. Despite the widespread uptake of the ChYMH in Canada, there is a lack of published research on the psychometric properties of the instrument. To address this important limitation, in the present study the inter-item reliability and criterion validity of several scales on the ChYMH were evaluated using data collected from 15 mental health agencies in Ontario. It was found that the embedded scales on the ChYMH (i.e., Aggressive/Disruptive Behavior Scale, Anhedonia Rating Scale, Anxiety Scale, Caregiver Distress Scale, Communication Scale, Cognitive Functioning Scale, Depressive Symptoms Scale, Distractibility/Hyperactivity Scale, Peer Conflict Scale, Sleep Difficulties Scale) demonstrated strong inter-item reliability, and correlated well with several criterion measures. Findings support the clinical utility of the ChYMH for use among clinically referred children and youth.
The findings of the present study are consistent with previous research demonstrating that interRAI adult assessment instruments have strong psychometric properties, including inter-item consistency and criterion validity [
15,
18‐
21]. All of the embedded scales assessed on the ChYMH demonstrated strong inter-item reliability, which suggests that the extensive construction process of the ChYMH yielded scales with items assessing conceptually similar domains well. Further, ChYMH scales were found to correlate with clinically similar domains on existing psychological assessments (e.g., Child Behavior Checklist, Brief Child and Family Phone Interview). In particular, ChYMH scales assessing internalizing behaviors (e.g., depression, anxiety) and externalizing behaviors (e.g., aggressive symptoms) showed strong correlations with other problem behavior assessment scales. Given that both internalizing and externalizing behaviors are widely occurring mental health concerns among clinically referred children and youth, the ChYMH can provide a comprehensive assessment of these problem areas [
4,
34].
Also noteworthy, are our findings around sex and age differences on the scales. More specifically, it was found that female children and youth were at higher risk for depressive symptoms and caregiver distress, whereas males were at higher risk for aggressive/disruptive behavior, distractibility/hyperactivity, communication problems and cognitive functioning difficulties. Our findings are consistent with a larger body of research, which has found that males tend to present more with externalizing behaviors, whereas females tend to report higher levels of internalizing behaviors [
35‐
37]. Moreover, our findings suggest that males may be more likely to be referred for clinical care than females, due to issues related to cognitive difficulties, problems with executive functioning (e.g., attention, memory) and communication issues (e.g., ability to speak with others), which have found to be associated with distractibility/hyperactivity in previous research [
38,
39].
It also was found that children (ages 4–12) reported more aggressive/disruptive behavior, distractibility/hyperactivity, as well as communication and cognitive functioning issues relative to youth (ages 13–18). Given that externalizing behaviors tend to have their onset in early childhood, it is not surprising that children, compared to youth, reported higher levels of risk for aggressive, disruptive, and hyperactive behavior at time of intake [
35,
40]. Moreover, attention problems may decline as children age, and regulatory capabilities strengthen with further brain development [
41,
42]. In contrast, compared to children, youth were at greater risk for depressive symptoms, and anhedonia – which coincides with later onset of depressive symptomology in adolescence [
43]. Research has shown that individuals lacking strong interpersonal relationships are at heightened risk for depressive symptoms, which is also consistent with findings that youth reported higher levels of peer conflict [
44], compared to children.
Overall, the findings of the present study offer further support that the ChYMH is a rigorously developed comprehensive assessment which can serve to support the early identification of mental health concerns among children and youth, as well as facilitate evidence-informed care planning, program evaluation, and outcome measurement. Indeed, although the present study utilized data collected from children and youth at time of intake into clinical treatment, the ChYMH is designed for re-assessment specifically to monitor change over time among children and youth. The ChYMH may be preferable to other assessments that are often narrowly focused by providing a more comprehensive assessment. Moreover, the ChYMH can also work in combination with other interRAI instruments to provide clinical information about change and stability in mental health over time among persons accessing care across the lifespan.
In working toward developing a coordinated approach to the assessment and delivery of children’s mental health care, the ChYMH can be used at both inpatient and outpatient mental health agencies to identify emerging or existing mental health concerns among children and youth, and support evidenced-informed and consistent responses to children’s mental health concerns. Other interRAI Child and Youth instruments have been developed and adapted for use in other service sectors (e.g., emergency rooms, schools) to create a seamless approach to assessment, prioritization, and triaging to support referrals from one service sector to another. This integrated approach also supports waitlist reduction strategies, provides tailored assessment modules for individualized care and facilitates information sharing among service providers, hospitals, and agencies, while preventing assessment burden and duplication [
45]. Importantly, the ChYMH was designed to complement the existing suite of interRAI assessments to allow for meaningful comparisons across service sectors and across the life course. Moving forward, data collected with the ChYMH could be linked longitudinally to examine developmental trajectories related to mental and physical health later in life, as well as to identify early predictors of more life course persistent symptomology and potential mitigating factors fostering resilience (e.g., effective early supports for families). In addition to contributing to continuity of care and assisting with transitions across service sectors, it will eventually allow opportunities to study rare diseases and disorders across the lifespan.
Despite the many strengths of the present study, including the use of a comprehensive mental health assessment designed specifically for children and youth, as well as a large clinical sample, the present study is not without limitations. First, the present sample consisted of children and youth referred for clinical care in the province of Ontario. Although the sample was large, and representative of the population of children and youth receiving clinical care in Ontario, additional studies assessing the use of the ChYMH in other clinical populations in varied geographic regions would increase confidence around the generalizability of the findings. In addition, despite the strong response rate among participants (given that the assessment was a standard of care), we could not compare youth (and their families) who refused to complete the assessment to youth who had an assessment completed. It is possible that families who agreed to the assessment differ relative to families who did not, and our results may not be generalizable to all families receiving care in Ontario. In addition, analyses for the criterion validity of the scales were conducted using smaller subsets of the large sample, resulting in less statistical power. Although having a larger subsample of participants complete the criterion measures may have resulted in stronger associations between the ChYMH scales and the criterion measures, conducting such studies are extremely challenging. Given the overwhelming demands of the children’s mental health care system in Canada, every effort is made to ensure that assessor burden on clinical staff, and the families of children receiving care, is limited. Thus, it is difficult to conduct additional assessments at time of intake into treatment (this limitation has also been noted by other researchers conducting comprehensive child and youth assessment) [
46]. Nevertheless, our study provides the first published empirical investigation of the psychometric properties of the ChYMH, and is especially timely given its widespread uptake across the province of Ontario (i.e., the instrument is currently being supported for use in over 60 mental health agencies).
Conclusions and directions for future research
The ChYMH was developed to support a coordinated and integrated (cross-sector) approach to the delivery of children’s mental health care services. Using data collected in Canada, it was found that the ChYMH demonstrated strong inter-consistency reliability, as well as good criterion validity, among a large sample of clinically referred children and youth. The present findings provide the first empirical data supporting the psychometric properties of the ChYMH, which has already had widespread uptake given its clinical utility. To extend these finding, additional research on the psychometric properties of the ChYMH with even larger samples of children and youth is warranted. In particular, assessments in different clinical care settings could allow for an examination of the psychometric properties of the ChYMH across diverse samples (e.g., inpatient vs outpatient vs primary care provider). Moreover, the ChYMH is also being translated for use in different languages, including French, thereby providing opportunities for cross-cultural comparisons. To further validate the assessment, it would also be worthwhile to include lab-based assessments of child behavior as criterion measures, to ensure the measures are assessing underlying mental health domains well across a variety of contexts. In addition, an important direction for future research will be to assess the test re-test reliability of the ChYMH using multiple assessment points, to examine the ChYMH’s effectiveness in assessing stability and change in mental health over time.
Given that the use of a standardized and comprehensive assessment could improve upon assessment information currently collected from non-standardized and non-integrated assessment instruments, research on the psychometric properties of the ChYMH is critically important. Our study provides strong support for the clinical utility of the instrument with clinically referred children and youth. Moving forward, the widespread usage of the ChYMH across various service providers could facilitate greater information sharing across service sectors (e.g., schools, mental health agencies, youth justice facilities, hospitals, daycares), ultimately to improve service system integration [
47].
Acknowledgements
The authors would like to thank the Child and Parent Resource Institute for their assistance in the coordination of data collection, as well as their efforts around implementation.