Predictive properties of the A-TAC inventory when screening for childhood-onset neurodevelopmental problems in a population-based sample

The Child and Adolescent Twin Study in Sweden, CATSS (for an overview, see [12]), is an ongoing study that aims to track all twins born in Sweden from 1992 on. A telephone interview is conducted with the parents of all participating twins around the children’s ninth birthday. The first three years of the CATSS interviews were conducted with parents of 12-year-old twins as well, in order to increase the number of birth cohorts included in the study. As of January 2010, 8610 parents had responded for 17 220 individual twins (8787 boys and 8433 girls). The overall response rate was 80%. All subjects are protected by the informed consent process and are given the opportunity to withdraw their consent and discontinue their participation at any time. The screening is conducted over the telephone by a market research centre, “Intervjubolaget,” using a computerized version of the A-TAC inventory. All interviews are performed by laypersons, who are given just a brief introduction to research methodology and child mental health problems. The interviewers enter the responses directly into an electronic database. The average interview time for A-TAC in the CATSS is about 30 minutes.

The A-TAC consists of 262 items that address clinical issues: symptom criteria listed in the DSM for NDPs [1], key items for other psychiatric disorders according to the DSM [1], and additional items from published questionnaires and clinical practice. The items are divided into 20 different modules: Motor Control; Perception; Concentration & Attention; Impulsiveness & Activity; Learning, Planning & Organizing; Memory; Language; Social Interaction; Flexibility; Tics; Compulsions; Feeding; Separations; Opposition; Conduct; Anxiety; Mood; Concept of Reality; and Miscellaneous (i.e. items addressing clinically specific problem areas not covered by the other modules: sleep, food fads, severe overweight, bodily functions, and substance abuse). The two modules Concentration & Attention and Impulsiveness & Activity correspond to the definition of AD/HD. Similarly, the modules Language, Social Interaction, and Flexibility reflect the ASDs. Each module includes a number of symptom items coded on a dimensional scale: 0 indicates the item does not apply; 0.5 indicates the item applies or has applied in the past to some extent; and 1 indicates the item applies or has applied in full. All modules except Miscellaneous are organized according to a “gate structure,” in which all interviewees are asked questions that address module-specific core symptoms. There are 96 gate items in all, and only if any of these is at least partially endorsed will the interviewer continue with more detailed symptom questions.

Twins born in the years 1993 to 1995 were eligible for the follow-up at age 15. Those born 1993 to June 1995 had been screened with the A-TAC at age 12, while those born in July to December 1995 had been screened at age 9. A sample of 15-year-olds was selected to create a population-based study group enriched for NDPs in the CATSS-15/DOGSS (Developmental Outcomes in a Genetic twin Study in Sweden) project to clinically assess the outcome of the A-TAC population screening for NDPs. Same-sex twin pairs in whom at least one of the siblings had screened positive for ASDs, AD/HD, TDs, LDs, DCD, and/or behavioural disorders with known NDP comorbidities, such as obsessive compulsive disorder (OCD), oppositional defiant disorder (ODD), conduct disorder (CD), and/or eating disorder (ED) were invited to participate (about 15% of the total population). In addition, a number of randomly selected population controls were included (5% of the total population). From the 1995 cohort, the inclusion criteria were narrowed to include ASDs and AD/HD only [12]. The final cohort selected by these criteria (referred to as the “follow-up” in this paper, see Figure 1) consisted of 860 individuals, of whom 452 (52%) consented to participate (247 screen-positive children, 157 screen-negative co-twins, and 46 randomly chosen controls matched for sex and age). For two of the included co-twins, full A-TAC information was lacking; these individuals were therefore not included in the longitudinal analyses.

All screen-positive children and their siblings were diagnosed at age 15 according to the DSM criteria [1], using the Schedule for Affective Disorders and Schizophrenia for School-Age Children: Kiddie – SADS – Present and Lifetime Version (K-SADS-PL) [13] as the principal diagnostic interview. Each pair was assessed in one day by two specially trained psychologists who were blind to the previous screening results and performed their assessments independently, each seeing only one of the twins and a caregiver. In addition to the K-SADS-PL, the following diagnostic protocols and tests were used: the Asperger Syndrome (and high-functioning autism) Diagnostic Interview [14], the Wechsler Intelligence Scale for Children [15], the QbTest [16], and structured diagnostic assessment used by the Paris Autism Research International Sib pair study [17]. By interviewing the parent and the child individually, the interviewer was able to record separate interview scores for each diagnosis listed in the K-SADS-PL. Based on all sources of information available, the interviewer subsequently made a diagnostic summary rating according to the DSM criteria and the K-SADS-PL algorithms. As a final step, the clinician, together with a senior clinical expert (co-author ENS, a clinical board-licensed specialist in child and adolescent psychiatry), scrutinized the summary ratings for all cases assessed by the clinicians, with the aim of ascertaining definitive diagnoses based on all accessible information from the clinical evaluations. These consensus conferences were also conducted without access to the results of the screening interviews or the key to pairing the adolescents. For the ASDs, these diagnoses were further validated by a consensus on diagnostic criteria by several experts with access to all data except the scores from the baseline A-TAC interviews.

The clinical diagnoses from the follow-up at 15 years of age (CATSS-15/DOGSS) were used as dependent variables, while the A-TAC scores from the CATSS-9/12 were used as independent predictors. Plotting sensitivity and specificity on a receiver operating characteristics (ROC) curve can help to determine the usefulness of the instrument and the optimal cut-off value (inflection point). On a ROC curve, the true-positive rate (sensitivity) is plotted against the false-positive rate (1 - specificity) for each possible threshold of the instrument. An ideal instrument will have a cut-off value that gives both a sensitivity of 1 (100% identification of true cases) and a specificity of 1 (100% exclusion of non-cases). Therefore, the score coming closest to this point determines the cut-off value. In reality, inventories have ROC curves between two extremes, and the greater the area under the curve (AUC), the more closely the instrument approximates the ideal. Hence, an AUC equal to 0.50 signals random prediction, an AUC of 0.60–0.70 indicates poor validity, 0.70–0.80 is fair, 0.80–0.90 is good, and AUC > 0.9 shows excellent validity [18]. The A-TAC cut-off values were determined and validated by two earlier studies [8, 9] and used to assign screening status from the A-TAC scores.

First, the number of screen-positive children with a neurodevelopmental diagnosis identified in the screening phase was compared to the number of children who were given a diagnosis in the clinical examination three years later. Next, predictive psychometrics were calculated: first the AUCs and then the probabilistic measures of sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and diagnostic odds ratio (DOR), using both the “low” cut-off (optimal for screening purposes) and the “high” cut-off (allotted for use in research as a proxy for clinical diagnosis), both established in the earlier study [9]. PPV is the ability of the instrument to correctly identify children who truly have the condition, and NPV is its ability to correctly identify those without the condition. The DOR of a screening test is the ratio of the odds of those with the condition being screen-positive to the odds of the non-afflicted being screen-positive. The value of a DOR ranges from zero to infinity, with higher values indicating better discriminatory test performance. The DOR increases with the test’s ability to discriminate, and rises steeply when sensitivity or specificity becomes near perfect [19, 20].

All statistics were calculated using the SPSS software package 19.0 and the online interactive statistical resource: StatPages [21].

All participants consented to the study after receiving written and oral information. Analyses were performed on anonymized data files. The study protocol accorded with the Helsinki declaration and was approved by the ethical review board of Karolinska Institutet, Solna, Sweden (No. 02–289).

Among the 247 screen-positive cases, 198 children (80%) were screen-positive for an NDP: 27 children (14%) were screen-positive for ASDs, 95 (48%) for AD/HD, 74 (37%) for LDs, 35 (18%) for TDs, and 38 (19%) for DCD (See Table 1, Baseline. Note that the percentage total does not add up due to the overlap of diagnoses).

Of the 27 children screen-positive for ASDs, 13 (48%) were diagnosed with an ASD at the follow-up, 12 (44%) were diagnosed with another NDP, 1 (4%) was diagnosed with a non-NDP mental health problem, and 1 (4%) had no DSM diagnosis at all (see Table 1, Clinical Diagnoses).

Of the 95 children who had screened positive for AD/HD, 46 (48%) were diagnosed with AD/HD at the follow-up, 15 (16%) with another NDP, 11 (12%) with a non-NDP mental disorder, and 23 (24%) had no DSM diagnosis.

Of the 74 who had screened positive for LDs, 12 (16%) were diagnosed with an LD at the follow-up, 24 (32%) with another NDP, 12 (16%) with another mental disorder (non-NDP), and 26 (35%) had no DSM diagnosis.

Of the 35 children who had screened positive for TDs, 21 (60%) were diagnosed with a TD at the follow-up, 7 (20%) with another NDP, 2 (6%) with another mental disorder (non-NDP), and 5 (14%) had no DSM diagnosis.

Of the 198 children who screened positive for any NDP at baseline, 108 (55%) received at least one clinical DSM diagnosis of an NDP at follow-up. Of the 252 children who were screen-negative for NDPs at baseline (the majority of whom were screen-negative co-twins to screen-positive siblings, and therefore constituted a high-risk group), 51 (20%) received at least one clinical diagnosis of an NDP, while 201 (80%) received no clinical NDP diagnosis at follow-up. A 2 × 2 contingency table of true and false positives and negatives for all NDPs yielded a sensitivity of 0.68, a specificity of 0.69, and a DOR of 4.7 for the A-TAC’s ability to capture any NDP (Table 1).

Among the screen-negative siblings, none were diagnosed with an ASD, 20 (13%) were diagnosed with AD/HD, two (1%) were diagnosed with an LD, and 16 (10%) were diagnosed with a TD.

Among controls, none was diagnosed with an ASD, but 3 (7%) were diagnosed with AD/HD, 2 (4%) with a TD, and 1 (2%) was diagnosed with an LD.

Predictive psychometrics are reported in Table 2, the AUCs are followed by the cut-off values, the low cut-off for screening possible caseness was used at baseline, and the supplementary high cut-off values for research diagnostic purposes are shown solely to complete the psychometric presentation. The ASDs domain in the A-TAC showed excellent predictive ability for clinical diagnoses of ASDs at age 15 (AUC = 0.91), followed by fair predictive ability for the AD/HD, LDs, and TDs modules (AUC = 0.77–0.80). Figures 2, 3, 4 and 5 illustrates the ROC plots for each of the targeted disorders. Overall, the low cut-off scores were coupled with higher sensitivity, while the high cut-off scores were, as expected, associated with better specificity. Specificity was generally higher than sensitivity, except for LDs. The low cut-off scores for each diagnosis showed a specificity and a sensitivity ≥ 64%. Table 2 shows the high cut-off scores, which had a specificity ≥ 93% for these disorders. The TDs module has just one cut-off value, with a sensitivity of 45% and a specificity of 93% in the general population group. ASDs yielded the highest DOR with a value of 29 using the low cut-off, and showed an even higher DOR (46) using the high cut-off. Among specific diagnoses, AD/HD and LDs showed the lowest DORs (6 for both) with the low cut-off, and more elevated values (8 and 15, respectively) using the high cut-off.

Because study questions on diagnostic accuracy generally evaluate the association between inventory scores and health status, a cross-sectional design is a natural basic design option. However, this basic design has various modifications, each with specific pros and cons in terms of scientific requirements, burden for the study subjects, and efficient use of resources. In this case, a major factor that affects the instrument’s performance in relation to clinical diagnoses of NDPs is the time between the behaviour sampled and the clinical examination. There was a three-year delay between the parental assessment on the A-TAC and the clinical follow-up. Asking parents to rate current behaviour when symptoms of NDPs may be at their most prototypical, and then clinically examining the children three years later could have contributed to the difficulties in differentiating between NDPs (apart from ASD, which is one of the most constant NDPs), at least between the ages of 9 or 12 and 15. Even if some of the major child psychiatric problem constellations are established by age 12, the complex psychosocial problems associated with puberty that emerge around the time of the clinical examination may interfere with interpreting the results. Moreover, the A-TAC showed comparatively low DOR/AUC for disorders other than ASDs (especially AD/HD). This may be attributed principally to the time lag between the screening and the clinical assessment and perhaps also to a “twin sample bias” suggested to be inherent in using a screen-negative group that largely consists of genetically at-risk siblings. Given that NDPs are under complex and multivariate genetic influences and tend to follow a waxing and waning course, a longitudinal twin sample may compromise probabilistic measures, including NPVs and PPVs, since discordant co-twins will be more likely than other pairs to oscillate above or under a cut-off. Despite this reasoning, however, the notion of a twin sample bias is dubious since numerous studies have reported that twins differ only marginally from singletons [31‐33]; even if same-sex twins may not be representative of the general population, is it unlikely that this circumstance would have had any substantial effects on the results presented here.

The strengths of this study lie in its investigation of the efficacy of the A-TAC in a population-based cohort of screen-positive children, their screen-negative siblings, and controls and in its rigorous assessments of neuropsychological outcomes. Psychiatric interviews were carried out for all children in the study using the K-SADS-PL schedule, and consensus diagnoses were made by specially trained psychologists and an experienced child psychiatrist.