Skip to main content
Erschienen in: Prevention Science 1/2018

22.04.2017

Addressing Methodologic Challenges and Minimizing Threats to Validity in Synthesizing Findings from Individual-Level Data Across Longitudinal Randomized Trials

verfasst von: Ahnalee Brincks, Samantha Montag, George W. Howe, Shi Huang, Juned Siddique, Soyeon Ahn, Irwin N. Sandler, Hilda Pantin, C. Hendricks Brown

Erschienen in: Prevention Science | Sonderheft 1/2018

Einloggen, um Zugang zu erhalten

Abstract

Integrative Data Analysis (IDA) encompasses a collection of methods for data synthesis that pools participant-level data across multiple studies. Compared with single-study analyses, IDA provides larger sample sizes, better representation of participant characteristics, and often increased statistical power. Many of the methods currently available for IDA have focused on examining developmental changes using longitudinal observational studies employing different measures across time and study. However, IDA can also be useful in synthesizing across multiple randomized clinical trials to improve our understanding of the comprehensive effectiveness of interventions, as well as mediators and moderators of those effects. The pooling of data from randomized clinical trials presents a number of methodological challenges, and we discuss ways to examine potential threats to internal and external validity. Using as an illustration a synthesis of 19 randomized clinical trials on the prevention of adolescent depression, we articulate IDA methods that can be used to minimize threats to internal validity, including (1) heterogeneity in the outcome measures across trials, (2) heterogeneity in the follow-up assessments across trials, (3) heterogeneity in the sample characteristics across trials, (4) heterogeneity in the comparison conditions across trials, and (5) heterogeneity in the impact trajectories. We also demonstrate a technique for minimizing threats to external validity in synthesis analysis that may result from non-availability of some trial datasets. The proposed methods rely heavily on latent variable modeling extensions of the latent growth curve model, as well as missing data procedures. The goal is to provide strategies for researchers considering IDA.
Anhänge
Nur mit Berechtigung zugänglich
Literatur
Zurück zum Zitat Barch, D. M., Gotlib, I. H., Bilder, R. M., Pine, D. S., Smoller, J. W., Brown, C. H., …, Farber, G. K. (2016). Common measures for national institute of mental health funded research. Biological Psychiatry, 79, e91–e96. doi:10.1016/j.biopsych.2015.07.006. Barch, D. M., Gotlib, I. H., Bilder, R. M., Pine, D. S., Smoller, J. W., Brown, C. H., …, Farber, G. K. (2016). Common measures for national institute of mental health funded research. Biological Psychiatry, 79, e91–e96. doi:10.​1016/​j.​biopsych.​2015.​07.​006.
Zurück zum Zitat Bauer, D. J., Howard, A. L., Baldasaro, R. E., Curran, P. J., Hussong, A. M., Chassin, L., & Zucker, R. A. (2013). A trifactor model for integrating ratings across multiple informants. Psychological Methods, 18, 475–493. doi:10.1037/a0032475.CrossRefPubMedPubMedCentral Bauer, D. J., Howard, A. L., Baldasaro, R. E., Curran, P. J., Hussong, A. M., Chassin, L., & Zucker, R. A. (2013). A trifactor model for integrating ratings across multiple informants. Psychological Methods, 18, 475–493. doi:10.​1037/​a0032475.CrossRefPubMedPubMedCentral
Zurück zum Zitat Bollen, K., & Lennox, R. (1991). Conventional wisdom on measurement: a structural equation perspective. Psychological Bulletin, 110, 305–314.CrossRef Bollen, K., & Lennox, R. (1991). Conventional wisdom on measurement: a structural equation perspective. Psychological Bulletin, 110, 305–314.CrossRef
Zurück zum Zitat Brown, C. H. (1983). Asymptotic comparison of missing data procedures for estimating factor loadings. Psychometrika, 48, 269–291.CrossRef Brown, C. H. (1983). Asymptotic comparison of missing data procedures for estimating factor loadings. Psychometrika, 48, 269–291.CrossRef
Zurück zum Zitat Brown, C. H. (1990). Protecting against nonrandomly missing data in longitudinal studies. Biometrics, 46, 143–155.CrossRefPubMed Brown, C. H. (1990). Protecting against nonrandomly missing data in longitudinal studies. Biometrics, 46, 143–155.CrossRefPubMed
Zurück zum Zitat Brown, C. H., Wyman, P. A., Brinales, J. M., & Gibbons, R. D. (2007). The role of randomized trials in testing interventions for the prevention of youth suicide. International Review of Psychiatry, 19, 617–631. doi:10.1080/09540260701797779.CrossRefPubMed Brown, C. H., Wyman, P. A., Brinales, J. M., & Gibbons, R. D. (2007). The role of randomized trials in testing interventions for the prevention of youth suicide. International Review of Psychiatry, 19, 617–631. doi:10.​1080/​0954026070179777​9.CrossRefPubMed
Zurück zum Zitat Brown, C. H., Sloboda, Z., Faggiano, F., Teasdale, B., Keller, F., Burhart, G., …, Prevention Science and Methodology Group (2013). Methods for synthesizing findings on moderation effects across multiple randomized trials. Prevention Science, 14, 144–156. doi:10.1007/s11121-011-0207-8.CrossRefPubMedPubMedCentral Brown, C. H., Sloboda, Z., Faggiano, F., Teasdale, B., Keller, F., Burhart, G., …, Prevention Science and Methodology Group (2013). Methods for synthesizing findings on moderation effects across multiple randomized trials. Prevention Science, 14, 144–156. doi:10.​1007/​s11121-011-0207-8.CrossRefPubMedPubMedCentral
Zurück zum Zitat Brown, C. H., Brincks, A., Huang, S., Perrino, T., Cruden, G., Pantin, H., …, Sandler, I. (2016). Two-year impact of prevention programs on adolescent depression: An integrative data analysis approach. Prevention Science, 1–21. Brown, C. H., Brincks, A., Huang, S., Perrino, T., Cruden, G., Pantin, H., …, Sandler, I. (2016). Two-year impact of prevention programs on adolescent depression: An integrative data analysis approach. Prevention Science, 1–21.
Zurück zum Zitat Edwards, M. C., & Wirth, R. J. (2012). Valid measurement without factorial invariance: a longitudinal example. In J. R. Harring, G. R. Hancock, J. R. Harring, & G. R. Hancock (Eds.), Advances in longitudinal methods in the social and behavioral sciences (pp. 289–311). Charlotte: IAP Information Age Publishing. Edwards, M. C., & Wirth, R. J. (2012). Valid measurement without factorial invariance: a longitudinal example. In J. R. Harring, G. R. Hancock, J. R. Harring, & G. R. Hancock (Eds.), Advances in longitudinal methods in the social and behavioral sciences (pp. 289–311). Charlotte: IAP Information Age Publishing.
Zurück zum Zitat Gibbons, R. D., & Hedeker, D. R. (1992). Full-information item bi-factor analysis. Psychometrika, 57, 423–436.CrossRef Gibbons, R. D., & Hedeker, D. R. (1992). Full-information item bi-factor analysis. Psychometrika, 57, 423–436.CrossRef
Zurück zum Zitat Hendershot, T., Pan, H., Haines, J., Harlan, W. R., Marazita, M. L., McCarty, C. A., …, Hamilton, C. M. (2015). Using the PhenX Toolkit to add standard measures to a study. Current Protocols in Human Genetics, 86, 1.21.21–17. doi:10.1002/0471142905.hg0121s86. Hendershot, T., Pan, H., Haines, J., Harlan, W. R., Marazita, M. L., McCarty, C. A., …, Hamilton, C. M. (2015). Using the PhenX Toolkit to add standard measures to a study. Current Protocols in Human Genetics, 86, 1.21.21–17. doi:10.​1002/​0471142905.​hg0121s86.
Zurück zum Zitat Howe, G. W., Dagne, G., Brincks, A., & Beardslee, W. (2016). Evaluating construct equivalence and harmonizing measurement of adolescent depression when synthesizing results across multiple studies. Submitted for publicaton. Howe, G. W., Dagne, G., Brincks, A., & Beardslee, W. (2016). Evaluating construct equivalence and harmonizing measurement of adolescent depression when synthesizing results across multiple studies. Submitted for publicaton.
Zurück zum Zitat Lavori, P. W., Brown, C. H., Duan, N., Gibbons, R. D., & Greenhouse, J. (2008). Missing data in longitudinal clinical trials. Part A. Design and conceptual issues. Psychiatric Annals, 38, 784–792.CrossRefPubMedPubMedCentral Lavori, P. W., Brown, C. H., Duan, N., Gibbons, R. D., & Greenhouse, J. (2008). Missing data in longitudinal clinical trials. Part A. Design and conceptual issues. Psychiatric Annals, 38, 784–792.CrossRefPubMedPubMedCentral
Zurück zum Zitat McArdle, J. J. (1988). Dynamic but structural equation modeling of repeated measures data. In J. R. Nesselroade & R. B. Cattell (Eds.), Handbook of multivariate experimental psychology (pp. 561–614). New York: Plenum Press.CrossRef McArdle, J. J. (1988). Dynamic but structural equation modeling of repeated measures data. In J. R. Nesselroade & R. B. Cattell (Eds.), Handbook of multivariate experimental psychology (pp. 561–614). New York: Plenum Press.CrossRef
Zurück zum Zitat McCarty, C. A., Huggins, W., Aiello, A. E., Bilder, R. M., Hariri, A., Jernigan, T. L., …, Zeng, Y. (2014). PhenX RISING: Real world implementation and sharing of PhenX measures. BMC Medical Genomics, 7, 16. doi:10.1186/1755-8794-7-16. McCarty, C. A., Huggins, W., Aiello, A. E., Bilder, R. M., Hariri, A., Jernigan, T. L., …, Zeng, Y. (2014). PhenX RISING: Real world implementation and sharing of PhenX measures. BMC Medical Genomics, 7, 16. doi:10.​1186/​1755-8794-7-16.
Zurück zum Zitat Muthén, B. O., Brown, C. H., Masyn, K., Jo, B., Khoo, S. T., Yang, C. C., …, Liao, J. (2002). General growth mixture modeling for randomized preventive interventions. Biostatistics, 3, 459–475. doi:10.1093/biostatistics/3.4.459. Muthén, B. O., Brown, C. H., Masyn, K., Jo, B., Khoo, S. T., Yang, C. C., …, Liao, J. (2002). General growth mixture modeling for randomized preventive interventions. Biostatistics, 3, 459–475. doi:10.​1093/​biostatistics/​3.​4.​459.
Zurück zum Zitat Muthén, B. O., Brown, C. H., Leuchter, A., & Hunter, A. (2010). General approaches to analysis of course: Applying growth mixture modeling to randomized trials of depression medication. In P. E. Shrout, K. M. Keyes, & K. Ornstein (Eds.), Causality and psychopathology: finding the determinants of disorders and their cures (pp. 159–178). Washington: American Psychiatric Publishing. Muthén, B. O., Brown, C. H., Leuchter, A., & Hunter, A. (2010). General approaches to analysis of course: Applying growth mixture modeling to randomized trials of depression medication. In P. E. Shrout, K. M. Keyes, & K. Ornstein (Eds.), Causality and psychopathology: finding the determinants of disorders and their cures (pp. 159–178). Washington: American Psychiatric Publishing.
Zurück zum Zitat Pan, H., Tryka, K. A., Vreeman, D. J., Huggins, W., Phillips, M. J., Mehta, J. P., …, Ramos, E. M. (2012). Using PhenX measures to identify opportunities for cross-study analysis. Human Mutation, 33, 849–857. doi: 10.1002/humu.22074. Pan, H., Tryka, K. A., Vreeman, D. J., Huggins, W., Phillips, M. J., Mehta, J. P., …, Ramos, E. M. (2012). Using PhenX measures to identify opportunities for cross-study analysis. Human Mutation, 33, 849–857. doi: 10.​1002/​humu.​22074.
Zurück zum Zitat Pathak, J., Pan, H., Wang, J., Kashyap, S., Schad, P. A., Hamilton, C. M., …, Chute, C. G. (2011). Evaluating phenotypic data elements for genetics and epidemiological research: Experiences from the eMERGE and PhenX Network Projects. AMIA Summits on Translational Science Proceedings, 2011, 41–45. Pathak, J., Pan, H., Wang, J., Kashyap, S., Schad, P. A., Hamilton, C. M., …, Chute, C. G. (2011). Evaluating phenotypic data elements for genetics and epidemiological research: Experiences from the eMERGE and PhenX Network Projects. AMIA Summits on Translational Science Proceedings, 2011, 41–45.
Zurück zum Zitat Perrino, T., Howe, G., Sperling, A., Beardslee, W., Sandler, I., Shern, D., Brown, C. H.(2013). Advancing science through collaborative data sharing and synthesis. Perspectives on Psychological Science, 8(4), 433–444. doi:10.1177/1745691613491579. Perrino, T., Howe, G., Sperling, A., Beardslee, W., Sandler, I., Shern, D., Brown, C. H.(2013). Advancing science through collaborative data sharing and synthesis. Perspectives on Psychological Science, 8(4), 433–444. doi:10.​1177/​1745691613491579​.
Zurück zum Zitat Perrino, T., Beardslee, W., Bernal, G., Brincks, A., Cruden, G., Howe, G., …, Brown, C. H. (2015). Toward scientific equity for the prevention of depression and depressive symptoms in vulnerable youth. Prevention Science, 16, 642–651. doi:10.1007/s11121-014-0518-7. Perrino, T., Beardslee, W., Bernal, G., Brincks, A., Cruden, G., Howe, G., …, Brown, C. H. (2015). Toward scientific equity for the prevention of depression and depressive symptoms in vulnerable youth. Prevention Science, 16, 642–651. doi:10.​1007/​s11121-014-0518-7.
Zurück zum Zitat Raudenbush, S. W., & Bryk, A. S. (2002). Hierarchical linear models: applications and data analysis methods. Thousand Oaks: Sage Publications. Raudenbush, S. W., & Bryk, A. S. (2002). Hierarchical linear models: applications and data analysis methods. Thousand Oaks: Sage Publications.
Zurück zum Zitat Royall, R. M., & Cumberland, W. G. (1981). An empirical study of the ratio estimator and estimators of its variance. Journal of the American Statistical Association, 76, 66–77.CrossRef Royall, R. M., & Cumberland, W. G. (1981). An empirical study of the ratio estimator and estimators of its variance. Journal of the American Statistical Association, 76, 66–77.CrossRef
Zurück zum Zitat Rubin, D. B. (1976). Inference and missing data. Biometrika, 63, 581–592.CrossRef Rubin, D. B. (1976). Inference and missing data. Biometrika, 63, 581–592.CrossRef
Zurück zum Zitat Siddique, J., Brown, C. H., Hedeker, D., Duan, N., Gibbons, R. D., Miranda, J., & Lavori, P. W. (2008). Missing data in longitudinal trials—part B, analytic issues. Psychiatric Annals, 38, 793–801.CrossRefPubMedPubMedCentral Siddique, J., Brown, C. H., Hedeker, D., Duan, N., Gibbons, R. D., Miranda, J., & Lavori, P. W. (2008). Missing data in longitudinal trials—part B, analytic issues. Psychiatric Annals, 38, 793–801.CrossRefPubMedPubMedCentral
Zurück zum Zitat Siddique, J., Crespi, C. M., Gibbons, R. D., & Green, B. L. (2011). Using latent variable modeling and multiple imputation to calibrate rater bias in diagnosis assessment. Statistics in Medicine, 30, 160–174. doi:10.1002/sim.4109.CrossRefPubMed Siddique, J., Crespi, C. M., Gibbons, R. D., & Green, B. L. (2011). Using latent variable modeling and multiple imputation to calibrate rater bias in diagnosis assessment. Statistics in Medicine, 30, 160–174. doi:10.​1002/​sim.​4109.CrossRefPubMed
Zurück zum Zitat Siddique, J., Reiter, J. P., Brincks, A., Gibbons, R. D., Crespi, C. M., & Brown, C. H. (2015). Multiple imputation for harmonizing longitudinal non-commensurate measures in individual participant data meta-analysis. Statistics in Medicine, 34, 3399–3414. doi:10.1002/sim.6562.CrossRefPubMedPubMedCentral Siddique, J., Reiter, J. P., Brincks, A., Gibbons, R. D., Crespi, C. M., & Brown, C. H. (2015). Multiple imputation for harmonizing longitudinal non-commensurate measures in individual participant data meta-analysis. Statistics in Medicine, 34, 3399–3414. doi:10.​1002/​sim.​6562.CrossRefPubMedPubMedCentral
Zurück zum Zitat Siddique, J., de Chavez, P. J., Howe, G., Cruden, G., & Brown, C. H. (2016). Limitations in using multiple imputation to harmonize individual participant data for meta-analysis. Prevention Science, 1–14. Siddique, J., de Chavez, P. J., Howe, G., Cruden, G., & Brown, C. H. (2016). Limitations in using multiple imputation to harmonize individual participant data for meta-analysis. Prevention Science, 1–14.
Zurück zum Zitat Tisak, J., & Meredith, W. (1990). Descriptive and associative developmental models. In A. Von Eye (Ed.), Statistical methods in longitudinal research (Vol. II, pp. 387–406). San Diego: Academic Press. Tisak, J., & Meredith, W. (1990). Descriptive and associative developmental models. In A. Von Eye (Ed.), Statistical methods in longitudinal research (Vol. II, pp. 387–406). San Diego: Academic Press.
Zurück zum Zitat Wang, C.-P., Brown, C. H., & Bandeen-Roche, K. (2005). Residual diagnostics for growth mixture models: Examining the impact of a preventive intervention on multiple trajectories of aggressive behavior. Journal of the American Statistical Association, 100, 1054–1076. doi:10.1198/016214505000000501.CrossRef Wang, C.-P., Brown, C. H., & Bandeen-Roche, K. (2005). Residual diagnostics for growth mixture models: Examining the impact of a preventive intervention on multiple trajectories of aggressive behavior. Journal of the American Statistical Association, 100, 1054–1076. doi:10.​1198/​0162145050000005​01.CrossRef
Metadaten
Titel
Addressing Methodologic Challenges and Minimizing Threats to Validity in Synthesizing Findings from Individual-Level Data Across Longitudinal Randomized Trials
verfasst von
Ahnalee Brincks
Samantha Montag
George W. Howe
Shi Huang
Juned Siddique
Soyeon Ahn
Irwin N. Sandler
Hilda Pantin
C. Hendricks Brown
Publikationsdatum
22.04.2017
Verlag
Springer US
Erschienen in
Prevention Science / Ausgabe Sonderheft 1/2018
Print ISSN: 1389-4986
Elektronische ISSN: 1573-6695
DOI
https://doi.org/10.1007/s11121-017-0769-1

Weitere Artikel der Sonderheft 1/2018

Prevention Science 1/2018 Zur Ausgabe