nach oben

Journal of Gambling Studies

Erschienen in:

01.12.2012 | Original Paper

The Validation of Screening Tests: Meet the New Screen same as the Old Screen?

verfasst von: Blase Gambino

Erschienen in: Journal of Gambling Studies | Ausgabe 4/2012

Einloggen, um Zugang zu erhalten

Abstract

The focus of this report is to examine the process of validation of new screening tests designed to detect the problem gambler in research and practice settings. A hierarchical or phases of evaluation model is presented as a conceptual framework to describe the basic features of the validation process and its implications for application and interpretation of test results. The report describes a number of threats to validity in the form of sources of unintended bias that when unrecognized may lead to incorrect interpretations of study results and the drawing of incorrect conclusions about the usefulness of the new screening tests. Examples drawn from the gambling literature on problem gambling are used to illustrate some of the more important concepts including spectrum bias and clinical variation in test accuracy. The concept of zones of severity and the bias inherent in selecting criterion thresholds are reviewed. A definition of reference or study gold standard is provided. The use of 2-stage designs to establish validity by efficiently using reference standards to determine indices of accuracy and prevalence is recommended.

Nur mit Berechtigung zugänglich

The term reference standard rather than gold standard is now preferred since gold standard connotes an errorless process whereas in practice this is rarely if ever true. Further, for most disorders there is no gold standard and alternative standards that are error-prone must be utilized. The study reference standard is an operational definition of diagnostic truth for evaluating the new test.

The occurrence of errors in gold standards may be due to a number of sources, for example, the condition does not produce the expected biochemical changes, or an alternative condition mimics the biomarker changes of the condition of interest, a laboratory test may be mislabeled, a tumor is missed by MRI because it is too small, an image may be misinterpreted, a biopsy occurs outside of the affected tumor, the key is the term nearly infallible which reflects the rarity of errors (Reitsma et al. 2009). Some gold standards have error rates of nearly 5% or greater (Valenstein 1990).

Any false positives among cases or false negatives among controls are ignored since it is assumed the test results apply on average and not to any individual (Schlesselman 1982).

The use of a 2 × 2 table can lead to confusion since the role of prevalence (P) in calculating predictive values is not obvious. For example, from Table 1, PPV = TP/(TP + FP), however, TP = P × Se × N and FP = (1 − P) × (1 − Sp) × N, similar values apply to NPV.

The version of DSM-IV used by Stinchfield (2003) is not identical to the NODS version used by Volberg (2002, 2003) and different rules for inclusion and exclusion were used. Further the results reported by Stinchfield were based on one-year intervals whereas those reported by Volberg were based on lifetime measures. These distinctions are real but would take us beyond the scope of this paper and are ignored for simplicity of presentation. The use of SOGS as reference standard for NODS was based on availability of data. The results in Table 4 are presented here solely for illustrative purposes.

The measure of choice in Toce-Gerstein et al. (2009) was lifetime occurrences rather than one-year therefore the screen should not be interpreted in terms of screening for current problem gamblers.

A familiar example would be evaluating a screening test to detect anxiety disorder and questions are included that might ask have you ever felt anxious when …

Rectangular confidence intervals may be drawn for any point on the curve, there are other displays more commonly used (e.g., ellipses) for confidence intervals; the rectangular interval is much simpler to interpret.

Abramson, J. H. (1996). Cross-sectional studies. In W. W. Holland, R. Detals, & G. Knox (Eds.), Methods in public health (pp. 517–535). Oxford: Oxford University Press.

Aertgeerts, B., Buntix, F., Ansoms, S., & Fevery, A. (2001). Screening properties of questionnaires and laboratory tests for the detection of alcohol abuse or dependence in a general practice population. British Journal of General Practice, 51, 206–217.PubMed

Alberg, A. J., Park, J. W., Hager, B. W., Brock, M. V., & Diener-West, M. (2004). The use of “overall accuracy” to evaluate the validity of screening or diagnostic tests. Journal of General Internal Medicine, 19, 460–465.PubMedCrossRef

Alonzo, T. A., & Pepe, M. S. (1999). Using a combination of reference tests to assess the accuracy of a new diagnostic test. Statistics in Medicine, 18, 2987–3003.PubMedCrossRef

American Psychiatric Association. (1994). Diagnostic and statistical manual of mental disorders (Fourth ed.). Washington, DC: American Psychiatric Association.

Bachman, L. M., Juni, P., Reichenbach, S., et al. (2005). Consequences of different diagnostic’gold standards’ in test accuracy research: Carpal tunnel syndrome as an example. International Journal of Epidemiology, 34, 953–955.CrossRef

Bachmann, L. M., Puhan, M. A., ter Riet, G., & Bossuyt, P. M. (2006). Sample sizes of studies on diagnostic accuracy: Literature survey. British Medical Journal, 332, 1127–1129.PubMedCrossRef

Bellringer, M., Abbott, M. W., Volberg, R. A., Garrett, N., & Coombes, R. (2007). Problem gambling assessment and screening instruments: Phase one report. Auckland: Gambling Research Centre, Auckland University of Technology.

Bossuyt, P. M., Reitsma, J. B., Bruns, D. E., et al. (2003). Towards complete and accurate reporting of studies of diagnostic accuracy: The STARD initiative. BMJ, 326, 41–44.PubMedCrossRef

Boudreau, B., & Poulin, C. (2006). The South Oaks gambling screen-revised adolescent (SOGS-RA) revisited: A cut-point analysis. Journal of Gambling Studies, 23, 299–308.CrossRef

Chipman, M., Govoni, R., Jazmaji, V., Wilson, S., & Gao, P. (2008). High vs. low risk gambling: What is the difference? Reported prepared for the Ontario Problem Gambling Research Centre.

Currie, S. R., Hodgins, D. C., Wang, J., el-Guebaly, N., Wynne, H., & Chen, S. (2006). Risk of harm among gamblers in the general population as a function of level of participation in gambling activities. Addiction, 101, 570–580.PubMedCrossRef

Dickerson, M. (1993). A preliminary exploration of a two-stage methodology in the assessment of the extent and degree of gambling related problems in the Australian population. In W. R. Eadington, J. Cornelius, & J. I. Tabor (Eds.), Gambling behavior and problem gambling (pp. 347–363). Reno, Nevada: Institute for the Study of Gambling and Commercial Gaming, University of Nevada.

Eaton, W. W., Martins, S. S., Nestadt, G., Bienvenu, O. J., Clarke, D., & Alexandre, P. (2008). The burden of mental disorders. Epidemiologic Reviews, 30, 1–14.PubMedCrossRef

Ferris, J., & Wynne, H. (2001). The Canadian problem gambling index: Final report. Ottawa: Canadian Centre on Substance Abuse.

Gambino, B. (1997). The correction for bias in prevalence estimation with screening tests. Journal of Gambling Studies, 13, 343–351.PubMedCrossRef

Gambino, B. (1999). An epidemiologic note on verification bias: Implications for estimation of rates. Journal of Gambling Studies, 15, 223–232.PubMedCrossRef

Gambino, B. (2005a). Interpreting prevalence estimates of pathological gambling: Implications for policy. Journal of Gambling Issues, 14. Available at http://www.camh.net/egambling/issue14/jgi_14_gambino.html.

Gambino, B. (2005b). Going for the gold. The brief addiction science information source. Available at http://www.basisonline.org.

Gambino, B. (2006a). A comment on the utility of prevalence estimates of pathological gambling. Journal of Gambling Studies, 22, 321–328.PubMedCrossRef

Gambino, B. (2006b). Reflections on accuracy. Journal of Gambling Studies, 22, 393–404.PubMedCrossRef

Gambino, B. (2006c). Clarifying the at-risk label: A commentary. Journal of Gambling Issues, 16. Available at http://www.camh.net/egambling/issue16/issue16/jgi_16_gambino.html.

Gambino, B. (2009). Should gambling be included in public health surveillance systems? Journal of Gambling Issues, Issue 23, June. Available at http://www.camh.net/egambling/issue23pdfs/07b-gambino.pdf.

Gambino, B., & Lesieur, H. (2006). The South Oaks gambling screen (SOGS): A rebuttal to critics. Journal of Gambling Issues, 17. Available at http://www.camh.net/egambling/issue17/gambino.html.

Gebauer, L., LaBrie, R., & Shaffer, H. J. (2010). Optimizing DSM-IV-TR classification accuracy: A brief biosocial screen for detecting current gambling disorders among gamblers in the general household population. Canadian Journal of Psychiatry, 55, 82–90.

Gerstein, D., Hoffman, J., Larison, C., Murphy, S., Palmer, A., Chuchro, L., et al. (1999). Gambling impact and behavior study. Report to the national gambling impact study commission. Chicago: NORC.

Grove, R., McBride, O., & Slade, T. (2010). Toward DSM-V: Exploring diagnostic thresholds for alcohol dependence and abuse. Alcohol and Addiction, 45, 45–52.

Hawkins, R. C. (2005). The evidence based medicine approach to diagnostic testing: Practicalities and limitations. Clinical Biochemical Review, 26, 1–7.

Hawkins, D. M., Garrett, J. A., & Stephenson, B. (2001). Some issues in resolution of diagnostic tests using an imperfect gold standard. Statistics in Medicine, 20, 1987–2001.PubMedCrossRef

Hayen, A., Macaskill, P., Irwig, L., & Bossuyt, P. (2010). Appropriate statistical methods are required to assess diagnostic tests for replacement, add-on, and triage. Journal of Clinical Epidemiology, 63, 883–891.PubMedCrossRef

Hodgins, D. C., Stea, J. N., & Grant, J. E. (2011). Gambling disorders. Lancet, online first pp. 1–11.

Irwin, R. J., & Irwin, T. C. (2011). A principled approach to setting optimal diagnostic thresholds: Where ROC and indifference curves meet. European Journal of Internal Medicine, 22, 230–234.PubMedCrossRef

Johnson, E. E., Hamer, R., & Nora, R. M. (1998). The Lie-Bet questionnaire for screening pathological gamblers: A follow-up study. Psychological Reports, 83, 1219–1224.PubMed

Kessler, R. C., & Ustun, T. B. (2004). The World Mental Health (WMH) survey initiative version of the World Health Organization (WHO) composite international diagnostic interview (CIDI). International Journal of Methods in Psychiatric Research, 13, 93–121.PubMedCrossRef

Kleinbaum, D. G., Kupper, L. L., & Morgenstern, H. (1982). Epidemiologic research: Principles and quantitative methods. New York: Van Nostrand Reinhold.

Knottnerus, J. A., & van Weel, C. (2002). General introduction: evaluation of diagnostic procedures. In J. A. Knottnerus (Ed.), The evidence base of clinical diagnosis (pp. 1–17). London: BMJ Publishing Group.

Kraemer, H. C. (1992). Evaluating medical tests: Objective and quantitative guidelines. Newbury Park, California: Sage.

Kraemer, H. C. (2010). Epidemiological methods: About Time. International Journal of Environmental Research & Public Health, 7, 29–45.CrossRef

Kraemer, H. C., Noda, A., & O’Hara, R. (2004). Categorical versus dimensional approaches to diagnosis: Methodological challenges. Journal of Psychiatric Research, 38, 17–25.PubMedCrossRef

Kroenke, K., & Spitzer, R. L. (2002). The PHQ-9: A new depression diagnostic and severity measure. Psychiatric Annals, 32, 1–7.

Ladouceur, R., Bouchard, C., Rheaume, N., Jacques, C., Ferland, F., Leblond, J., et al. (2000). Is the SOGS an accurate measure of pathological gambling among children, adolescents and adults? Journal of Gambling Studies, 16, 1–24.PubMedCrossRef

LaPlante, D. A., Nelson, S. E., LaBrie, R. A., & Shaffer, H. J. (2008). Stability and progression of disordered gambling: Lessons from longitudinal studies. Canadian Journal of Psychiatry, 53, 52–60.

Lesieur, H. R., & Blume, S. B. (1987). South Oaks Gambling Screen (SOGS): A new instrument for the identification of pathological gamblers. American Journal of Psychiatry, 144, 1184–1188.PubMed

Lichtenstein, M., & Kiefe, C. (1990). Incorporating severity of illness into estimates of likelihood ratios. The American Journal of Medical Science, 299, 38–42.CrossRef

Lijmer, J. G., Leeflang, M., & Bossuyt, P. M. M. (2009). Proposals for a phased evaluation of medical tests. Medical Decision Making, 29, E13–E21.PubMedCrossRef

McNamee, R. C. (2003). Efficiency of two-phase designs for prevalence estimation. International Journal of Epidemiology, 32, 1072–1078.PubMedCrossRef

Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational measurement (3rd ed., pp. 13–103). Washington, DC: American Council on Education and National Council on Measurement in Education.

Mulherin, S. A., & Miller, W. C. (2002). Spectrum bias or spectrum effect? Annals of Internal Medicine, 137, 598–602.PubMed

Neal, P., Delfabbro, P., & O’Neil, M. (2005). Problem gambling and harm: Towards a national definition. Adelaide: South Australian Centre for Economic Studies.

Obuchowski, N. A. (2005). ROC analysis. American Journal of Roentgenology, 184, 364–372.PubMed

Obuchowski, N. A., & Zhou, X. (2002). Prospective studies of diagnostic test accuracy when disease prevalence is low. Biostatistics, 3, 477–492.PubMedCrossRef

Obuchowski, N. A., Graham, R. J., Baker, M. E., & Powell, K. A. (2001). Ten criteria for effective screening. American Journal of Roentgenology, 176, 1357–1362.PubMed

Pepe, M. S. (2003). The statistical evaluation of medical tests for classification and prediction. Oxford UK: Oxford University Press.

Perkins, N. J., & Schisterman, E. F. (2006). The inconsistency of “optimal” cut points obtained using two criteria based on the receiver operating characteristic curve. American Journal of Epidemiology, 163, 670–675.PubMedCrossRef

Petry, N. M. (2006). Should the scope of addictive behaviors be broadened to include pathological gambling? Addiction, 101(Supplement 1), 152–160.PubMedCrossRef

Petry, N. (2011). Non-substance related addictions: Their place in DSM-V (abstract). Paper to be presented at American Public Health Association Meetings, October 31, 2011.

Pfeiffer, R. M., & Castle, P. E. (2005). With or without a gold standard. Epidemiology, 16, 595–597.PubMedCrossRef

Ransohoff, D. J., & Feinstein, A. R. (1978). Problems of spectrum and bias in evaluating the efficacy of diagnostic tests. New England Journal of Medicine, 299, 926–930.PubMedCrossRef

Reid, M. C., Lachs, M. S., & Feinstein, A. R. (1995). Use of methodological standards in diagnostic test research: Getting better but still not good. Journal of American Medical Association, 274, 645–651.CrossRef

Reitsma, J. B., Rutjes, A. W. S., Khan, K. S., Coomarasamy, A., & Bossuyt, P. M. (2009). A review of solutions for diagnostic accuracy studies with an imperfect or missing reference standard. Journal of Clinical Epidemiology, 62, 797–806.PubMedCrossRef

Rosenthal, R. J. (2003). Distribution of the DSM-IV criteria for pathological gambling. Addiction, 98, 1674–1675.PubMedCrossRef

Rutjes, A. W. S., Reitsma, J. B., DiNisio, M., et al. (2006). Evidence of bias and variation in diagnostic accuracy studies. CMAJ, 174, 469–476.PubMed

Rutjes, A. W., Reitsma, J. B., Coomarasamy, A., Khan, K. S., & Bossuyt, P. M. (2007). Evaluation of diagnostic tests when there is no gold standard. Health, Technological Assessment, 11, 50.

Sackett, D. L., & Haynes, R. B. (2002). Evidence base of clinical diagnosis: The architecture of diagnostic research. BMJ, 324, 539–541.PubMedCrossRef

Sackett, D. L., Haynes, R. B., & Tugwell, P. (1985). Clinical epidemiology: A basic science for clinical medicine. Boston: Little Brown.

Schatzkin, A., Connor, R. J., Taylor, P. R., & Bunnag, B. (1987). Comparing new and old screening tests when a reference procedure cannot be performed on all screenees: Example of automated cytometry for early detection of cervical cancer. American Journal of Epidemiology, 125, 672–678.PubMed

Schlesselman, J. J. (1982). Case-control studies. New York: Oxford University Press.

Shaffer, H. J., & Gambino, B. (1990). Epilogue: Integrating treatment choices. In H. B. Milkman & L. I. Sederer (Eds.), Treatment choices for alcoholism and substance abuse (pp. 351–375). Lexington, MA: Lexington Books.

Shaffer, H. J., Hall, M. N., & Vander Bilt, J. (1997). Estimating the prevalence of disordered gambling behavior in the United States and Canada: A meta-analysis Boston. MA: Harvard Medical School Division on Addictions.

Sica, G. T. (2006). Bias in research studies. Radiology, 238, 780–789.PubMedCrossRef

Sox, H. C., Jr., Blatt, M. A., Higgins, M. C., & Marton, K. I. (1988). Medical decision making. Boston: Butterworths-Heinemann.

Statistics Canada (2009). Table 3. Stratum-specific likelihood ratios for selected health status characteristics. Validation of disability categories derived from Health utilities index mark 3 scores. Health reports, 20, no. 2. Available at http://www.statcan.gc.ca.

Stinchfield, R. (2002). Reliability, validity, classification accuracy of the South Oaks Gambling Screen (SOGS). Addictive Behaviors, 27, 1–19.PubMedCrossRef

Stinchfield, R. (2003). Reliability, validity, and classification accuracy of a measure of DSM-IV diagnostic criteria for pathological gambling. American Journal of Psychiatry, 160, 180–182.PubMedCrossRef

Stinchfield, R., Govoni, R., & Frisch, G. R. (2007a). A review of screening and assessment instruments for problem and pathological gambling. In G. Smith, D. Hodgins, & R. J. Williams (Eds.), Research and measurement issues in gambling studies (pp. 179–213). New York: Academic Press.

Stinchfield, R., Winters, K. C., Botzet, A., Jerstad, S., & Breyer, J. (2007b). Development and psychometric evaluation of the gambling treatment outcome monitoring system (GAMTOMS). Psychology of Addictive Behaviors, 21, 174–184.PubMedCrossRef

Stucki, S., & Rihs-Middel, M. (2007). Prevalence of adult problem and pathological gambling between 2000 and 2005: An update. Journal of Gambling Studies, 23, 245–257.PubMedCrossRef

Thomas, S., Jackson, A., & Blaszczynski, A. (2003). Measuring problem gambling: Evaluation of the Victorian gambling screen. Melbourne: Gambling Research Panel.

Toce-Gerstein, M., & Gerstein, D. R. (2004). Of time and the chase: Lifetime versus past-year measures of pathological gambling. E Journal of Gambling Issues, 10. Available at: http://www.camh.net/egambling/issue10/ejgi_10_gerstein.html.

Toce-Gerstein, M., Gerstein, D. R., & Volberg, R. A. (2009). The NODS-CLIP: A rapid screen for adult pathological and problem gambling. Journal of Gambling Studies, 25, 541–555.PubMedCrossRef

Trikalinos, T. A., Siebert, U., & Lau, J. (2009). Decision-analytic modeling to evaluate benefits and harms of medical tests: Uses and limitations. Medical Decision Making, 29, E22–E29.PubMedCrossRef

Valenstein, P. N. (1990). Evaluating diagnostic tests with imperfect standards. American Journal of Clinical Pathology, 93, 252–258.PubMed

Van den Bruel, A., Aertgeerts, B., & Buntix, F. (2006). Results of diagnostic accuracy studies are not always validated. Journal of Clinical Epidemiology, 59, 559–566.PubMed

Van Holst, R. J., van den Brink, W., & Veltman, D. J. (2010). Brain imaging studies in pathological gambling. Current Psychiatry Reports, 12, 418–425.PubMedCrossRef

Volberg, R. A. (2002). Gambling and problem gambling in Nevada. Report to the Nevada Department of Human Resources.

Volberg, R. A. (2003). Gambling and problem gambling in Arizona. Report to the Arizona Lottery.

Volberg, R. A., & Bernhard, B. (2006). The 2006 study of gambling and problem gambling in New Mexico. Report to the responsible gaming association of New Mexico.

Volberg, R. A., & Young, M. M. (2008). Using SOGS vs. CPGI in problem gambling screening and assessment. Toronto: Ontario Problem Gambling Research Centre.

Wenzel, M., McMillen, J., Marshall, D., & Ahmed, E. (2004). Validation of the Victorian gambling screen. Melbourne: Australian National University.

Whiting, P., Rutjes, A. W., Reitsma, J. B., Bossuyt, P. M., & Kleijnen, J. (2003). The development of QUADAS: A tool for the quality assessment of studies of diagnostic accuracy included in systematic reviews. BMC Medical Research Methodology, 3, 25.PubMedCrossRef

Whiting, P., Rutjes, A. W., Reitsma, J. B., Glas, A. S., Bossuyt, P. M., & Kleijnen, J. (2005). Sources of variation and bias in studies of diagnostic accuracy. Annals of Internal Medicine, 140, 189–202.

Wiebe, J., Maitland, S. B., Hodgins, D., Davey, A., & Gottlieb, B. (2009). Transition and stability of problem gambling behaviours. Final report to the addiction foundation of Manitoba.

Worster, A., & Carpenter, C. (2008). Incorporation bias in studies of diagnostic tests: How to avoid being biased about bias. CJEM, 10, 174–175.PubMed

Youden, J. (1950). Index for rating diagnostic tests. Cancer, 3, 32–35.PubMedCrossRef

Zhou, X., Obuchowski, N. A., & McClish, D. K. (2002). Statistical methods in diagnostic medicine. New York: Wiley and Sons.CrossRef

Titel: The Validation of Screening Tests: Meet the New Screen same as the Old Screen?
verfasst von: Blase Gambino
Publikationsdatum: 01.12.2012
Verlag: Springer US
Erschienen in: Journal of Gambling Studies / Ausgabe 4/2012
Elektronische ISSN: 1573-3602
DOI: https://doi.org/10.1007/s10899-011-9285-6

Neu im Fachgebiet Psychiatrie

Mehr Zeit in den sozialen Medien kann zu Angststörungen führen

05.06.2024 Prävention und Gesundheitsförderung in der Pädiatrie Nachrichten

Eine Studie aus Finnland hat die Social-Media-Nutzung unter 16-jährigen Mädchen genauer unter die Lupe genommen. Lange Nutzdauer von Instagram & Co. war mit höheren Raten an Angststörungen assoziiert. Insgesamt waren rund 17% der Mädchen abhängig von Social Media.

Springer Medizin

The Validation of Screening Tests: Meet the New Screen same as the Old Screen?

Abstract

Neu im Fachgebiet Psychiatrie

Mehr Zeit in den sozialen Medien kann zu Angststörungen führen

Intoxikation ohne Alkoholaufnahme: An das Eigenbrauer-Syndrom denken!

Placebo kann psychische Erkrankungen lindern

Prämenstruelle Beschwerden mit Suizidrisiko assoziiert

Springer Medizin

Abstract

Bitte loggen Sie sich ein, um Zugang zu diesem Inhalt zu erhalten

Weitere Artikel der Ausgabe 4/2012

Sympathetic Magic and Gambling: Adherence to the Law of Contagion Varies with Gambling Severity

Validation of the Chinese Version of the Gamblers’ Belief Questionnaire (GBQ-C)

Neural Correlates of Pathological Gamblers Preference for Immediate Rewards During the Iowa Gambling Task: An fMRI Study

Low Trait Self-Control in Problem Gamblers: Evidence from Self-Report and Behavioral Measures

The Effect of Recessions on Gambling Expenditures

Arousal and Affective Differences Between Student Gamblers and Non-gamblers during a Card Game

Neu im Fachgebiet Psychiatrie

Mehr Zeit in den sozialen Medien kann zu Angststörungen führen

Intoxikation ohne Alkoholaufnahme: An das Eigenbrauer-Syndrom denken!

Placebo kann psychische Erkrankungen lindern

Prämenstruelle Beschwerden mit Suizidrisiko assoziiert