Skip to main content
SpringerLink
Log in

Preliminary Factor Analyses Raise Concerns about Script Concordance Test Utility

Exploring SCT constructs

  • Original Research
  • Published:
Medical Science Educator Aims and scope Submit manuscript

Abstract

Purpose

Script concordance tests (SCTs) are assessments purported to measure one’s ability to interpret clinical data, a key component of the clinical diagnostic reasoning process. The aim of this research was to directly examine the construct validity/dimensionality of SCTs using factor analysis.

Method

At Indiana University School of Medicine, a confirmatory factor analysis was conducted on three SCT datasets to assess whether SCT scores represented a single dimension of clinical reasoning. Exploratory factor analyses were also performed to determine if the SCTs represented multiple dimensions and to examine the effects of various SCT scoring methods on test dimensionality.

Results

Results of the confirmatory factor analysis revealed that the SCTs did not have a unidimensional factor structure. Subsequent exploratory factor analyses of six different scoring methods confirmed these results. Factor loadings were weak and the factors explained only a small portion of the total variance in SCT scores.

Conclusions

The results of this study challenge the assertion that SCTs measure one dimension of clinical reasoning. Because the outcomes of this study raise questions about the latent constructs SCTs measure, recommendations concerning the utility of SCT scores are also presented.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Sibert L, Charlin B, Corcos J, Gagnon R, Lechevallier J, Grise P (2002) Assessment of clinical reasoning competence in urology with the Script Concordance Test: an exploratory study across two sites from different countries. Eur Urol 41(3):227–233

    Article  Google Scholar 

  2. Charlin B, Brailovsky C, Leduc C, Blouin D (1998) The diagnosis script questionnaire: a new tool to assess a specific dimension of clinical competence. Adv Health Sci Educ 3(1):51–58

    Article  Google Scholar 

  3. Lubarsky S, Charlin B, Cook DA, Chalk C, Van Der Vleuten CPM (2011) Script concordance testing: a review of published validity evidence. Med Educ 45(4):329–338

    Article  Google Scholar 

  4. Handfield-Jones R, Brown JB, Biehn J, Rainsberry P, Brailovsky CA (1996) Certification examination of the College of Family Physicians of Canada. Part 3: Short-answer management problems. Can Fam Physician 42(Handfield-Jones R, Brown JB, Biehn J, Rainsberry P, Brailovsky CA):1353

    Google Scholar 

  5. Williams RG, Klamen DL, Hoffman RM (2008) Medical student acquisition of clinical working knowledge. Teach Learn Med 20(1):5

    Article  Google Scholar 

  6. Wainer H, Mee J (2004) On assessing the quality of physicians’ clinical judgment. Eval Health Prof 27(4):369–382

    Article  Google Scholar 

  7. Khonputsa P, Besinque K, Fisher D, Gong WC (2006) Use of script concordance test to assess pharmaceutical diabetic care: a pilot study in Thailand. Med Teach 28(6):570–573

    Article  Google Scholar 

  8. Lubarsky S, Chalk C, Kazitani D, Gagnon R, Charlin B (2009) The Script Concordance Test: a new tool assessing clinical judgement in neurology. Can J Neurol Sci 36(3):326–331

    Google Scholar 

  9. Carrière B, Gagnon R, Charlin B, Downing S, Bordage G (2008) Assessing clinical reasoning in pediatric emergency medicine: validity evidence for a Script Concordance Test. Ann Emerg Med 53(5):647–652

    Article  Google Scholar 

  10. Lambert C, Gagnon R, Nguyen D, Charlin B. (2009) The script concordance test in radiation oncology: validation study of a new tool to assess clinical reasoning. Radiat Oncol 4 (7)

  11. Humbert AJ, Besinger B, Miech EJ (2011) Assessing clinical reasoning skills in scenarios of uncertainty: convergent validity for a script concordance test in an emergency medicine clerkship and residency. Acad Emerg Med 18(6):627–634

    Article  Google Scholar 

  12. Dory V, Gagnon R, Vanpee D, Charlin B (2012) How to construct and implement script concordance tests: insights from a systematic review. Med Educ 46(6):552–563

    Google Scholar 

  13. Cronbach LJ (1971) Test validation. In: Thorndike RL (ed) Educational measurement, 2nd edn. American Council on Education, Washington, DC, pp 443–507

    Google Scholar 

  14. Messick S (1989) Validity. In: Linn RL (ed) Educational measurement, 3rd edn. Macmillan, New York, pp 13–103

    Google Scholar 

  15. Humbert AJ, Johnson MT, Miech E, Friedberg F, Grackin JA, Seidman PA (2011) Assessment of clinical reasoning: a script concordance test designed for pre-clinical medical students. Med Teach 33(6):472–477

    Article  Google Scholar 

  16. Humbert A (2008) Assessing the clinical reasoning skills of emergency medicine clerkship students using a script concordance test. Acad Emerg Med 15:S230–S231

    Article  Google Scholar 

  17. Fournier J, Demeester A, Charlin B (2008) Script concordance tests: guidelines for construction. BMC Med Inform Decis Mak 8(1):18

    Article  Google Scholar 

  18. Bollen KA (1989) Structural equations with latent variables. Wiley, New York

    Google Scholar 

  19. Joreskog KG. Structural equation modeling with ordinal variables using LISREL. (2005) http://www.ssicentral.com/lisrel/techdocs/ordinal.pdf. Accessed 3 Aug 2012.

  20. Brown TA (2006) Confirmatory factor analysis for applied research (methodology in the social sciences)

  21. Harrington D (2009) Confirmatory factor analysis. Oxford University Press, Oxford

    Google Scholar 

  22. Hu L, Bentler PM (1999) Cutoff criteria for fit indexes in covariance structure analysis: conventional criteria versus new alternatives. Struct Equ Model: A Multidiscip J 6(1):1–55

    Article  Google Scholar 

  23. Cole DA (1987) Utility of confirmatory factor analysis in test validation research. J Consult Clin Psychol 55(4):584

    Article  Google Scholar 

  24. Brown TA (2006) Confirmatory factor analysis for applied research. The Guilford Press, New York

    Google Scholar 

  25. Bernstein IH, Teng G (1989) Factoring items and factoring scales are different: spurious evidence for multidimensionality due to item categorization. Psychol Bull 105(3):467

    Article  Google Scholar 

  26. Meterissian S, Zabolotny B, Gagnon R, Charlin B (2007) Is the script concordance test a valid instrument for assessment of intraoperative decision-making skills? Am J Surg 193(2):248–251

    Article  Google Scholar 

  27. Charlin B, Desaulniers M, Gagnon R, Blouin D, van der Vleuten C (2002) Comparison of an aggregate scoring method with a consensus scoring method in a measure of clinical reasoning capacity. Teach Learn Med 14(3):150–156

    Article  Google Scholar 

  28. Bland AC, Kreiter CD, Gordon JA (2005) The psychometric properties of five scoring methods applied to the script concordance test. Acad Med 80(4):395

    Article  Google Scholar 

  29. Cronbach LJ, Meehl PE. (1955) Construct validity in psychological tests. Psychol Bull 52 (4)

  30. Collard A, Gelaes S, Vanbelle S et al (2009) Reasoning versus knowledge retention and ascertainment throughout a problem‐based learning curriculum. Med Educ 43(9):854–865

    Article  Google Scholar 

  31. Charlin B, Brailovsky C, Brazeau-Lamontagne L, Samson L, Leduc C, Van der Vleuten C (1998) Script questionnaires: their use for assessment of diagnostic knowledge in radiology. Med Teach 20(6):567–571

    Article  Google Scholar 

  32. Brailovsky C, Charlin B, Beausoleil S, Cote S, Van der Vleuten C (2001) Measurement of clinical reflective capacity early in training as a predictor of clinical reasoning performance at the end of residency: an experimental study on the script concordance test. Med Educ 35(5):430–436

    Article  Google Scholar 

  33. Fabrigar LR, Wegener DT, MacCallum RC, Strahan EJ (1999) Evaluating the use of exploratory factor analysis in psychological research. Psychol Methods 4(3):272

    Article  Google Scholar 

  34. Kreiter CD, Bergus GR (2007) Case specificity: empirical phenomenon or measurement artifact?

  35. Norman G, Bordage G, Page G, Keane D (2006) How specific is case specificity? Med Educ 40(7):618–623

    Article  Google Scholar 

  36. Rindskopf D, Rose T (1988) Some theory and applications of confirmatory second-order factor analysis. Multivar Behav Res 23(1):51–67

    Article  Google Scholar 

  37. Wainer H, Kiely GL (1987) Item clusters and computerized adaptive testing: a case for testlets. J Educ Meas 24(3):185–201

    Article  Google Scholar 

Download references

Acknowledgments

The authors would like to thank Drs. James Brokaw and Mark Seifert for their input and critique of early drafts of this manuscript.

Funding/Support

None

Other disclosures

None

Ethical approval

Indiana University-Purdue University Indianapolis institutional review board approved this study.

Disclaimers

None

Previous presentations

None

Author information

Authors and Affiliations

  1. Department of Surgery, Indiana University, 545 Barnhill Drive, Emerson Hall 543, Indianapolis, IN, 46202-5120, USA

    Adam B. Wilson

  2. Office of Information Management and Institutional Research, Indiana University–Purdue University Indianapolis, Indianapolis, IN, USA

    Gary R. Pike

  3. Department of Emergency Medicine, Office of Undergraduate Medical Education, Indiana University, Indianapolis, IN, USA

    Aloysius J. Humbert

Authors
  1. Adam B. Wilson
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Gary R. Pike
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Aloysius J. Humbert
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Adam B. Wilson.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Wilson, A.B., Pike, G.R. & Humbert, A.J. Preliminary Factor Analyses Raise Concerns about Script Concordance Test Utility. Med.Sci.Educ. 24, 51–58 (2014). https://doi.org/10.1007/s40670-014-0013-6

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s40670-014-0013-6

Keywords

Search

Navigation