Skip to main content

Advertisement

SpringerLink
Log in

Automatic nonverbal behavior indicators of depression and PTSD: the effect of gender

  • Original Paper
  • Published:
Journal on Multimodal User Interfaces Aims and scope Submit manuscript

Abstract

Recently there has been arising interest in automatically recognizing nonverbal behaviors that are linked with psychological conditions. Work in this direction has shown great potential for cases such as depression and post-traumatic stress disorder (PTSD), however most of the times gender differences have not been explored. In this paper, we show that gender plays an important role in the automatic assessment of psychological conditions such as depression and PTSD. We identify a directly interpretable and intuitive set of predictive indicators, selected from three general categories of nonverbal behaviors: affect, expression variability and motor variability. For the analysis, we employ a semi-structured virtual human interview dataset which includes 53 video recorded interactions. Our experiments on automatic classification of psychological conditions show that a gender-dependent approach significantly improves the performance over a gender agnostic one.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

Notes

  1. Sample interaction between the virtual agent and a human actor can be seen here: http://www.youtube.com/watch?v=ejczMs6b1Q4.

  2. http://www.mathworks.com/products/statistics.

References

  1. Baltrusaitis T, Robinson P, Morency L (2012) 3d constrained local model for rigid and non-rigid facial tracking. In: CVPR, 2012 IEEE Conference

  2. Littlewort G, Whitehill J, Wu T, Fasel I, Frank M, Movellan J, Bartlett M (2011) The computer expression recognition toolbox (cert). In: 2011 IEEE International Conference on FG 2011

  3. Ekman P, Matsumoto D, Friesen WV (1997) Facial expression in affective disorders. What the face reveals: Basic and applied studies of spontaneous expression using the Facial Action Coding System (FACS) 2:331–342

  4. Ellgring H (1989) Nonverbal communication in depression. Cambridge University Press, Cambridge

    Google Scholar 

  5. Cohn JF, Kruez TS, Matthews I, Yang Y, Nguyen MH, Padilla MT, Zhou F, De la Torre F (2009) Detecting depression from facial actions and vocal prosody. In: 3rd international conference on ACII 2009, pp 1–7

  6. McIntyre G, Gocke R, Hyett M, Green M, Breakspear M (2009) An approach for automatically measuring facial activity in depressed subjects. In: 3rd international conference on ACII 2009

  7. Maddage MC, Senaratne R, Low L-SA, Lech M, Allen N (2009) Video-based detection of the clinical depression in adolescents. In: EMBC 2009. Annual International Conference of the IEEE, pp 3723–3726

  8. Diagnostic and Statistical Manual of Mental Disorders DSM-IV-TR Fourth Edition (Text Revision). Amer Psychiatric Pub, 4th edition, July 2000

  9. Diagnostic and Statistical Manual of Mental Disorders DSM-V Fifth Edition. American Psychiatric Association, 5th edition, June 2013

  10. Russell JA, Barrett LF (1999) Core affect, prototypical emotional episodes, and other things called emotion: dissecting the elephant. J Personal Soc Psychol 76(5):805–819

    Article  Google Scholar 

  11. Ian Reed L, Sayette MA, Cohn JF (2007) Impact of depression on response to comedy: a dynamic facial coding analysis. J Abnorm Psychol 116:804–809

    Article  Google Scholar 

  12. Kirsch A, Brunnhuber S (2007) Facial expression and experience of emotions in psychodynamic interviews with patients with ptsd in comparison to healthy subjects. Psychopathology 40(5):296–302

    Article  Google Scholar 

  13. Troisi A, Moles A (1999) Gender differences in depression: an ethological study of nonverbal behavior during interviews. J Psychiat Res 33(3):243–250

    Article  Google Scholar 

  14. Wolfsdorf BA, Mattia JI, Zlotnick C, Zimmerman M (2001) Gender differences in patients with posttraumatic stress disorder in a general psychiatric practice. Am J Psychiatry 158(11):1923–1925

    Article  Google Scholar 

  15. Draijer N, Gersons BP, Olff M, Langeland W (2007) Gender differences in posttraumatic stress disorder. Psychol Bull 133(2):183–204

    Article  Google Scholar 

  16. Green B (2003) Post-traumatic stress disorder: symptom profiles in men and women. Curr Med Res Opin 19(3):200–204

    Article  Google Scholar 

  17. Cummins N, Joshi J, Dhall A, Sethu V, Goecke R, Epps J (2013) Diagnosis of depression by behavioural signals: a multimodal approach. In: Proceedings of the 3rd ACM international workshop on audio/visual emotion challenge, AVEC ’13, pp 11–20, New York, NY, USA, 2013. ACM

  18. Meng H, Huang D, Wang H, Yang H, AI-Shuraifi M, Wang Y (2013) Depression recognition based on dynamic facial and vocal expression features using partial least square regression. In: Proceedings of the 3rd ACM International Workshop on Audio/Visual Emotion Challenge, AVEC ’13, pp 21–30, New York, NY, USA, 2013. ACM

  19. Girard JM, Cohn JF, Mahoor MH, Mavadati SM, Hammal Z, Rosenwald DP Nonverbal social withdrawal in depression: evidence from manual and automatic analyses. Image and Vision Computing

  20. D’Mello S, Graesser A (2012) Dynamics of affective states during complex learning. Learn Instr 22(2):145–157

    Article  Google Scholar 

  21. Joshi J, Dhall A, Goecke R, Cohn JF (2013) Relative body parts movement for automatic depression analysis. In: Proceedings of the 2013 Humaine Association Conference on Affective Computing and Intelligent Interaction, ACII ’13, pp 492–497, Washington, DC, USA, 2013. IEEE Computer Society

  22. Scherer S, Stratou G, Mahmoud M, Boberg J, Gratch J, Rizzo A, Morency L-P (2013) Automatic behavior descriptors for psychological disorder analysis. In: IEEE conference on automatic face and gesture recognition, 2013

  23. Blanchard EB, Jones-Alexander J, Buckley TC, Forneris CA (1996) Psychometric properties of the ptsd checklist (pcl). Behav Res Therapy 34(8):669–673

    Article  Google Scholar 

  24. Kroenke K, Spitzer RL (2002) The phq-9: a new depression and diagnostic severity measure. Psychiatr Ann 32:509–521

    Article  Google Scholar 

  25. Kroenke K, Spitzer RL, Williams JBW (2001) The phq-9. J Gen Intern Med 16(9):606–613

    Article  Google Scholar 

  26. Bolton EE, Gray MJ, Litz BT (2006) A cross-lagged analysis of the relationship between symptoms of ptsd and retrospective reports of exposure. J Anxiety Disord 20(7):877–895

    Article  Google Scholar 

  27. Hoge CW, Castro CA, Messer SC, McGurk D, Cotting DI, Koffman RL (2004) Combat duty in iraq and afghanistan, mental health problems, and barriers to care. N Engl J Med 351(1):13–22

    Article  Google Scholar 

  28. Andrykowski MA, Cordova MJ, Studts JL, Miller TW (1998) Posttraumatic stress disorder after treatment for breast cancer: prevalence of diagnosis and use of the ptsd checklistcivilian version (pclc) as a screening instrument. J Consult Clin Psychol 66(3):586–590

    Article  Google Scholar 

  29. Stefan S, Giota S, Jonathan G, Morency L-P (2013) Investigating voice quality as a speaker-independent indicator of depression and ptsd. In: Proceedings of interspeech

  30. DeVault D, Georgila K, Artstein R, Morbini F, Traum D, Scherer S, Rizzo A, Morency L-P (2013) Verbal indicators of psychological distress in interactive dialogue with a virtual human. In SIGDIAL, Metz, France, August

    Google Scholar 

  31. Ekman P, Rosenberg EL (2005) What the face reveals: basic and applied studies of spontaneous expression using the facial action coding system (FACS) (series in affective Science). Oxford UP, USA, New York

  32. Bonanno GA, Keltner D (1997) Facial expressions of emotion and the course of conjugal bereavement. J Abnorm Psychol 106:126–137

    Article  Google Scholar 

  33. Morency L, Whitehill J, Movellan J (2008) Generalized adaptive view-based appearance model: Integrated framework for monocular head pose estimation. In: 8th IEEE International Conference on FG ’08

  34. Kublbeck C, Ernst A (2006) Face detection and tracking in video sequences using the modifiedcensus transformation. Image Vis Comput 24(6):564–572

    Article  Google Scholar 

  35. Hedges LV (1981) Distribution theory for glass’s estimator of effect size and related estimators. J Educ Behav Stat 6(2):107–128

    Article  Google Scholar 

  36. Deutsch FM, LeBaron D, Fryer MM (1987) What is in a smile? Psychol Women Q 11(3):341–352

    Article  Google Scholar 

  37. Hess U, Adams RB, Kleck RE Jr (2004) Facial appearance, gender, and emotion expression. Emotion 4(4):378

    Article  Google Scholar 

  38. Adams R Jr, Kleck R, Hess U (2005) Who may frown and who should smile? dominance, affiliation, and the display of happiness and anger

  39. Krämer NC, Hoffmann L, Kopp S (2010) Know your users! empirical results for tailoring an agent’s nonverbal behavior to different user groups. In: Proceedings of IVA, IVA’10, pp 468–474

  40. Kulms P, Kramer NC, Gratch J, Kang S-H (2011) It’s in their eyes: a study on female and male virtual humans gaze. In: IVA, volume 6895 of, Lecture Notes in Computer Science, pp 80–92

  41. Rief W, Nanke A, Klaiberg A, Braehler E (2004) Base rates for panic and depression according to the brief patient health questionnaire: a population-based study. J Affect Disord 82(2):271–276

  42. Campbell DG, Felker BL, Liu C-F, Yano EM, Kirchner JE, Chan D, Rubenstein LV, Chaney EF (2007) Prevalence of depressionptsd comorbidity: implications for clinical practice guidelines and primary care-based interventions. J Gen Intern Med 22(6):711–718

    Article  Google Scholar 

Download references

Acknowledgments

This work is supported by DARPA under contract (W911NF-04-D-0005) and U.S. Army Research, Development, and Engineering Command. The content does not necessarily reflect the position or the policy of the Government, and no official endorsement should be inferred.

Author information

Authors and Affiliations

  1. Institute for Creative Technologies, University of Southern California, Los Angeles, California

    Giota Stratou, Stefan Scherer, Jonathan Gratch & Louis-Philippe Morency

Authors
  1. Giota Stratou
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Stefan Scherer
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jonathan Gratch
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Louis-Philippe Morency
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Giota Stratou.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Stratou, G., Scherer, S., Gratch, J. et al. Automatic nonverbal behavior indicators of depression and PTSD: the effect of gender. J Multimodal User Interfaces 9, 17–29 (2015). https://doi.org/10.1007/s12193-014-0161-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12193-014-0161-4

Keywords

Search

Navigation