Skip to main content
SpringerLink
Log in

A mandala of faculty development: using theory-based evaluation to explore contexts, mechanisms and outcomes

  • Published:
Advances in Health Sciences Education Aims and scope Submit manuscript

Abstract

Demonstrating the impact of faculty development, is an increasingly mandated and ever elusive goal. Questions have been raised about the adequacy of current approaches. Here, we integrate realist and theory-driven evaluation approaches, to evaluate an intensive longitudinal program. Our aim is to elucidate how faculty development can work to support a range of outcomes among individuals and sub-systems in the academic health sciences. We conducted retrospective framework analysis of qualitative focus group data gathered from 79 program participants (5 cohorts) over a 10-year period. Additionally, we conducted follow-up interviews with 15 alumni. We represent the interactive relationships among contexts, mechanisms, and outcomes as a “mandala” of faculty development. The mandala illustrates the relationship between the immediate program context, and the broader institutional context of academic health sciences, and identifies relevant change mechanisms. Four primary mechanisms were collaborative-reflection, self-reflection and self-regulation, relationship building, and pedagogical knowledge acquisition. Individual outcomes, including changed teaching practices, are described. Perhaps most interestingly, secondary mechanisms—psychological and structural empowerment—contributed to institutional outcomes through participants’ engagement in change leadership in their local contexts. Our theoretically informed evaluation approach models how faculty development, situated in appropriate institutional contexts, can trigger mechanisms that yield a range of benefits for faculty and their institutions. The adopted methods hold potential as a way to demonstrate the often difficult-to-measure outcomes of educational programs, and allow for critical examination as to how and whether faculty development programs can accomplish their espoused goals.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

References

  • Abhyyankar, P., Cheyne, H., Maxwell, M., Harris, F., & McCourt, C. (2013). A realist evaluation of a normal birth programme. Evidence Based Midwifery, 11(4), 112–119.

    Google Scholar 

  • Ashford, S. (2003). Reflections on the looking glass: A review of research on feedback-seeking behavior in organizations. Journal of Management, 29(6), 773–799.

    Article  Google Scholar 

  • Astbury, B. (2013). Some reflections on Pawson’s science of evaluation: A realist manifesto. Evaluation, 19(4), 383–401.

    Article  Google Scholar 

  • Astbury, B., & Leeuw, F. L. (2010). Unpacking black boxes: Mechanisms and theory building in evaluation. American Journal of Evaluation, 31(3), 363–381.

    Article  Google Scholar 

  • Baldwin, C. K., Hutchinson, S. L., & Magnuson, D. R. (2004). Program theory: A framework for theory-driven programming and evaluation. Therapeutic Recreation Journal, 38(1), 16–31.

    Google Scholar 

  • Blamey, A., & Mackenzie, M. (2007). Theories of change and realistic evaluation: Peas in a pod or apples and oranges? Evaluation, 13(4), 439–455.

    Article  Google Scholar 

  • Boud, D. (2001). Using journal writing to enhance reflective practice. New Directions for Adult and Continuing Education, 2001(90), 9–18.

    Article  Google Scholar 

  • Box, G., & Draper, N. (1987). Empirical model-building and response surfaces. New York, NY: Wiley.

    Google Scholar 

  • Butts, M. M., Vandenberg, R. J., DeJoy, D. M., Schaffer, B. S., & Wilson, M. G. (2009). Individual reactions to high involvement work processes: Investigating the role of empowerment and perceived organizational support. Journal of Occupational Health Psychology, 14(2), 122–136.

    Article  Google Scholar 

  • Centra, J. A. (1978). Types of faculty development programs. The Journal of Higher Education, 49(2), 151–162.

    Article  Google Scholar 

  • Chen, H., & Rossi, P. H. (1983). Evaluating with sense: The theory driven approach. Evaluation Review, 7(3), 283–302.

    Article  Google Scholar 

  • Chen, H., & Rossi, P. H. (1989). Issues in the theory-driven perspective. Evaluation and Program Planning, 12(4), 299–306.

    Article  Google Scholar 

  • Chen, H., & Rossi, P. H. (1992). Using theory to improve program and policy evaluations. Santa Barbara, CA: Greenwood Press.

    Google Scholar 

  • Davis, P. (2005). The limits of realist evaluation: Surfacing and exploring assumptions in assessing the best value performance regime. Evaluation, 11(3), 275–295.

    Article  Google Scholar 

  • Dey, I. (1999). Grounding grounded theory: Guidelines for qualitative inquiry. San Diego: Academic Press.

    Book  Google Scholar 

  • Greenhalgh, T., Humphrey, C., Hughes, J., Macfarlane, F., Butler, C., & Pawson, R. (2009). How do you modernize a health service? A realist evaluation of whole-scale transformation in London. Milbank Quarterly, 87(2), 391–416.

    Article  Google Scholar 

  • Haji, F., Morin, M.-P., & Parker, K. (2013). Rethinking programme evaluation in health professions education: Beyond “did it work?”. Medical Education, 47(4), 342–351.

    Article  Google Scholar 

  • Hetzner, S., Heid, H., & Gruber, H. (2012). Change at work and professional learning: How readiness to change, self-determination and personal initiative affect individual learning through reflection. European Journal of Psychology of Education, 27(4), 539–555.

    Article  Google Scholar 

  • Hutchinson, L. (2003). Educational environment. BMJ (Clinical research ed.), 326(7393), 810–812.

    Article  Google Scholar 

  • Kanter, R. M. (1993). Men and women of the corporation. New York, NY: BasicBooks.

    Google Scholar 

  • Karoly, P. (1993). Mechanisms of self-regulation: A systems view. Annual Review of Psychology, 44(1), 23–52.

    Article  Google Scholar 

  • Knol, J., & Van Linge, R. (2009). Innovative behaviour: The effect of structural and psychological empowerment on nurses. Journal of Advanced Nursing, 65(2), 359–370.

    Article  Google Scholar 

  • Laschinger, H., Finegan, J., Shamian, J., & Wilk, P. (2001). Impact of structural and psychological empowerment on job strain in nursing work settings: Expanding Kanter’s model. The Journal of Nursing Administration, 31(5), 260–272.

    Article  Google Scholar 

  • Laschinger, H., Finegan, J., Shamian, J., & Wilk, P. (2004). A longitudinal analysis of the impact of workplace empowerment on staff nurses’ work satisfaction. Journal of Organizational Behaviour, 25, 527–545.

    Article  Google Scholar 

  • Leslie, K., Baker, L., Egan-Lee, E., Esdaile, M., & Reeves, S. (2013). Advancing faculty development in medical education: A systematic review. Academic Medicine, 88(7), 1038–1045.

    Article  Google Scholar 

  • Lieff, S. (2009). Evolving curriculum design: A novel framework for continuous, timely, and relevant curriculum adaptation in faculty development. Academic Medicine, 84(1), 127–134.

    Article  Google Scholar 

  • Lyso, I. H. (2010). Managerial learning as co-reflective practice: Management development programs—Don’t use it if you don’t mean it. Dissertation, Norges Teknisk-Taturvitenskapelige Universitet.

  • McEvoy, P., & Richards, D. (2003). Critical realism: A way forward for evaluation research in nursing? Journal of Advanced Nursing, 43(4), 411–420.

    Article  Google Scholar 

  • McLeod, P. J., & Steinert, Y. (2010). The evolution of faculty development in Canada since the 1980s: Coming of age or time for a change? Medical Teacher, 32(1), e31–e35.

    Article  Google Scholar 

  • Monaghan, M. (2012). An Interview with Ray Pawson. Evidence and Policy, 8(4), 511–515.

    Article  Google Scholar 

  • Moon, J. A. A. (2004). Handbook of reflective and experiential learning: Theory and practice. London: Routledge Falmer.

    Google Scholar 

  • Nesbit, P. L. (2012). The role of self-reflection, emotional management of feedback, and self- regulation processes in self-directed leadership development. Human Resource Development Review, 11(2), 203–226.

    Article  Google Scholar 

  • Ng, S. L., Kinsella, E. A., Friesen, F., & Hodges, B. (2015). Reclaiming a theoretical orientation to reflection in medical education research: A critical narrative review. Medical Education, 49(5), 461–475.

    Article  Google Scholar 

  • O’Sullivan, P. S., & Irby, D. M. (2011). Reframing research on faculty development. Academic Medicine, 86(4), 421–428.

    Article  Google Scholar 

  • Parker, K., Burrows, G., Nash, H., & Rosenblum, N. D. (2011). Going beyond Kirkpatrick in evaluating a clinician scientist program: It’s not “if it works” but “how it works”. Academic Medicine, 86(11), 1389–1396.

    Article  Google Scholar 

  • Patton, M. Q. (2011). Developmental evaluation: Applying complexity concepts to enhance innovation and use. New York, NY: Guilford Press.

    Google Scholar 

  • Pawson, R. (2013). The science of evaluation: A realist manifesto. London: Sage Publications.

    Book  Google Scholar 

  • Pawson, R., & Tilley, N. (1997). Realistic evaluation. Thousand Oaks, CA: Sage.

    Google Scholar 

  • Ritchie, J., & Spencer, L. (1994). Qualitative data analysis for applied policy research. In A. Bryman & R. Burgess (Eds.), Analysing qualitative data (pp. 173–194). London: Routledge.

    Chapter  Google Scholar 

  • Rogers, P. J., Hasci, T. A., Petrosino, A., & Huebner, T. A. (Eds.). (2000). Program theory in evaluation: Challenges and opportunities. New directions for evaluation (No. 87). San Francisco: Jossey-Bass.

    Google Scholar 

  • Rushmer, R., Kelly, D., Lough, M., Wilkinson, J. E., & Davies, H. T. O. (2004). Introducing the learning practice-II. Becoming a learning practice. Journal of Evaluation in Clinical Practice, 10(3), 387–398.

    Article  Google Scholar 

  • Singh, T. (2013). Medical education faculty development : Why it fails to impress us. The National Medical Journal of India, 26(2), 96–99.

    Google Scholar 

  • Snape, D., & Spencer, L. (2003). The foundations of qualitative research. In J. Ritchie & J. Lewis (Eds.), Qualitative research practice: A guide for social science students and researchers. London: Sage Publications.

    Google Scholar 

  • Sorinola, O. O., Thistlethwaite, J., Davies, D., & Peile, E. (2015). Faculty development for educators: A realist evaluation. Advances in Health Sciences Education, 20(2), 385–401. doi:10.1007/s10459-014-9534-4.

    Article  Google Scholar 

  • Spreitzer, G. M. (1995a). An empirical test of a comprehensive model of intrapersonal empowerment in the workplace. American Journal of Community Psychology, 23(5), 601–629.

    Article  Google Scholar 

  • Spreitzer, G. M. (1995b). Psychological empowerment in the workplace: Dimensions, measurement, and validation. Academy of Management Journal, 38(5), 1442–1465.

    Article  Google Scholar 

  • Spreitzer, G. M. (2009). Taking stock: A review of more than twenty years of research on empowerment at work. In S. R. Clegg & C. L. Cooper (Eds.), The SAGE handbook of organizational behaviour (Vol. 2). Thousand Oaks, CA: SAGE.

    Google Scholar 

  • Stame, N. (2004). Theory-based evaluation and types of complexity. Evaluation, 10(1), 58–76.

    Article  Google Scholar 

  • Steinert, Y. (2010). Faculty development: From workshops to communities of practice. Medical Teacher, 32(5), 425–428.

    Article  Google Scholar 

  • Steinert, Y. (2014). Faculty development in the health professions: A focus on research and practice. New York: Springer.

    Book  Google Scholar 

  • Thomas, K. W., & Velthouse, B. A. (1990). Cognitive elements of empowerment: An “interpretive” model of intrinsic task motivation. Academy of Management Review, 15, 666–681.

    Google Scholar 

  • Ward, D. J., Furber, C., Tierney, S., & Swallow, V. (2013). Using Framework Analysis in nursing research: A worked example. Journal of Advanced Nursing, 69(11), 2423–2431.

    Google Scholar 

  • Watkins, K. E., Lyso, I. H., & deMarrais, K. (2011). Evaluating executive leadership programs: A theory of change approach. Advances in Developing Human Resources, 13(2), 208–239.

    Article  Google Scholar 

  • West, S. G., & Aiken, L. S. (1997). Toward understanding individual effects in multicomponent prevention programs: Design and analysis strategies. In K. J. Bryant, M. Windle, & S. G. West (Eds.), The science of prevention: Methodological advances from alcohol and substance abuse research (pp. 167–209). Washington, DC: American Psychological Association.

    Chapter  Google Scholar 

  • Young, J. E., Williamson, M. I., & Egan, T. G. (2015). Students’ reflections on the relationships between safe learning environments, learning challenge and positive experiences of learning in a simulated GP clinic. Advances in Health Sciences Education,. doi:10.1007/s10459-015-9611-3.

    Google Scholar 

  • Yukawa, J. (2006). Co-reflection in online learning: Collaborative critical thinking as narrative. International Journal of Computer-Supported Collaborative Learning, 1(2), 203–228.

    Article  Google Scholar 

  • Zibrowski, E. M., Weston, W. W., & Goldszmidt, M. A. (2008). ‘I don’t have time’: Issues of fragmentation, prioritisation and motivation for education scholarship among medical faculty. Medical Education, 42(9), 872–878.

    Article  Google Scholar 

Download references

Acknowledgments

The authors would like to acknowledge Jasmine Sheehan for her coordination of the ESP program, Amy Dionne for her project management and support of the project, and Farah Friesen and Daniel Miller for their editorial support during manuscript preparation.

Author information

Authors and Affiliations

  1. Centre for Faculty Development, Faculty of Medicine, Li Ka Shing Knowledge Institute, University of Toronto at St. Michael’s Hospital, 4th Floor, 209 Victoria Street, Toronto, ON, M5B 1W8, Canada

    Betty Onyura, Stella L. Ng, Lindsay R. Baker, Susan Lieff, Barbara-Ann Millar & Brenda Mori

  2. Department of Speech-Language Pathology, University of Toronto, Toronto, ON, Canada

    Stella L. Ng

  3. Centre for Ambulatory Care Education, Women’s College Hospital, Toronto, ON, Canada

    Stella L. Ng

  4. Rehabilitation Sciences Institute (RSI), University of Toronto, Toronto, ON, Canada

    Stella L. Ng

  5. The Wilson Centre, Faculty of Medicine, University of Toronto at University Health Network, Toronto, ON, Canada

    Stella L. Ng

  6. Li Ka Shing Knowledge Institute, St. Michael’s Hospital, Toronto, ON, Canada

    Lindsay R. Baker

  7. Department of Psychiatry, Faculty of Medicine, University of Toronto, Toronto, ON, Canada

    Susan Lieff

  8. Department of Radiation Oncology, Faculty of Medicine, University of Toronto, Toronto, ON, Canada

    Barbara-Ann Millar

  9. Department of Physical Therapy, Faculty of Medicine, University of Toronto, Toronto, ON, Canada

    Brenda Mori

Authors
  1. Betty Onyura
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Stella L. Ng
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Lindsay R. Baker
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Susan Lieff
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Barbara-Ann Millar
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Brenda Mori
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Betty Onyura.

Ethics declarations

Conflict of interest

The authors of this study work at the Centre for Faculty Development.

Ethical standard

The study was approved by St. Michael’s Hospital Research Ethics Board (REB 07-182).

Appendices

Appendix 1: Logic model for longitudinal faculty development program

figure a

Appendix 2: Semi-structured interview protocol for longitudinal follow-up with program graduates

  1. 1.

    Please describe your current role(s) in education and what it involves.

    • Probes

    • How does that (your role) affect the people you work with internally/externally?

  2. 2.

    We sent you the last reflection paper you wrote while in the program for review. We are interested in exploring your experiences since that point in time.

    • Probes

    • Please describe your journey since your completion of the program?

    • Have there been any changes for you since that point in time?

    • What has your involvement been in education (in the academic health professions education system) since the program?

    • Have you had any influence in your role/capacity/system? In what ways have you influenced your surroundings?

  3. 3.

    Please describe any successes you have had since completing the program

    • Probes

    • In what contexts have you had successes?

    • How do you define success?

    • Why do you think you were successful?

    • What are the factors (individual or contextual) that influenced your success?

    • Are there areas where you feel you haven’t achieved what you wanted to achieve?

    • Why do you think you were unsuccessful?

    • What would success look like for you over the next 5 years?

  4. 4.

    Who do you talk to about teaching and education?

    • Probes

    • Why (do you talk to these people)?

    • Have the people you talk to (engage with) about education changed since your completion of the program? Please explain

    • Has your approach toward them changed? Please explain

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Onyura, B., Ng, S.L., Baker, L.R. et al. A mandala of faculty development: using theory-based evaluation to explore contexts, mechanisms and outcomes. Adv in Health Sci Educ 22, 165–186 (2017). https://doi.org/10.1007/s10459-016-9690-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10459-016-9690-9

Keywords

Search

Navigation