In the United States, anesthesia departments are required to provide the hospitals with physician-specific metrics that demonstrate competence in their professional practice in order for the physicians to maintain hospital privileges.1 The Joint Commission refers to the process as “Ongoing Professional Practice Evaluation” (OPPE).1 Six core competencies2 are often used as a framework for OPPE assessment.1 A competency is said to be a “core” competency when it applies to every physician, regardless of medical specialty. The core competencies include patient care and procedural skills, interpersonal and communication skills, and professionalism.Footnote 1 The recent review article by Yang addressing professionalism in anesthesia emphasized that feedback is essential to improve professionalism, whether in “the form of annual reappointment assessments, peer assessments, 360° evaluations,” and so on.3 A limitation, however, is that none of these areas is measured on a specific scale (instrument). Thus, the development of a reliable, valid way to assess anesthesiologists’ professionalism is needed.4

Many anesthesiologists working in operating rooms and other procedural locations (henceforth referred to as “ORs”) provide clinical supervision (e.g., generally with anesthesia residents and, in the United States, nurse anesthetists).5,6 “Supervision” entails clinical oversight functions directed toward ensuring the quality of clinical care whenever the anesthesiologist is not the sole anesthesia care provider. When the anesthesiologist is supervising an OR, depending on other responsibilities – such as in another OR and/or a post-anesthesia care unit – it can be challenging to be present at critical portions of every anesthetic delivery and still contribute fully to each patient’s care.5,7–9 Thus, our department monitors the quality of anesthesiologists’ supervision and provides them with periodic feedback.5,10

Individual faculty anesthesiologists can be reliably and dependably assessed using the de Oliveira Filho nine-item clinical supervision scale (Table 1).11–13 Low supervision scores have been associated with resident reports of increased errors with adverse effects on patients.14–16 The nine-item scale was designed to measure all attributes of faculty anesthesiologists’ supervision of anesthesia residents (Table 1). The scale has repeatedly been shown in multiple studies to do so as a unidimensional construct (including assessments by nurse anesthetists).13,15,17,18 Along with the daily numerical scores of faculty supervision, our residents may also provide a written free-text comment about the faculty anesthesiologist.18 The supervision scores and comments are the cumulative result of how the anesthesiologists perform in clinical environments (i.e., they reflect in situ performance and improve with feedback).6,10,17

Table 1 de Oliveira Filho Scale17 for Measuring Faculty Anesthesiologists’ Supervision of Residents During Clinical Operating Room Care
Full size table

In the current study, we performed a content analysis of a cohort of our department’s residents’ written evaluations of faculty anesthesiologists to test two hypotheses. One hypothesis was that the clinical supervision scale includes attributes of professionalism and thus is in part assessing anesthesiologists’ professionalism. The basis for our hypothesis was that de Oliveira Filho et al. labelled their question #4 (Table 1) as a reflection of “professionalism.”17 Our second hypothesis was that professionalism has other attributes not measured by the supervision scale and not reported by our residents (i.e., supervision and professionalism are not equivalent constructs). The basis for our hypothesis was that previous studies have listed words and phrases associated with professionalism but seemingly of different themes (Tables 2-4).2,19–23

Table 2 Words and Phrases in the Quality of Supervision Scale and their Association with Professionalism and other Competencies
Full size table

Methods

The University of Iowa Institutional Review Board declared (November 9, 2015) that this investigation did not meet the regulatory definition of human subjects research. All analyses were performed with de-identified data.

From July 1, 2013 to the present, our department has utilized the de Oliveira Filho supervision scale6,10–17 to assess the quality of clinical supervision provided by our anesthesiologists (Table 1). The cohort reported herein included all of the residents’ evaluations of all faculty anesthesiologists over the 2.5-year period, from the starting date of July 1, 2013 through December 31, 2015; the study was initiated February 2016 (see Discussion newsletter article). The evaluation process consisted of daily, automated e-mail requests24 to anesthesia residents to evaluate the supervision provided by each anesthesiologist with whom they worked the previous day in an OR setting, including obstetrics and/or non-operating room anesthesia (e.g., radiation therapy).6,10–12,18 The residents provided numerical scores to answer the scale’s nine questions (Table 1). The 13,664 numerical evaluations were supplemented with 1387 free-text comments.Footnote 2 Comments and scores are confidential, with scores provided only after averaging. Comments were never provided directly to the individual anesthesiologist because their usefulness depended on the procedural context, which often revealed the identity of the evaluating resident.10,18,25,26

Comments were searched automatically using Excel 2010 (Microsoft Corp., Redmond, WA, USA) for the presence or absence of words associated with physicians’ professionalism, without regard to capitalization.18 Words and phrases (Tables 2-4) associated with professionalism were obtained from the Accreditation Council for Graduate Medical Education’s Program Requirements for Anesthesiology and from previous studies.2,19–23 For testing the hypotheses, the words and phrases needed to represent attributes of professionalism. Tables 2-4 show these attributes and their corresponding references. The right-most columns of Tables 2-4 with the head “references for association with professionalism” were completed independently by each of this article’s authors to ensure an absence of coding error. Only words and phrases with concordance among the authors as being in a reference were included in Tables 2-4. This method was suitable for testing the hypotheses because the words and phrases needed not (and do not) represent a complete set of the attributes of professionalism. The hypotheses are deterministic (i.e., not based on “average,” “most,” and so on) and thus are not tested inferentially. The hypotheses are, however, potentially sensitive to a few words, and to residents, and/or faculty. Therefore, counts of each of these categories are included as sensitivity analyses.

Results

Table 2 shows that the nine-item supervision scale assesses some attributes of professionalism. There are eight “words or phrases” pertaining to professionalism. Table 3 contains examples of words and phrases that (a) residents provided with below-average supervision scores and (b) are related to professionalism. The 17 words or phrases (made by 11 residents) pertaining to professionalism are from 18 comments obtained on 18 different dates in regard to 14 anesthesiologists. Together, Tables 2 and 3 show that the unidimensional construct of clinical supervision has attributes in common with professionalism. Based on the findings of Tables 2 and 3, our hypothesis was satisfied and was insensitive to exclusion of specific dates, anesthesiologists, or residents.

Table 3 Words and Phrases Associated with Professionalism and Present in the 215 Resident Comments with Below Average Rating of Faculty Quality of Supervision
Full size table

In contrast, Table 4 contains 68 words and phrases related to professionalism that are present in zero (i.e., none) of the residents’ 1,387 comments (i.e., the comments associated with any score, low or high). Notably, this zero tally represented the absence of these words and phrases over the course of 13,664 evaluations, on 581 dates, by 90 anesthesiologists and 59 residents. Consequently, Table 4 shows that multiple attributes of professionalism are not elements of the clinical supervision scale. Thus, our second hypothesis also was satisfied – a result that also was insensitive to specific exclusion of dates, anesthesiologists, or residents.

Table 4 Words and Phrases Associated with Professionalism and Not Present in the Quality of Supervision Scale Questions17 or in any of the Residents’ 1,387 Positive or Negative Comments Made While Evaluating the Faculty
Full size table

Discussion

Our content analysis was motivated by a newsletter article from the Anesthesia Quality Institute Anesthesia Incident Reporting System Steering Committee, “Professionalism: Will we know it when we see it?”21 The article discussed a case report from an anesthesia incident-reporting system that described an event related, in part, to inadequate faculty supervision of an anesthesia resident. Specifically, there was insufficient faculty presence – an attribute of professionalism associated with low supervision scores (Table 3).13,18 There was concern regarding the resident’s honesty, another attribute of professionalism (Table 4).21 There was poor interpersonal communication (a core competency distinct from professionalism) between the resident and the faculty anesthesiologist, which may have contributed to the event. Thus, we had reason to expect that the supervision scale encompasses more than one of the six core competencies2 (Table 2) and that professionalism includes not only the dimension of supervision (Tables 2 and 3) but other attributes as well (Table 4) – i.e., the core competency of professionalism is multidimensional. Our results show that our conclusion holds. The conclusion matches that of Symons et al. in which factor analysis of family practice residents’ self-assessments identified two dimensions of professionalism: “interpersonal relations” and “conveying medical information [to patients].”27

The data in Tables 2 and 3 show that the conclusions are not self-evident. Regarding the implication that professionalism includes the dimension of supervision, three articles about professionalism in anesthesiology lacked the word “supervision,”3,22,23 including the review by Yang.3 Regarding the implication that the clinical supervision scale assesses anesthesiologists’ professionalism, our earlier article, showing that supervision scores were sufficient to identify anesthesiologists with disrespectful behaviour, did not include the word “professionalism.”18 Although de Oliveira Filho et al. labelled their question #4 (Table 1) “professionalism,” they did not show that it was measuring professionalism. We think that question #4’s three phrases assess professionalism, but so do eight other phrases in the supervision scale (Table 2).

We considered attributes of professionalism in clinical practice as it relates to maintenance of clinical privileges and OPPE. However, there are other attributes of professionalism. For example, among medical students and resident physicians, unprofessional behaviours include cheating, misrepresentation of publications, plagiarism, and falsification of documentation such as work hours.4 The addition of such words and phrases to Table 4 for these behaviours would have strengthened our finding that the constructs of “supervision” and “professionalism” are not equivalent, despite their extensive overlap of attributes (Tables 2 and 3).

Some of the attributes of professionalism in Table 4 may be unimportant to the residents and, hence, were not “worthy” of a comment.28 Alternatively, and we think more likely, during the course of their daily interaction in the OR with an anesthesiologist, anesthesia residents would not know if these attributes were present in their faculty. For example, we speculated that residents would care if the anesthesiologist violated the patient’s confidentiality (Table 4), in part because the faculty anesthesiologist’s patient is also the resident’s patient. However, it would be unlikely that the resident would know that a breach of confidentiality occurred because usually such an event occurs outside the OR. Thus, although Tables 2 and 3 show that residents’ daily assessments of faculty clinical supervision measure some attributes of professionalism, Table 4 shows that our residents’ assessments of the anesthesiologists’ professionalism are incomplete. There are two implications of these findings. Whereas less than frequent supervision10,29 shows unprofessional behaviour, it is entirely possible for an anesthesiologist to exhibit unprofessional behaviours and yet provide consistently high-quality supervision. Therefore, in addition to supervision scores and comments, other methods/metrics are necessary for full assessment of anesthesiologists’ professionalism.

To demonstrate competence in professional practice suitable for maintaining clinical privileges (i.e., OPPE), our department assesses and reports physician-specific practice metrics.1 Although the supervision scale includes several attributes of professionalism (Tables 2 and 3), it excludes many others (Table 4). For OPPE reporting,1 our department’s metrics also include (a) faculty completion rates for trainee evaluations over six-month periods and (b) faculty completion rates of anesthesia record attestations within six days after the date of service, also over the six-month periods. These additional metrics assess the professionalism measures of maintaining accurate records and being accountable (Table 4).

Tables 2 and 3 show that the supervision scale6,10–17 assesses three of the six core competencies to be assessed and reported: professionalism, interpersonal and communication skills, and patient care and procedural skills. Validity of its assessment of the competency of patient care and procedural skills is based also on the supervision scores being correlated with the raters’ choice of the anesthesiologist to care for their family.11 The supervision scale is a measure of an anesthesiologist’s individual contribution to the department, statistically independent of clinical production, specifically weekly anesthesia hours.6 The supervision scale also quantifies faculty anesthesiologists’ quality of clinical teaching.18 These multiple roles for the supervision scale are not obtained because the supervision scale is multi-dimensional. In fact, it has been found in multiple departments and settings to be unidimensional (Cronbach α = 0.948).11,13,15,17,18 Rather, it is because the core competencies,2 as well as clinical teaching, do not themselves represent distinct (i.e., separable) dimensions for assessment.

In conclusion, although the de Oliveira Filho clinical supervision scale includes some attributes of anesthesiologists’ professionalism. However, the core competency of professionalism is multidimensional. Thus, many attributes of professionalism were not addressed by the supervision scale and/or were not included in the residents’ comments.