Background
Methods
Overview
Study population
Formative evaluation: Telephone interviews
Formative evaluation: Pre-program web-based survey
Development and implementation of the PSI educational program
Post-program evaluation: Web-based survey
Results
Obtain VA stakeholders’ input on their educational needs related to the PSIs.
Formative evaluation: Telephone interviews
-
Some informants had heard about the PSIs at meetings held by national VA quality and safety offices, whereas other informants had not heard about the PSIs and were not aware that PSI rates were in reports that they received.
-
Some were unclear on how to interpret PSIs.
-
While some had received education on other performance measures from national VA offices, most reported that they had not received any formal education on the PSIs.
-
Informants suggested ways they would like the PSI education to be delivered, such as web-based training and information placed on a VA SharePoint.
-
Informants suggested making the educational program more relevant to the audience we were trying to educate and also more interactive.
-
Informants believed that improvement priorities can be heavily driven by a hospital’s performance on quality/safety measures.
-
Informants felt that improvement priorities are largely set by senior management according to internal hospital priorities; however, they can also be set according to mandates from the VA National and Regional Offices.
-
Web conferencing (e.g. LiveMeeting)
-
Video conferencing (e.g., v-tel)
-
Reports or journal articles
-
Written case studies
-
Video/audio materials (e.g., links to pre-recorded materials)
-
Face-to-face conference
-
Q&A sessions (e.g., via teleconference)
Concept | Interview question → Survey question |
---|---|
PSI Knowledge | Have you heard of the PSIs prior to this interview? → I am aware of the PSIs. |
Have you ever received a report that contains the PSI rates? → I currently receive reports containing rates for selected PSIs from Inpatient Evaluation Center (IPEC). | |
PSI Education | Have you received any education about the PSIs? → I have received education about the PSIs from VA Central Office [e.g., Inpatient Evaluation Center (IPEC), National Center for Patient Safety (NCPS)]. |
If you were to receive education on the PSIs, how would you like it to be delivered? → I would prefer to learn about the PSIs through: web conferencing, video conferencing, reports or journal articles, written case studies, video/audio materials (e.g., links to pre-recorded materials), face-to-face conference, Q&A sessions (e.g., via teleconference). | |
Improvement Prioritization | Are priorities within your organization set in response to VACO or VISN mandates? → Patient safety/quality priorities for my facility are set mostly in response to: VACO mandates, VISN mandates, within the facility. |
Are priorities driven by performance measures vs. driven by other indicators (e.g., adverse events that occur, employee concerns, strategic planning, etc.)? → Patient safety/quality priorities at my facility are mostly driven by VACO performance measures. → Patient safety/quality priorities at my facility are mostly driven by other factors: adverse events that have occurred, employee concerns, strategic planning, press/media/public relations. |
Formative evaluation: Web-based survey
Position title | Number | Percent |
---|---|---|
Middle Managers | ||
Quality/Performance Improvement Manager/Officer | 50 | 28 |
Patient Safety Manager/Officer | 41 | 22 |
Senior Managers | ||
Medical Center Director | 31 | 17 |
Associate Director for Nursing and Patient Care Services/ Chief Nurse Executive | 27 | 15 |
Chief of Staff (i.e., Chief Clinical Officer) | 17 | 10 |
Associate Medical Center Director | 15 | 8 |
Total | 181 | 100 |
Develop and implement a PSI educational program based on stakeholders’ input.
Program delivery mode
-
We offered six cyberseminar sessions via LiveMeeting (a web-conferencing platform used throughout the VA) because survey respondents preferred this mode. Each session was 45 min and was offered once. On average, nine participants attended each of the cyberseminar sessions. A majority of cyberseminar participants were from our middle manager stakeholder group.
-
In addition, by offering the program as cyberseminars, we were able to easily record and archive the sessions on the program’s SharePoint site so that stakeholders who were unable to attend the live cyberseminars, or who wanted to revisit the sessions, could readily access them. (Recordings can be made available upon request from the corresponding author.) This allowed us to tailor the program based on stakeholders’ input from the interview.
-
To make the sessions more interactive, the cyberseminars were conducted in an interview format, similar to the style that AHRQ uses in its podcasts about the Quality Indicators and Toolkit, with a moderator interviewing a speaker. Cyberseminar speakers were study team members with extensive knowledge about the topic presented, or representatives from our VA national partners’ offices [VA National Center for Patient Safety (NCPS) and VA Inpatient Evaluation Center (IPEC)]. Using LiveMeeting also facilitated an interactive platform, as it allowed participants to ask questions at any time during the presentation and receive answers directly from the speakers at that time.
-
We integrated snapshots of the VA’s PSI report into our cyberseminars so that participants could directly apply what they learned from our program because survey respondents indicated that they were interested in learning how to use the PSIs, and interview informants suggested making the education more relevant to participants.
Program content
-
Cyberseminar session topics and content corresponded to survey respondents’ expressed needs and interests. For example, because respondents indicated that they did not understand how to interpret the PSI rates in reports, we developed two sessions substantially devoted to this (Session 2: The PSIs and Your Facility’s Report; Session 3: How to Interpret PSI Rates). These sessions also focused on two of the three content areas that were of most interest to survey respondents (how to interpret PSI rates and specific definitions of selected PSIs). Table 3 provides additional information about content areas of interest as indicated by survey respondents, each cyberseminar session, topic, and outline of content covered.
-
We also developed educational materials to complement what participants were learning in the cyberseminars: 1) an informational sheet, “Interpreting the AHRQ PSIs: A Basic Overview,” which was to be used, along with the cyberseminars, as a tool to help in interpreting and understanding the PSIs, as provided in Additional files 4 and 2) a matrix that highlighted information on each session and provided a list of materials referenced in each session for ease of access to materials, as provided in Additional file 5.
Content Area of Interest for the PSI Educational Program (Survey Response %, Fig. 2) | Cyberseminar Session and Topic | Outline of Information Covered |
---|---|---|
Session 1: An Overview of the PSI Educational Program | • Brief history about the study and the development of the PSI program • Road map of the cyberseminars and what topics will be covered during the cyberseminars | |
• How to interpret PSI rates (92%) • Specific definitions of selected PSIs (90%) | Session 2: The PSIs and Your Facility’s Reports | • Overview of the PSIs (e.g., definitions of PSIs) • Public reporting of the PSIs • Centers for Medicare and Medicaid Services Hospital Compare website for non-VA hospitals • VA reporting of the PSIs • Review of a sample VA PSI report • Where to find and request additional information on your facility’s VA PSI report • Who to contact with questions about your facility’s VA PSI report |
• How to interpret PSI rates (92%) • Specific definitions of selected PSIs (90%) | Session 3: How to Interpret PSI Rates | • Further review of the PSIs (e.g., definitions of additional PSIs) • How PSIs are calculated (e.g., observed rates vs. risk adjusted rates) • How valid are the PSIs • Positive predictive values • Reasons for false positive findings • Negative predictive values • Processes of care |
• How to use the PSIs for monitoring trends (93%) • How to use the PSIs for case-finding (86%) | Session 4: How to Use the PSIs and Organizational Factors to Consider | • AHRQ Quality Indicators Toolkit • Monitoring trends/benchmarking/case finding • Identifying priorities for PSI rate improvement • Organizational factors to consider |
• How to use the PSIs for QI (86%) | Session 5: Using the PSIs for QI: Experiences Within and Outside the VA | • Using the PSIs for quality improvement • IHI Virtual Breakthrough Series on Postoperative Respiratory Failure • AHRQ Quality Indicators Toolkit • Case study from AHRQ newsletters |
Session 6: Wrap up call/Q&A session | • Summary of what cyberseminars covered • Q&A session |
Explore stakeholders’ perceptions about the program.
Post-program evaluation: Web-based survey
Question | Response | Number of Respondents | % Positive Change* | % Negative Change^ | |
---|---|---|---|---|---|
Pre | Post | ||||
I have received education about the PSIs from VA Central Office (e.g., IPEC, NCPS). | Yes | Yes | 4 | 3/13 = 23% | 1/13 = 8% |
No | Yes | 2* | |||
No response | Yes | 1* | |||
No | No | 5 | |||
Yes | No | 1^ | |||
I understand how the PSI rates are calculated in reports. | Yes | Yes | 2 | 8/13 = 62% | 2/13 = 15% |
No | Yes | 4* | |||
No response | Yes | 4* | |||
No | No | 1 | |||
No response | No | 2^ | |||
I understand how to interpret the PSI rates contained in reports. | Yes | Yes | 2 | 8/13 = 62% | 2/13 = 15% |
No | Yes | 4* | |||
No response | Yes | 4* | |||
No | No | 1 | |||
No response | No | 2^ | |||
My facility has established reducing PSI rates as a priority. | Strongly agree | Agree | 2^ | 1/13 = 8% | 4/13 = 31% |
Agree | Strongly agree | 1* | |||
Agree | Agree | 2 | |||
Agree | Disagree | 1^ | |||
Neither agree nor disagree | Neither agree nor disagree | 3 | |||
Neither agree nor disagree | No response | 1^ | |||
Disagree | Disagree | 2 | |||
No response | No response | 1 | |||
The PSIs are a valuable quality/patient safety measure. | Strongly agree | No response | 1^ | 1/13 = 8% | 6/13 = 46% |
Agree | Strongly agree | 1* | |||
Agree | Agree | 5 | |||
Agree | Neither agree nor disagree | 2^ | |||
Agree | No response | 1^ | |||
Neither agree nor disagree | Neither agree nor disagree | 1 | |||
Neither agree nor disagree | No response | 2^ | |||
The PSIs will be a QI priority in my facility over the next year. | Strongly agree | Strongly agree | 1 | 1/13 = 8% | 5/13 = 38% |
Agree | Agree | 3 | |||
Agree | Neither agree nor disagree | 1^ | |||
Agree | Disagree | 1^ | |||
Neither agree nor disagree | Neither agree nor disagree | 2 | |||
Neither agree nor disagree | Disagree | 1^ | |||
Neither agree nor disagree | No response | 2^ | |||
No response | Agree | 1* | |||
No response | No response | 1 | |||
I plan to devote time over the next year to improve my facility’s PSI rates. | Strongly agree | Agree | 1^ | 2/13 = 15% | 5/13 = 39% |
Agree | Agree | 2 | |||
Agree | Neither agree nor disagree | 2^ | |||
Neither agree nor disagree | Neither agree nor disagree | 3 | |||
Neither agree nor disagree | Agree | 1* | |||
Neither agree nor disagree | No response | 2^ | |||
No response | Agree | 1* | |||
No response | No response | 1 |