Background
Methods
Study population
The intervention
Program theory
Requirements
Intervention phase 1: needs assessment
| Source | |||||||
No.
|
Process component
|
Requirement based on program theory
|
Operationalization
|
Quan.
|
Qual.
| |||
Q1
|
Q2
|
Logs
|
I1
|
I2
| ||||
1 | Initiationa
| Were senior and middle management committed to the intervention at the start? | • Reasons for middle and senior management to participate | X | X | |||
2 | Communicationa
| Was the intervention project communicated to the employees at the start? | • The manner in which the project was communicated to the participants | X | ||||
3 | Participationa
| Was a participatory group formed? Did the employees feel involved in the intervention? | • Composition of group and procedure was in line with protocol • The majority of the participants scored above the cut-off point on the ‘employee involvement’ scale at T1 | X | X | X | ||
4 | Fidelitya
| Was intervention phase 1 delivered by HM facilitator according to protocol? | • The extent to which the HM facilitator complied with the needs assessment protocol, according to the facilitator and researcher | X | ||||
5 | Reacha
| Was intervention phase 1 received by majority of the employees? | • Attendance of employees in each step of the needs assessment according to objective attendance rates. The rate expresses the number of those who actually participated in each step out of those who were asked to participate in each step. • The majority of the participants scored above the cut-off point on the ‘exposure’ scale at T1 | X | X | |||
6 | Communicationa
| Were results of each step in phase 1 communicated to employees by HM facilitator? | • Percentage of participants who reported to have received a report of intervention step 1 interviews • Percentage of participants who reported to have received a report of intervention step 2 questionnaires • Percentage of participants who reported to have received a report of intervention step 3 group sessions | X | ||||
7 | Satisfactiona
| Were the employees satisfied with intervention phase 1? | • Satisfaction of all employees with (elements of the) needs assessment (ie interviews, questionnaire, group session, advisory report, overall) was moderate (≥6–7.4) or high (≥7.5)d
| X | X | |||
8 | Middle management supporta
| Was managerial support present at T1 according to management and employees? | • The majority of the participants scored above the cut-off point on the ‘line manager attitudes and actions’ scale at T1 • The managers demonstrated their support of the advisory report | X | X | |||
9 | Readiness for changec
| Was the majority of the employees at T1 ready for the change? | • The majority of the participants scored above the cut-off point on the ‘readiness for change’ scale at T1 | X | ||||
Intervention phase 2: implementation
| ||||||||
No.
|
Process component
|
Requirement based on program theory
|
Operationalization
|
Quan.
|
Qual.
| |||
Q1
|
Q2
|
Logs
|
I1
|
I2
| ||||
10 | Middle management supporta
| Was an action plan formulated by middle managers based on the advisory report? Were quick wins formulated? | • Middle managers visibly supported the project by designing an action plan including quick wins • The majority of the employees scored above the cut-off point on the ‘line manager attitudes and actions’ scale at T2 | X | X | |||
11 | Participationa
| Did employees participate in formulating an action plan? | • The extent to which employees felt responsible for the action plan and the result of implementing the action plan (ie ownership) • The majority of the employees scored above the cut-off point on the ‘employee involvement’ scale at T2 • Formal representatives had a role • Middle managers encouraged active participation by employees | X | X | |||
12 | Targetinga
| Did the action plan target the right problems in the workplace? | • The action plan was applicable to the problems of the workplace • Satisfaction with content action plan was moderate (6.0–7.4) or high (≥7.5) | X | X | |||
13 | Senior management supporta
| Did senior management support the action plan? | • Senior managers supported the project throughout • Senior managers allocated the necessary resources | X | X | |||
14 | Communicationa
| Was the action plan communicated to the employees? Were small successes celebrated? | • Employees were informed about the action plan and the progress towards its goals • Small successes were celebrated | X | X | |||
15 | Deliverya
| Was the action plan implemented by middle managers? | • Perceived implementation of the action plan, including quick wins, according to the implementers | X | ||||
16 | Exposurea
| Were the employees exposed to implementation of the action plan? | • Perceived implementation of the action plan, including quick wins, according to the employees • The majority of the participants scored above the cut-off point on the ‘exposure to intended intervention’ scale at T2 | X | X | |||
17 | Cultureb
| Did the organizational culture facilitate the implementation of the action plan? | • Inherent characteristics of the organizational culture that facilitated or impeded the implementation of the action plan | X | ||||
18 | Conditionsb
| Did the organization have the capacity to implement the action plan? | • The organizational characteristics that affected the (implementation) of the action plan • The organization had the capacity and skills to implement the action plan | X | ||||
19 | Eventsb
| Did no events interfere with the implementation of the action plan? | • Events that occurred and influenced the content or the execution of the action plan | X | X | |||
20 | Readiness for changec
| Was the majority of the employees at T2 ready for the change? | • The majority of the participants scored above the cut-off point on the ‘readiness for change’ scale at T2 • The extent to which employees were ready for change and how this influenced the execution of the action plan | X | X | |||
21 | Satisfactiona
| Were the employees satisfied with intervention phase 2? | • Satisfaction with intervention phase 2 was moderate (6.0–7.4) or high (≥7.5)d
| X | X | |||
22 | Perceptionc
| Did implementers and employees perceive the intervention as positive? | • The perception of the action plan was positive • Identify the common grounds and changes in the perception of the action plan | X |
The process evaluation framework
Intervention
Context
Participant’s mental models
Procedure of applying the framework
Data collection
Questionnaire (measures)
Interviews
Logbook
Data analysis
Results
Intervention phase 1: needs assessment
Were senior and middle management committed to the intervention at the start? (initiation)
Process component
|
Requirement
|
School A
|
School B
| |||
Intervention phase 1: needs assessment
|
Quan.
|
Qual.
|
Quan.
|
Qual.
| ||
1 | Initiation | Were senior and middle management committed to the intervention at the start? | - | Yes | - | Yes |
2 | Communication | Was the intervention project communicated to the employees? | - | Yes | - | Yes |
3 | Participation | Was a participatory group formed? | - | Yes | - | Partly |
Did the employees feel involved in the intervention? | Yes | - | No | - | ||
4 | Fidelity | Was intervention phase 1 delivered by HM-facilitator according to protocol? | - | Yes | - | Yes |
5 | Reach | Was intervention phase 1 received by majority of the employees? | Yes | Partly | No | Partly |
6 | Communication | Were results of each step in phase 1 communicated to employees by HM-facilitator? | Yes | Yes | Yes | Yes |
7 | Satisfaction | Were the employees satisfied with intervention phase 1? | Yes | Partly | Yes | Partly |
8 | Managerial support | Was managerial support present at T1 according to management and employees? | Yes | Yes | Yes | No |
9 | Readiness for change | Was the majority of the employees at T1 ready for the change? | Yes | - | No | - |
Process component
|
Requirement
|
School A
|
School B
| |||
Intervention phase 2: implementation
|
Quan.
|
Qual.
|
Quan.
|
Qual.
| ||
10 | Middle management support | Was an action plan formulated by middle managers based on the advisory report? | Yes | Yes | Yes | No |
Were quick wins formulated? | Yes | Yes | No | Partly | ||
11 | Participation | Did the employees participate in formulating an action plan? | Yes | No | Partly | No |
12 | Targeting | Did the action plan target the right problems in the workplace? | Yes | Partly | No | No |
13 | Senior management support | Did senior management support the action plan? | - | Partly | - | Partly |
14 | Communication | Was the action plan communicated to the employees? | Yes | Yes | No | Partly |
15 | Delivery | Was the action plan implemented by middle managers? | - | Yes | - | Partly |
16 | Exposure | Were the employees exposed to implementation of the action plan? | Partly | Partly | Partly | Partly |
17 | Culture | Did the organizational culture facilitate the implementation of the action plan? | - | No | - | No |
18 | Conditions | Did the organization have the capacity to implement the action plan? | - | Partly | - | No |
19 | Events | Did no events interfere with the implementation of the action plan? | - | No | - | No |
20 | Readiness for change | Was the majority of the employees at T2 ready for the change? | No | No | Yes | No |
21 | Satisfaction | Were the employees satisfied with intervention phase 2? | No | - | No | - |
22 | Perception | Did implementers and employees perceive the intervention as positive? | - | Partly | - | No |
Was the intervention project communicated to the employees at the start? (communication)
Was a participatory group formed? And did the employees feel involved in the intervention? (participation)
School A | School B | |
---|---|---|
3. Participation | ||
% of employees who feel involved in the intervention | 71.9% (n = 23/32) | 34.4% (n = 22/64) |
5. Reach (% yes) | ||
Participation in interviewsa
| 91.7% (n = 11/12) | 100% (n = 12/12) |
Participation in questionnairea
| 39.3% (n = 59/150) | 47.8% (n = 77/161) |
Participation in group sessionsb
| 73.3% (n = 110/150) | 54.0% (n = 71/161) |
% of participants who feel exposed to the intervention | 68.8% (n = 22/32) | 29.7% (n = 28/64) |
6. Communication (% yes) | ||
Received report on interviews? | 53.1% (n = 17/32) | 57.1% (n = 40/70) |
Received report on questionnaire? | 53.1% (n = 17/32) | 68.6% (n = 48/70) |
Received advisory report? | 93.8% (n = 30/32) | 65.7% (n = 46/70) |
7. Satisfactionc (SD)d
| ||
Overall | 6.5 (1.19) (n = 32) | 5.9 (1.58) (n = 63) |
Interviews | 7.5 (.57) (n = 4) | 8.0 (0.89) (n = 11) |
Questionnaire | 7.1 (1.14) (n = 26) | 7.3 (0.99) (n = 55) |
Group sessions | 6.7 (1.13) (n = 26) | 6.8 (1.58) (n = 38) |
Advisory report correct summary of opinions/wishes/needs? (%yes) | 64.5% (n = 20/31) | 65.4% (n = 34/52) |
8. Middle management support | ||
% employees who feel supported | 68.8% (n = 22/32) | 53.1% (n = 34/64) |
9. Readiness for change | ||
% employees who feel ready for the change | 81.3% (n = 26/32) | 54.7% (n = 35/64) |
Was intervention phase 1 delivered by HM facilitator according to protocol? (fidelity)
Was intervention phase 1 received by the majority of the employees? (reach)
Were results of each step in phase 1 communicated to employees by HM facilitator? (communication)
Were the employees satisfied with intervention phase 1? (satisfaction)
Main content of advisory report delivered by facilitator
|
Main content of action plan
a
constructed by management team
| |
---|---|---|
School A | i) professionalize the teams; | The director, assisted by an HM facilitator, translated the recommendations into an action plan with three goals, six changes and a set of quick wins. GOALS: i) unambiguous management control; ii) competence and professionalism in the teams, and iii) adequate facilities CHANGES: (i) compliance to the workload policy, (ii) structured performance reviews; (iii) a continuous dialogue on the organization of the educational programs; (iv) a leading team activities plan; (v) weekly work meetings; and (vi) personalized competence development plans. QUICK WINS: create adequate facilities by creating a staff room at both locations; place extra walls in some classrooms; place beamers in all class rooms; improve the service by the facilitation services office. |
(ii) professionalize the management; | ||
(iii) improve the administrative support and facilities. | ||
School B | (i) create adequate and effective management control by installing a management team that is approachable, coaching and leading; | The directors of the management team decided to integrate the facilitator’s recommendations in the annual agreements (ie a Management Contract) they made with the Executive Board, instead of writing a separate action planb. A third party coach was attracted to support teams in a previously initiated change towards becoming self-managing. GOALS in the Management Contract were formulated in five headlines: i) strategy; ii) education; iii) personnel; iv) organization; and v) business operations. The most important CHANGE per headline was: i) form alliances with care partners in the region; ii) change the class bound curriculum of two educational programs into more concentrated ‘learning units’; iii) implement performance review policies; iv) make teams function as self-managing units; and v) develop a multi-annual housing plan for the school. No QUICK WINS were formulated. |
(ii) strengthen the power of teams within the school, by letting them develop a team program that can guide their daily work and makes them actually ‘self-managing’; | ||
(iii) make administrative procedures more efficient. |
Was managerial support present at T1 according to management and employees? (middle management support)
Intervention phase 2: implementation
Was an action plan formulated by middle managers based on the advisory report? And were quick wins formulated? (middle management support)
School A | School B | |
---|---|---|
10. Middle management support (% employees that perceives managerial support for action plana) | 57.1% (n = 32/56) | 51.1% (n = 23/45) |
11. Participation (% yes) | ||
I feel responsible for the implementation of the action plan
a
| 73.2% (n = 41/56) | 76.1% (n = 35/46) |
I feel responsible for the content of the action plan
a
| 71.4% (n = 40/56) | 76.1% (n = 35/46) |
Employee involvement (% employees who feel involved in action plan) | 55.4% (n = 31/56) | 37.0% (n = 17/46) |
12. Targeting (SD)b
| ||
Satisfaction with content of the action plana
| 6.5 (1.31) (n = 61) | 5.4 (1.96) (n = 49) |
14. Communication (% yes) | ||
Are you informed about the progress in the action plan
a
?
| 86.7% (n = 52/60) | 38.3% (n = 18/47) |
How were you informed (more answers possible)?
| ||
Via information meetings organized by the management team | 75% (n = 39/52) | 61.1% (n = 11/18) |
Via work meetings | 44.2% (n = 23/52) | 44.4% (n = 8/18) |
Via the news letter | 40.4% (n = 21/52) | 38.9% (n = 7/18) |
In another way | 7.7% (n = 4/52) | 16.7% (n = 3/18) |
16. Exposure (% yes) | ||
Have you been informed on the existence of the action plan
a
?
| 100% (n = 61/61) | 87.8% (n = 43/49) |
Are you familiar with the goals of the action plan
a
?
| ||
unambiguous management control teams | 83.6% (n = 51/61) | - |
competence and professionalism in the teams | 90.2% (n = 55/61) | - |
adequate facilities | 90.2% (n = 55/61) | - |
make teams the central executive units | - | 89.8% (n = 44/49) |
adequate and effective management control | - | 67.3% (n = 33/49) |
Have you seen changes towards these goals:
| ||
unambiguous management control teams | 43.3% (n = 26/60) | - |
competence and professionalism in the teams | 43.3% (n = 26/60) | - |
adequate facilities (quick wins) | 83.6% (n = 51/61) | - |
make teams the central executive units | - | 59.6% (n = 28/47) |
adequate and effective management control | - | 25.5% (n = 12/47) |
Are the quick wins
c
celebrated as a success?
| 67.2% (n = 41/61) | - |
Did you notice a change in:
| ||
a dialogue on the organization of education | 78.3% (n = 47/60) | - |
performance reviews | 76.7% (n = 46/60) | - |
team activities plan | 58.3% (n = 35/60) | - |
work meetings | 56.7% (n = 34/60) | - |
workload policy | 45% (n = 27/60) | - |
competence development plan | 28.3% (n = 17/60) | - |
teams develop a team- and educational program conform the regulations | - | 66.0% (n = 31/47) |
a better and larger management team | - | 57.4% (n = 27/47) |
management sets guiding standards | - | 31.9% (n = 15/47) |
management is permanently accessible | - | 36.2% (n = 17/47) |
management coaches teachers in educational leadership | - | 19.1% (n = 9/47) |
management eliminates ‘cumbersome’ administrative procedures | - | 8.5% (n = 4/47) |
% of participants that feels exposed to the intervention | 56.9% (n = 33/58) | 39.1% (n = 18/46) |
20. Readiness for change (% employees who feel ready for the change) | 46.4% (n = 26/56) | 54.3% (n = 25/46) |
21. Satisfaction (SD)b with implementation of the action plan/advisory report | 5.7 (1.4) (n = 58) | 4.4 (1.7) (n = 46) |