Factors affecting the performance of maternal health care providers in Armenia

Training and service delivery organizations have tried for decades to improve the quality and access of healthcare services in developing countries by providing training to health care providers. The assumption has been that the gap in providers' performance is attributable to inadequate knowledge and skills – and therefore, training is what is called for. Millions of training dollars later, healthcare indicators are little improved and providers are still in need of support [1].

Realizing that training is oftentimes not the only solution, organizations such as IntraHealth International have searched for a key to improving outcomes at the most basic point of contact between the provider and the client. IntraHealth analyzed both domestic research in Human Performance Technology (HPT), as well as the current research on the systems influencing family planning provider performance within developing countries themselves. A line of research on family planning provider performance has focused on capturing a holistic view of the entire family planning program in a given region or country, where data is collected from experts in the area [2]. Although this research has offered evidence of overall program effects on population-level outcomes, it is of limited use in understanding specific conditions affecting the performance of providers at the facility level.

HPT experts in domestic countries, on the other hand, have long established that optimal worker performance is predicated on supporting the worker in a variety of areas, including: (1) clear job expectations; (2) timely performance feedback; (3) adequate environment and tools; (4) internal motivation and/or external incentives; (5) knowledge and skills; (6) capacity to do the job. IntraHealth/PRIME II adapted this list of factors to the international development field, subtracting 'capacity' and adding 'organizational support' to address inadequacies in the other five areas, such as supervision to deliver feedback and clarify expectations for providers. With these premises at hand, organizations have recently developed and utilized the Performance Improvement (PI) approach to improve provider performance and project interventions in the developing world [3‐8].

Although the logic of looking at workers' overall environment when analyzing productivity has been widely documented [9], there is little empirical research exploring the relationships between worker performance and the performance factors, and even less in developing or newly-independent countries. One of a few domestic studies was carried out by Hwang using a mail survey among employees and supervisors at the Office of the Inspector General in the Florida Department of Children and Families. Using an elaborate framework of five components of performance (quantity, quality, efficiency, problem-solving capacity and adaptability) and seven performance factors (performance specification, capacities, knowledge and skills, job/task design, incentives, feedback and resources and tools), he finds that in the employees' questionnaire, feedback is related to performance, in particular the efficiency dimension of performance. Though there are relationships between the factors and performance, in the questionnaires filled out by supervisors, results are less definite. Among the limitations of the study is that its cross-sectional design precluded making cause-effect inferences [10].

This study uses a simple framework: it assumes that human performance is facilitated and/or hindered by the 'performance factors' listed above and hypothesizes that not all factors are equal in their effects on performance. The objective of the study is to determine which factor(s) have a higher association with provider performance (See study framework in Figure 1). In order to control for extraneous factors, all providers are of similar cadre (i.e. nurse-midwives) and working in the same institution (MOH). Also, the type of establishment is limited to the primary/intermediate level of care (which is the PRIME project mandate), namely polyclinics, health centers and posts. Although human performance is comprised of both behavior and its accomplishments [11], for this study only the behavioral component of performance is investigated and will be determined through observation of health worker's realization of job tasks.

The study was conducted in Armenia as an extension to a baseline assessment of provider performance to inform the implementation of a USAID-funded project to improve maternal and neonatal health. It constituted a facility-based survey drawing on a sample of nurses and midwives working at three types of service delivery points: polyclinics, health centers and primary posts or FAPs, in the regions of Lori and Shirak. Consenting health care providers and clients were observed while providing two kinds of reproductive health services (prenatal and postpartum care); trained clinical data collectors recorded the completion of clinical and non-clinical tasks using a checklist derived from the MEASURE Evaluation's Quick Investigation of Quality (QIQ) tool [12]. Because it was not always possible to observe a real client visit (the total fertility rate in Armenia is a low 1.7 live births/woman – DHS 2000) [13], providers were sometimes asked to simulate a situation with a second interviewer standing in for the client (analyses will take this into account). After the provider was observed, she was interviewed about her work environment and the other performance factors.

Using the StatCalc feature of Epi Info 6.04, the desired sample size was estimated to be 300 based on a total provider population of 3000; an expected frequency of 40% (average performance for clinical skills); a worst acceptable error of 15%; a confidence level of 95%; and a 10% margin for loss of data. Given the high numbers required, all nurse-midwives working at outpatient facilities in both Lori and Shirak regions were selected for interview. The final number of providers observed and interviewed was 285.

A total of 11 two-person teams, an observer (physician) and an interviewer (non-clinician), collected data in the field. The clinician carried out the observations while the non-clinicians administered the performance factors questionnaires. Data collectors participated in a weeklong training on proper methods of completing questionnaires and checklists. Study instruments had been reviewed by local clinical experts, translated into Armenian, and pre-tested twice for consistency and comprehensibility.

Field work took place simultaneously in Lori and Shirak regions. An average of two providers was observed/ interviewed each day per data collector team spending a total of 13 working days to complete observations/interviews during August 2002. Some providers, especially at the FAP level, were visited multiple times to complete the interviews, as they were not found at their post. Supervisors ensured correct application and completion of instruments and a field coordinator organized all logistics and performed quality control of the field work.

The average performance score for each provider was compared to the dichotomous answers received from the performance factor questionnaire. The resulting significant factors were entered into a stepwise multiple regression model to discover which factor(s) would become better overall predictors of performance. Some background variables, such as clinic type, provider's age and years working at the facility were added to the model.

Among the 19 variables representing all the performance factors, 11 were found to have a significant relationship to performance (see Table 3). The most significant are:

Appropriate feedback does not reach significance by close margin (p = 0.047 and p = 0.05) while the more direct question on whether the provider receives feedback about his/her performance was not found to be influential.

Whether the provider received supervision in the last 6 months was related to performance only in the area of prenatal care. The kind of supervision, on the other hand, produced some small differences only in the area of prenatal care, which did not reach statistical significance.

Variables significantly associated with performance at the bivariate level plus available background characteristics were selected as independent variables in multiple regression analyses of performance (the dependent variable) – see Table 2. Analyses were done separately for prenatal and postpartum care performance.

The first important finding of the study is the statistical significance of relationships between performance and the performance factors. On the one-to-one relationships, 11 of the 21 variables tested had a significant effect on performance and more importantly, 7 of the 11 variables were the same for both clinical areas. They are:

Although further studies of this nature are needed to confirm the relative importance of factors, in this study it is interesting to find that a number of aspects in the providers' work environment (as perceived by them) do correlate with observed performance. The multivariate analysis further determines a subset of factors and variables that seem more critical to performance. For example, results seem to indicate is not so much the theoretical knowledge but the practical application of skills in the use of everyday clinic tools that associates with improved performance. This finding seems to reinforce the need for 'essential learning' approaches for improved performance that several training organizations, including IntraHealth, are developing for use in developing countries.

Another predictor of importance is receiving non-monetary incentives by either providers' employers or the community for whom they work. In a context of economic hardship affecting fair and prompt payment of salaries to health workers such as in Armenia, non-monetary incentives in the form of recognition, in-kind contributions, community respect and assistance with services can become powerful motivators to enhance performance. The difference between employer's and community incentives between prenatal and postpartum care seems to indicate where these services occur mostly, prenatal care facilities level and postpartum care at the community level.

For postpartum care, having a performance review from a supervisor as a predictor of performance suggests that this mechanism is most important when people work in less structured environments, often working away from the clinic or the office.

This study is largely of exploratory nature. As mentioned, there have been no empiric studies that we know of combining perceived performance factors and observed performance in the field of FP/RH in low-resource settings. Hence, interpretation of findings needs to be done with caution. Several caveats to the study should be mentioned.

First, the cross-sectional nature of the study, and corresponding associations found, do not imply causal relationships between the factors and provider performance. Though questions related to performance factors were asked with a retrospective connotation (e.g. "in the past 6 months, have you received any supervisory visit?"), memory recall and rationalization may curtail providers' recollection of factors affecting their performance. The multivariate analyses has been carried out by merging data from two different exercises and data collectors, thus arguably adding variation (and a degree of error) to the usual limitations.

Another aspect worth mentioning is the inherent challenges in the application of the factors questionnaire by interviewers. Though pre-tested and improved through pilot testing, several concepts with heavy Western connotations (e.g. what providers answered as "feedback" seems to have been other types of interaction).

We have also used a "proxy" of performance, through the observation of skills. There are other components of performance (e.g. accomplishments) not included in the study. The low R square rates obtained implies our model has captured only a small percentage of the total variation, suggesting that there might be several other factors associated with performance or perhaps better instruments/methods to ascertain it.

In addition, there is an added bias associated with the "Hawthorne effect" in both direct observation and "hypothetical client" approaches to measuring provider performance. Although these approaches are widely recognized as having inherent limitations, both are considered among the best methods for assessing a provider's clinical behavior in a truly technical and systematic way [14, 15]. One such study, however, directly stated that the data collected through direct observation using the QIQ tool, which was utilized in the study, is as reliable as alternative methods for data collection, such as client exit interviews and "mystery clients" [16]. For our purposes, however, where we were assessing a provider's technical performance, direct observation was key for obtaining reliable data.

Furthermore, there is a perceived loss of complexity within the provider-client interaction when a "hypothetical client" is used in lieu of a normal client, as was used in some of the data collection. Case simulation might not replicate entirely the quality and completeness of the normal exchange occurring during direct observation of a provider with a real client, thus producing lower performance scores and possibly affecting the appearance of performance factors in unknown ways [17]. However, once again, "hypothetical clients" have been widely recognized as a reliable and systematic way to assess technical competency when other alternatives are not viable due to low case load for years [18]. In our case, though unavoidable, differences did appear in our measures of performance for postpartum care between real and hypothetical scenarios.

In all, this study has demonstrated the feasibility of the model. As applied, performance was associated mostly with workers' perceptions of having the practical knowledge of tools existing in the facility and of being acknowledged by employers or clients, two important factors to have in mind when considering quality and productivity of work. Further refinement of observation and interview tools are needed. Replication of this study (planned for Nigeria and Bolivia) should shed light about the relative importance of factors in other contexts. Their results can assist development efforts involving human resources by focusing on priority areas, and save valuable financial resources.