Validity of a questionnaire measuring the world health organization concept of health system responsiveness with respect to perinatal services in the dutch obstetric care system

The debate on the organization of perinatal care in the Netherlands has intensified over recent years. The Dutch perinatal health care system has come under pressure since the national perinatal mortality rates were shown to be among the highest in Europe [1]. This system can be regarded as a sequential chain of health care services, each dedicated to a different phase of the perinatal experience: antenatal care, birth and post partum care. Antenatal, birth-related and post partum care are provided by different caregivers with different responsibilities, for different risk groups, and in different settings. In the Netherlands independently operating community midwives provide care for low-risk pregnant women (primary healthcare) while gynecologists provide in-hospital care for high-risk women (secondary care). All women receive post partum care by a community midwife.

The performance of perinatal care is often judged by its endpoints such as clinical outcomes and costs. However, quality of care literature supports the view that non-clinical aspects of quality are important too, and affect clinical outcomes [2]-[4]. Better service quality is thought to increase compliance with medical treatment and to improve the transfer of information and appropriate utilization of health care [5]-[8]. Governments of Western countries increasingly acknowledge the importance of the non-clinical aspects of quality of care and incorporate these when the provision of care is monitored [9],[10].

Sofar, no attempts have been made to evaluate the non-clinical aspects of the Dutch perinatal care system such that not only the heterogeneity in the quality with respect to different perinatal services is identified, but also that international comparisons with other obstetric care systems are possible [11]-[13]. The concept of responsiveness, introduced by the World Health Organization (WHO) in 2000, seems apt to this task as it was specifically developed to refer to patients’ experiences when interacting with health care providers. The concept reviews eight predefined domains relevant to non-clinical aspects of service quality regardless of provider, country, health system or health condition. Responsiveness is defined as “aspects related to the way individuals are treated and the environment in which they are treated during health system interactions”, encompassing the notions of both non-clinical quality and patient experience [14]. The concept excludes the financial and clinical domains of quality and focuses on a set of non-clinical domains that reflect respect for human dignity and the client orientation of the care process and setting. While these domains may influence health outcomes, health outcomes are not part of the responsiveness concept. The relevance of an independent set of non-clinical domains to health systems performance is supported by the discipline of medical ethics and in human rights law, which argue that responsiveness features of a health system are important in their own right [14]-[16]. The concept of responsiveness aims to support measurement of service quality in an internationally comparable way and to enable quantitative trade-offs between non-clinical aspects of service quality and clinical outcomes [14]. The concept of responsiveness aims to capture information on the non-clinical quality of the patient’s actual experience in contrast to patient satisfaction questionnaires. Literature has shown that expectations may strongly influence patient satisfaction, which makes international comparisons of non-clinical service quality challenging since expectations are in turn influenced by economic and political influences [17]-[20].

Adopting the responsiveness concept, the Responsiveness in Perinatal and Obstetric Health Care Questionnaire questionnaire (ReproQ) was developed by adapting the existing generic World Health Survey questionnaire responsiveness module into a questionnaire dedicated to maternal experiences during perinatal care. The aim of this study is to investigate the psychometric properties of the ReproQ.

Of a total of 274 women who were identified for study participation 94 women could not be reached or they declined the invitation; many women could not be reached using the cell phone number they had provided; we were unable to differentiate with limited means whether they refused the call, changed phone number, or had provided the wrong number. Other reasons for non-participation included lack of time, and feeling at unease of having a stranger visit their home. 180 women (66%) agreed to be interviewed. Of these seven interviews (7/180, 4%) were cancelled and two interviews (2/180, 1%) were discontinued because the respondents did not speak Dutch with sufficient fluency and no translator could be made available. The remaining 171 interviews were used for analysis. The interviews took between 20 and 40 minutes. Table 2 describes the characteristics of the participants.

The mean maternal age was 31 years (95% CI 30.3–31.7). The majority of mothers were primiparous (57%). A substantial proportion of mothers was of non-Dutch origin (43%) or lived in underprivileged neighborhoods (51%). Few had low education (4%) or were single (18%). Approximately 11% spoke weak/poor Dutch as judged by the interviewer. Referral to gynecologists had occurred in approximately 55% of women. Post partum hospital admission had occurred in 26 (15%) of all newborns.

Table 3 describes the missing rates per domain for each phase separately. The table also describes the maximum missing rate per item for that domain. The results for four women, with no response in the birth care phase, were excluded.

The average item missing rate over all phases was 8% (1,349 out of 17,624 questions). Missing rates per domain were all below the predefined threshold of 20%. Average missing rates across domains were highest in the birth phase (8%). Maximum item missing rates per domain ranged from 1.8% to 11.1% for the antenatal phase, from 5.3% to 31.6% for the birth phase, and from 5.3% to 14.6% for the post partum phase (see Additional file 1: Table S6 Appendix table for detailed description of all items). The highest item missing rate was for two questions relevant to the birth phase: ‘Able to be referred to a medical specialist during birth care’ (31.6%) and ‘Consideration of home situation when planning appointments/examinations during birth care’ (22.8%). Item missing pertained mainly to the birth care phase and rates were higher among women of Dutch origin. There were no differences in missing rates by age, educational level and health utilization.

Table 4 displays the transformed scores per domain and phase (1–10 scale). Mean transformed scores were negatively skewed (7.1–8.4) as were the median scores (7.2-7.8). Floor effects were observed for up to 0.6% of women responding to a set of items in a particular domain for a particular phase, while ceiling effects were observed for up to 24% of cases. Mean scores and ceiling effects differed most across the domains in the antenatal phase and least across the domains in the post partum phase.

The Cronbach’s alpha ranged from 0.57-0.82 for the antenatal phase, from 0.66-0.92 for the birth care phase and from 0.62-0.95 for the post partum phase. For all phases the domain ‘Quality of Basic Amenities’ had lowest alphas.

Mean overall sum scores were higher for women whose child was not admitted after childbirth: 61.8 (sd 7.4) versus 58.3 (sd 5.1) (p = 0.02) in the antenatal phase; 61.9 (sd 8.4) versus 57.9 (sd 7.7) (p = 0.06) in the birth phase; and 62.1 (sd 9.2) versus 55.2 (sd 13.0) (p = 0.01) in the post partum phase.

Eight factors corresponding to the domain structure of the WHO responsiveness concept explained 69% of the variance in the antenatal phase, 69% in the birth phase and 76% in the post partum phase. Table 5 shows the final results of the oblique promax rotated factor loadings of the birth phase (the patterns of the antenatal and post partum phase were similar). Items that were expected to belong to one domain are outlined. The rotated solution of grouped items generally confirmed the hypothesized domain taxonomy within the birth and post partum phase. For the antenatal phase however, the hypothesized domain taxonomy was less evident with regard to ‘Social Consideration’ and ‘Choice and Continuity’, which appeared to be associated with other domains.

With the support of both patients and health care providers, we adapted the WHO’s concept of responsiveness and the World Health Survey responsiveness module into the ReproQ instrument to measure responsiveness in the Dutch obstetric care system antenatally, during childbirth and post partum. ReproQ was administrated in a face-to-face interview context and appears to be a potential instrument for reporting perinatal service quality from the client's perspective. The perinatal responsiveness items grouped in the original eight domain based structure found in the World Health Survey and the World Health Survey questionnaire and appeared to be comprehensive, as judged by the experts. The ReproQ demonstrated satisfactory psychometric properties to describe the responsiveness outcomes of perinatal care in the Netherlands, with preliminary evidence on the questionnaire’s ability to discriminate between levels of non-clinical quality of care.

Particular strengths of adapting an existing WHO responsiveness concept and measurement approach are noted first. The eight domains were adopted a pre-existent conceptual structure that was identified in WHO’s review of the patient satisfaction and quality of care literature, which also included the examination of different survey instruments [15]. During this review, it was noted that the domains value is supported by human rights law which argues that the responsiveness features of a health system are important in their own right [14]-[16]. In contrast to patient satisfaction questionnaires, the concept of responsiveness aims to capture the patient’s actual experiences, since literature has shown that expectations strongly influence patient satisfaction. Expectations may be influenced by economic influences, political influences, prior experiences and socio-demographic characteristics [17]-[20]. Fourthly, the WHO concept of responsiveness represents an universal concept (e.g. suitable for developing and developed countries, different ethnicities, different care systems) which allows valid comparisons across different countries, ethnicities or health care systems [14],[28]. The Responsiveness concept is challenged by a number of issues. Firstly, although responsiveness aims to measure the patient’s actual experience, it is still disturbed by at least some extent of ‘subjectivity’. Secondly, capturing responsiveness by a limited number of questions with fixed answering categories is quite challenging. Combining qualitative research and different (quantitative) survey techniques, one can produce a richer, more valid, and more reliable findings than when adopting qualitative or quantitative methods alone [29].

In spite of the existing strengths of the Responsiveness concept and measurement approach, our study contributes to addressing some of the challenges of the Responsiveness concept and its measurement approach. This includes whether it can truly be adapted to specific areas of health systems, like perinatal care, and elicit participation from specific groups of user interacting with specific health service. In particular, we found that users of perinatal services were interested in participating in the survey on non-clinical aspects of their care experience. Participation rates were equal or higher than the participation rates found in other perinatal satisfaction studies [30]-[32]. Participation rates were equal to participation rates found in surveys measuring similar domains of quality of care, and better than obtained by WHO’s Multi-Country Survey (MCS) study administered in the Netherlands in 2001 (59%) [21],[25],[33]. Comparisons are made with the MCS Study that was conducted in the Netherlands in 2001 as the questionnaire contained multiple items for each responsiveness domain, whereas the subsequent World Health Survey only contained one question per domain [14],[21].

An optimal data collection method includes one with an explicit trade-off balance between cost and errors including nonsampling error, coverage error, nonresponse error and measurement error [34]. To ensure data quality we chose face-to-face interviews with an independent interviewer for data collection. Compared to self-administered forms face-to-face interviews perform better in terms of non-sampling and non-response error but may perform worse when sensitive questions are asked and are more costly [35]. Internet or web surveys are less costly and more time efficient but also have limitations especially including coverage error [36],[37]. Mixed-mode approaches, combining the best of both worlds (being less costly and having less error than in a unimode approach) are very promising and should be considered [38].

The average item missing rates across domains was 8%, which according to literature can be considered acceptable. Within the framework of the MCS study, slightly lower overall missing rate was reported (5.0%) [39]. However, our survey dealt with a group of women who were extremely occupied with the challenging demands of new life, being interviewed post partum. Our survey focused on three phases of a specific health event, which may have been more cognitively demanding than the MCS study, which focused on reporting on an average experience in the previous 12 months, and was shorter (on average 25 minutes) [32],[40],[41]. As found in the MCS, the domain missing rate was highest for the domains of ‘Autonomy’ and ‘Choice and Continuity’ which are typically cognitively demanding domains. Across phases of perinatal care, the missing rate was highest for birth phase. But in general we found the proportion of missing rates per item to be similar across items. There were two items in the birth phase that had notably higher missing rates. Most likely this is the consequence of these items pointing to service events that do not always taking place. For this paper we excluded them from the analysis. We do not feel this hindered our ability to test the ReproQ psychometric properties, but have noted difficulties with these items for future surveys. Alternatively, when not all women experience all the events that can occur, different responsiveness scores may be presented for certain service events that occurred as well as in absence of those events.

The transformed scale scores were satisfactory. A floor effect was almost absent as is frequently the case in negatively skewed assessments of self-reported health or self-rated experiences of (maternity) care [30]-[32],[42]. There was surprisingly less skewing towards use of the most positive category (ceiling effects) compared to other surveys e.g. in the MCS [32],[40],[41]. Comparisons of the domain scores across the three phases showed a non-uniform pattern, suggesting that respondents judged each phase separately as was intended by the questionnaire design.

Within each phase and for all domains, the questionnaire’s internal consistency was good. Cronbach’s Alpha coefficients in ReproQ were similar compared to the CAHPS and WHO surveys [21],[25], except for the domain ‘Quality of Basic Amenities’ which showed poor alphas in all phases. This domain contained questions about sanitary hygiene, comfort of waiting room and quality of food. It can be argued that these elements of basic amenities were too diverse to achieve internal consistency (see Table 5) and one might improve reporting of results from the questionnaire by analyzing these items separately. The Pregnancy and Childbirth questionnaire (PSQ) covering personal treatment patient satisfaction outcomes for the antenatal and birth phase [43] showed higher Cronbach’s Alpha coefficients for the antenatal phase (0.89 vs.0.73) and for the birth phase (0.86 vs. 0.84). However, no predefined domain structure was used.

Overall, the taxonomy of domains from the WHO concept and measurement approach held for the adapted items in the ReproQ across all phases, although this taxonomy was weaker in the antenatal phase. This could possibly be due to factors such as; recall bias introduced by assessing all phases together, contamination by pregnancy outcome, focusing on one particular event or the heterogeneity in measurements since antenatal care consists of multiple visits. Underlying patterns are still to be explored. One may consider presenting a questionnaire on the antenatal phase within the antenatal phase, separately from a questionnaire on the birth and post partum phase. The total explained variance for the birth phase was higher in our study compared to the PCQ [43] (69% vs. 56%) as for the antenatal phase (69% vs. 53%).

The ability of the instrument to discriminate between good and less good experiences will be of paramount importance for its future use. We found some promising test results. The respondents clearly expressed different opinions on their experiences in the different phases of perinatal care. The non-uniform pattern of domain scores across the three phases suggested that respondents judged each phase separately as was intended by the questionnaire design. Furthermore discrimination between women whose infants were admitted to hospital subsequent to birth, was reflected in the lower sum scores across all phases. However, to test the difference in mean responsiveness of the birth phase between the mothers whose infant was hospitalised and the mothers whose infants were not hospitalised (mean difference: 3.8, pooled SD 6.5), at least 194 mothers had to be included in the analysis (type I error = 0.05 (two-sided), power = 0.80, control/case-ratio: 6/1). This implies that different responses on antenatal sum scores may reflect a true outcome on non-clinical aspects of care or may be contaminated by pregnancy outcomes. This again stresses the need to present a questionnaire on the antenatal phase separately.

Test-retest reliability was not performed in this stage. Reasons were to avoid the already large burden for the participants and to avoid associated potential recall bias effects due to having at this time a too demanding interview.

Overall, our study found that ReproQ demonstrated satisfactory psychometric properties to describe the responsiveness outcomes of perinatal care in the Netherlands, with preliminary evidence supporting the questionnaire’s ability to discriminate between levels of non-clinical quality of care.’ In general, psychometric properties were in line with results obtained for other survey instruments that have been tested and promoted as part of quality assessment effort. In conclusion, given the lack of comparable instruments and the overall favorable study results, we feel that this unique adaptation of the WHO responsiveness questionnaire to evaluate the various phases of perinatal care has been relatively successful. With some minor adaptations as suggested throughout the discussion we believe that this questionnaire can be used to evaluate the quality of perinatal care in the Netherlands and elsewhere.