Patient satisfaction in pediatric outpatient settings from the parents’ perspective - The Child ZAP: A psychometrically validated standardized questionnaire

The goal of the present study was to adapt two existing questionnaires to a survey of parents whose children are seen in pediatric and adolescent primary care practices: the ZAP questionnaire and the questionnaire of parent satisfaction with hospital care [14]). The notion to adapt the instrument to a children and adolescent self-report was abandoned because in German pediatric and adolescent primary care practices more than 50% of the patients are younger than 5 years of age [17], and thus unable to complete a written survey [18, 19]. Instead, we decided to evaluate parents' assessment of health care quality and satisfaction from their child's perspective (proxy report). However, although children and adolescents are the main recipients of care, physicians still need to address and involve the child’s parents. Therefore, it seems necessary to obtain the assessment of satisfaction and involvement with pediatric care reported by the parents.

The parent survey was designed to assess aspects of both the child-physician and the parent-physician relationship. Byczkowski et al. (2010) found a large degree of corresponding assessments between adolescent patients and their parents regarding patient satisfaction. Further, we decided against the use of different versions for various age groups to ensure that the questionnaire can be widely used for parents of children of all ages in every day pediatric outpatient care [20].

The Child-ZAP was developed from the validated ZAP questionnaire for adults [13] which contains four scales (23 items), and from a questionnaire of parent satisfaction with hospital care [14]. A preliminary qualitative study was conducted to assess the comprehensibility and relevance of questions of the adult version of the ZAP. This was done by conducting guided interviews with parents whose children were currently seen in pediatric and adolescent primary care practices as well as with older children and adolescents. Based on the results of the preliminary study and theoretical considerations, the adult version of the ZAP was modified into a preliminary version for children (Child ZAP).

The preliminary version of the Child ZAP contained six scales (24 items) for parent assessment of parent-physician interaction (self-report), three scales (14 items) for parent assessment of child-physician interaction (proxy report), and three global items (overall satisfaction with the physician, trust in the physician, and quality of care).

The preliminary version was then subjected to psychometric testing and modified accordingly into a second and final version of the Child ZAP, which then was tested in the main study with a random sample from selected pediatric and adolescent primary care practices. The results of the psychometric analysis, based on data gathered in the main study, are presented below.

The parents were instructed to answer the questions based on their overall satisfaction with the pediatrician in general, i.e., not merely based on the last doctor visit. This was included to ensure that the participants responded independent of their current doctor’s appointment. All items were rated on a four-point scale ranging from "very satisfied" to "very unsatisfied". Two scales, the proxy report scales "Child - Information" and "Child - Involvement" were coded with an additional response "Does not apply". The rationale behind this decision was the assumption that it would not always be possible or appropriate to provide information to very young children or involve them in decision making.

The questionnaire also contained supplementary questions on the current doctor visit and on socio-demographic characteristics of the parents and their children.

We decided to test the instrument in 20 practices in Northern Germany. In collaboration with the German Association of Pediatric and Adolescent Physicians (Berufsverband der Kinder- und Jugendärzte e.V.), all respective physicians in Lower Saxony, Bremen and Hamburg (n = 750) were invited to take part in the study. Of 222 practices who responded positively, 20 practices were randomly selected.

No specific inclusion criteria were used for recruiting parents. However staff members were instructed to ensure that the participants were sufficiently fluent in German. Each practice received 100 questionnaires (including a letter and a stamped return envelope each), which were distributed randomly. The staff members instructed the parents to fill out the questionnaire at home and to return the questionnaire anonymously to the Hannover Medical School (MHH). The data collection for the preliminary study took place in winter 2007/2008, and for the main study in May 2009.

Participating doctors and patients were informed about the study. The Ethics Committee of the MHH approved the study (No 379, 2008).

To identify potential methodological weaknesses of the instrument and to reduce the number of items if possible, the preliminary study data collected in 19 pediatric practices were used to perform item and factor analyses considering wording quality criteria (item wordings related to quality management should be as specific and detailed as possible) and statistical criteria (including factor loadings of <0.5 for the respective factor and correlation of <0.8 with the respective subscale; item difficulty defined as the percentage of parents correctly answering the item.). The factorial structure of the preliminary version revealed high construct validity. However, considering the above mentioned criteria 5 items were removed from the questionnaire in order to reduce its length.

The optimized instrument consisted of three scales (14 items) for parent assessment of the child's interaction, information and involvement in decision making and five scales (19 items) for assessment of the parents' own experience (with practice organization, practice facilities, the provision of information, involvement in decision making, and professional competence).

The questionnaire items are shown in Table 1 (child scales) and Table 2 (parent scales).

The final psychometric analysis consisted of item analyses (e.g., discriminatory power, distribution and item difficulty) and tests of reliability and validity of the scales/subscales. Reliability was measured using a standardized coefficient of internal consistency (Cronbach's alpha). Construct validity was assessed by confirmatory factor analysis, which has been shown to be an adequate method for testing theoretically assumed factor structures of multidimensional scales [21, 22]. Two confirmatory factor analyses were performed: the first analysis consisted of the 19 items which exclusively covered the assessment of the parents' own experiences; the second consisted of the 14 items for parent assessment regarding the relationship between child and physician. Included into the analyses were all cases with a valid response to all items needed for the respective confirmatory analysis. The Maximum Likelihood method was used to estimate the parameters, a procedure used if a sufficient sample size is available and the Bollen-Stine bootstrap is relatively insensitive to a violation of the normal distribution assumption [22, 23].

To determine whether the measurement model was appropriate for both younger and older children, the model previously defined for the overall sample was analysed in two parallel groups: children up to age 6 and children older than 6 years of age. Three models were compared with the unconstrained baseline model.

Model 1 referred to the assumption that there would be no statistically significant differences in any of the estimated parameters within the individual subsamples.

Model 2 was based on the assumption that the parameter estimates would differ only in terms of the error variance and that there would be no statistically significant differences between both subsamples in terms of the regression coefficients of the latent constructs for the observed individual items and of the inter-correlation of the latent constructs.

The third model implied that there would be no statistically significant differences in the regression coefficients of the latent constructs of the observed individual items in both subsamples.

Stability of the measurement model is not given unless the chi-square statistic of the model does not differ significantly from that of the unconstrained model [24].

The test of model quality was performed using the chi-square statistic, the comparative fit index (CFI), the Tucker-Lewis index (TLI), the Root Mean Square Error of Approximation (RMSEA), and the Standardized Root Mean Square Residual (SRMR). Model adaptation was determined to be acceptable based on the following criteria: non-statistically significant difference in chi-square statistic, high CFI (> 0.95), high TLI (> 0.95), low RMSEA (< 0.06), and low SRMR (< 0.11) [25‐29].

An initial assessment of the convergent reliability was obtained by calculating correlations (Spearman correlation coefficient) between the subscales and the global items for satisfaction. We compared subscale means along different categories of wait times and consultation length by one way analysis of variance.

All statistical analyses were performed using SPSS for Windows V.17 and AMOS17 software.

A total of 979 valid questionnaires were available for analysis (range of responses by pediatric practice: 21% to 72%; median: 50%). More than 90% of the questionnaires were completed by the mothers (see Table 3). The mean age of respondents was 35.4 years (SD: 6.6; range: 41), and the mean age of children treated in the pediatric and adolescent practices was 4.7 years (SD: 4.0; range: 16). The most frequent reasons for consultation were routine check-ups, vaccinations and acute disease symptoms.

Missing values ranged from 0.2% to 46% (Table 4; Table 5). All item difficulty values were above the theoretical scale midpoint; the mean values are 0.81 (child scales) and 0.79 (adult scales). The distributions of responses at item level were asymmetrical, with the majority of responses in the upper quarter of the response scales (Table 4; Table 5). The discriminatory power coefficients [30] were all significantly higher than r = 0.40.

The items of each dimension were summarized into subscales. High subscale scores reflect a high degree of satisfaction. In addition, each subscale was transformed into scales ranging from 0 to 100 (Table 6; Table 7).

All subscales showed that the distribution of responses was skewed to the left (subscale median values in the upper third, negative skewness and ceiling effects). This tendency was most pronounced in the subscales "Child – Interaction" and "Child - Information". Ceiling effects were much less pronounced in all other scales. The internal consistency of the child-subscales was acceptable to good [31] with Cronbach's alpha ranging from 0.72 to 0.95.

A confirmatory factor analysis of the dimensionality of the construct "patient satisfaction", as derived from the preliminary study data, was conducted using main study data (the parent and child scales of the Child ZAP were tested separately). The regression weights of the individual items of the respective scales on the latent factors, the squared multiple correlations of the individual item, and the correlations of the latent factors with each other are presented in ( Additional file 1: Figure S1) and ( Additional file 2: Figure S2). The parameters used to assess the quality of the overall model are specified.

Multiple criteria should be used for assessing the confirmatory factor analyses [21, 22].

Finally, different indicators of the overall quality of the model should be used.

As shown in the excerpts of the model estimates for the two models ( Additional file 1: Figure S1) and ( Additional file 2: Figure S2), all parameter estimates of the individual items exhibited positive correlations with the latent variables, and none of the parameter estimates was implausible. The correlations of the observed variables with the respective latent dimensions were high. The squared multiple correlations for the child scales ranged from 0.62 to 0.87 and were therefore within an acceptable range. The individual items of the parent scales were explained by the latent factors to a lesser degree; most of the squared multiple regression values ranged from 0.6 to 0.7, but some values were lower.

As expected, the correlation between the latent factors was high, which implies that the individual dimensions of satisfaction with the pediatric and adolescent primary care physicians were not independent of each other. There were strong correlations between the three domains information, involvement in decision making and professional competence covered in the child scales and the parent scales covered in the child scales, and further between parent scales for information, involvement in decision making and professional competence.

The indicators for assessing the quality of the overall models were acceptable to good for both the child scales and the adult scales of the Child ZAP. Initial variance can be largely attributed to the model (CFI 0.95; TLI 0.95). Conversely, the proportion of residual variance not explained by the model were low (RMSEA 0.6/046 and SRMR 0.038/0.049). Thus, the values were within the recommended range. The chi-square statistic of the two models was highly significant. However, considering the large sample size, this should not be overrated [22].

The results of the test of stability of the measurement model are shown in Table 8. All compared models (i.e., those for both the child scales and the parent scales) featured comparable quality indices in the different subsamples of older and younger children. Regarding the child scales, in both samples with older and younger children, the latent factors were comparable and not significantly different in terms of explaining the observed individual items (as shown by the non-significant p-value for the chi-square statistic as compared to Model 3 with the unrestricted baseline model). Regarding the parent scales, the model with assumed non-statistically significant regression weights also exhibited a slightly significant difference from the unrestricted baseline model. Therefore, the measurement model and the assumed theoretical structure of the child scales can be replicated in different subgroups. The same applies to the parent scales, however, to a slightly lower extent only.

All subscales featured a positive and statistically significant correlation with the three global items overall satisfaction with the pediatrician, quality of care, and trust in the physician (Table 9). The subscales "interaction", "information" and "professional competence" showed a satisfying correlation with the three global items (correlation coefficient range: 0.55 to 0.63), whereas the subscale "practice organization" correlated slightly lower with the global items (correlation coefficient range: 0.39 to 0.41).

As shown in ( Additional file 3: Figure S3), a wait time of more than 30 minutes was associated with significantly lower satisfaction (10 points less than average) for all investigated aspects. The highest decrease was observed for satisfaction with practice facilities and practice organization (−17 and −13 points, respectively). A patient-physician contact shorter than 10 minutes was also associated with lower satisfaction ( Additional file 4: Figure S4). Short contact times with the physician had the highest effect on satisfaction with "Child – Involvement in decision making" and "Parent – Information" (−12 and - 11 points, respectively) and the lowest effect on satisfaction with patient organization and practice facilities (−7 and −8 points, respectively). All differences were statistically significant on a 5% level. Interestingly, a very short wait time (less than ten minutes) as well as very long physician contact time (more than fifteen minutes) were also associated with lower satisfaction. These findings are consistent with findings from the validation study of the original ZAP questionnaire.

The construct validity of the Child ZAP is supported by the confirmatory factor analyses which were plausible in content and offer adequate results according to statistical criteria. The child and parent scales of the Child ZAP give a good representation of the multidimensional and multi-perspective construct of patient satisfaction with process quality in pediatric and adolescent primary care practices. The inter-correlations of the subscales, which were sometimes rather high, could be interpreted as a sign of redundancy in the instrument, leading to considerations to shorten the instrument or the individual subscales further. However, due to content related reasons (to obtain a wide diversity of potential aspects for quality management) and statistical reasons, it was decided not to shorten the tool. Moreover, the Child-ZAP was not reduced any further, because it is designed for quality assurance in outpatient pediatric practices. Thus, not only the scale scores, but also the results at single item level is of interest and important, for example, for the scale "professional competence", which consists of the items "collaboration with other medical faculties", "thoroughness and diligence in examinations" and "readiness to refer the child." For this reason and due to specific requirements in outpatient pediatric practice the child-ZAP is significantly longer than the original adult ZAP.

Although the ceiling effects were pronounced and corresponded to the "high" patient satisfaction observed in other studies [32, 33], they were lower than those obtained from the ZAP questionnaire for adults [34]. For example, in the validation study for the adult ZAP the subscales "practice organization" and "professional competence" received maximum scores from more than 40% of persons surveyed, whereas in this study these subscales received a maximum score of only 27.2% and 36.4%, respectively. Nevertheless, the skewness of distribution and the pronounced ceiling effects limit an interpretation of the results.

Further, it has to be noted that the participating practices and parents were not randomly selected, and not representative of the population. Although we provided an instruction for this random distribution, it was not possible to control that this was done correctly. Thus, a selection bias can not be ruled out. It would be desirable to validate the questionnaire in a population-based sample.

The relatively high percentage of missing values for the parent scale "professional competence" appears to be problematic. However, the two items on the "professional competence" scale that resulted in the high number of missing responses concerned the parent's opinion of the pediatrician's collaboration with other medical facilities and the pediatrician's readiness to refer the child. Thus, it can be assumed that many parents who did not provide a valid answer to these items did not have any experience regarding the physician's readiness to refer the patient or collaborate with other facilities, and are not able to answer these questions. As the probability of parents having such an experience presumably increases with their child’s age, missing responses for these questions are most likely for parents of younger children. Post-hoc analyses of our data showed, that this assumption is supported by the decrease of percentage of missing data with increasing age of the child (data not shown).

Basically it can be assumed that the pediatric and adolescent primary care physicians were motivated to participate in the study. The response rate of nearly 50% is comparable to similar patient surveys in German private practices. [12, 35, 36].

When assessing the process quality from the patient’s point of view in pediatric and adolescent primary care rather than in adult health care, both the parents’ as well as the child’s perspective must be included. Therefore, the notion of "patient satisfaction in pediatric and adolescent primary care" is more complex than in adult primary care. This was taken into account while developing questions and scales which explicitly relate to the interaction between the physician and the child as well as to experiences and opinions of the accompanying parent.

The inclusion of the patient partnership in decision making as a component of health care process quality assessments, particularly patient satisfaction surveys, is a recent development [10, 11], and failure to do so is seen as a methodological deficiency [37, 38]. Therefore, the Child ZAP questionnaire developers decided to include items and subscales to measure child and parent involvement in decision making even though it was known that this would lead to a high rate of missing data, particularly in cases where the patients were very young children. Considering that the Child ZAP is designed to apply to children of all ages, content-related considerations and the stability of the measurement models in both younger and older children support the decision to include the scale in the Child ZAP. However, it is recommended to restrict the analysis of this scale to children 5 to 6 years of age and older.

The fact that in a questionnaire used in outpatient pediatric practice, only the parents were interviewed, may at first seem strange. However, given the fact that more than half of the target population can not be interviewed directly due to their age, this decision appears to be justified. In addition, studies in which self-assessments of children and assessments by their parents are compared show a high correlation between these two measures [39‐41].