Criterion and content validity of a novel structured haggling contingent valuation question format versus the bidding game and binary with follow-up format

Abstract

Contingent valuation question formats that will be used to elicit willingness to pay for goods and services need to be relevant to the area they will be used in order for responses to be valid. A novel contingent valuation question format called the “structured haggling technique” (SH) that resembles the bargaining system in Nigerian markets was designed and its criterion and content validity compared with those of the bidding game (BG) and binary-with-follow-up (BWFU) technique. This was achieved by determining the willingness to pay (WTP) for insecticide-treated nets (ITNs) in Southeast Nigeria. Content validity was determined through observation of actual trading of untreated nets together with interviews with sellers and consumers. Criterion validity was determined by comparing stated and actual WTP. Stated WTP was determined using a questionnaire administered to 810 household heads and actual WTP was determined by offering the nets for sale to all respondents one month later. The phi (correlation) coefficient was used to compare criterion validity across question formats. The phi coefficients were SH (0.60: 95% C.I. 0.50–0.71), BG (0.42: 95% C.I. 0.29–0.54) and the BWFU (0.32: 95% C.I. 0.20–0.44), implying that the BG and SH had similar levels of criterion-validity while the BWFU was the least criterion-valid. However, the SH was the most content-valid. It is necessary to validate the findings in other areas where haggling is common. Future studies should establish the content validity of question formats in the contexts in which they will be used before administering questionnaires.

Introduction

Willingness to pay (WTP) elicited through the contingent valuation method (CVM) is increasingly being used to measure the benefits of goods and services and for market research in the health-care sector. The theoretical underpinning of the CVM is the measurement of consumer surplus using the Kaldor-Hicks Pareto-improvement criterion in welfare economics (Donaldson, 1993). Consumer surplus is the area below the demand curve but above the market price (Dinwiddy & Teal, 1996). Maximum WTP is the amount of money, which the consumer is willing to give up to receive the service such that they are as well off as they were without the service. The CVM is the theoretically correct technique for valuing the maximum amount the gainers are willing to pay or the minimum amount the losers are willing to accept (Drummond, O’Brien, Stoddart, & Torrance, 1997).

Hence, the CVM is a direct measure of value. According to proponents of the CVM, asking people directly has the potential to inform about the nature, depth, and economic significance of these values (Mathiyazhagan, 1998). Also, the method has an advantage over many of its alternatives, since it is believed to be able to estimate a total economic value, rather than just components of the total value (Frykblom, 1997).

Nevertheless, a long-standing criticism of the CVM is that stated WTP may be a poor indicator of actual WTP (Diamond & Hausman, 1994). Similarly, though one of the aims of WTP is to combine, in one measure, people's valuations of attributes like health gain and quality of care, there is no reason to believe that such implied values will accurately reflect patients’ values or those of the community (Donaldson, 1993). However, the validity of the CVM can be assessed using different tests, but criterion (or criterion related) validity is regarded as offering a more definitive test of CVM results (Asgary & Willis, 1997).

Criterion validity is the correlation of a scale with a gold standard (Streiner & Norman, 1995) and it is the best approach for determining the validity of the CVM (Loomis, Brown, Lucero, & Peterson, 1997). It compares CVM values either with identical markets in which the same good is bought and sold, or with an experimental market which creates or simulates a market in which the good is actually bought and sold (Asgary & Willis, 1997). The most appropriate comparison is between stated WTP and actual WTP, and it is this comparison which can most appropriately determine whether or not the CVM can be used to elicit accurate estimates of WTP.

The degree of correspondence between the test (stated WTP) and the criterion (actual WTP) is usually estimated by the size of their correlation, through a validity coefficient. However, there is no single criterion-related coefficient for determining criterion validity (Carmines & Zeller, 1979). One coefficient that is used to determine criterion validity is the phi coefficient (Streiner & Norman, 1995). Estimates of positive predictive validity can also be used to determine criterion validity, but the phi coefficient is a more sensitive measure since it considers both positive and negative predictions.

There have been very few investigations of criterion validity in the CVM literature, possibly reflecting the difficulty of undertaking such a study. Five CVM studies in the health care sector were identified that directly determined the criterion validity of the CVM (Chestnut, Keller, Lambert, & Rowe, 1996; Dgedge et al., 1999; Rowley, Cham, & Pinder, 1999; Onwujekwe et al., 2001; Blumenschein, Johannesson, Yokoyama, & Freeman, 2001). The levels of criterion validity in these studies ranged from low (Dgedge et al., 1999), to moderate (Rowley et al., 1999) and moderate-to-high (Chestnut et al., 1996; Onwujekwe et al., 2001). Blumenschein et al. (2001) found low levels of criterion validity without calibration of stated WTP, but high levels of criterion validity when stated WTP was calibrated using degree of certainty as the basis for the calibration.

It is possible that valid and reliable question formats for eliciting WTP can help overcome the problem of stated responses not reflecting actual preferences. There are many CVM question formats for eliciting WTP and they have been described in detail by several authors (Mitchell & Carson, 1989; Russell, Fox-Rushby, & Arhin, 1995; Klose, 1999; Smith, 2000; Liljas & Blumenschein, 2000). A review of the different contingent valuation question formats that had been used in health care has been provided (Smith, 2000). However, although the application of the CVM in health care is growing rapidly (Diener, O’Brien, & Gafni, 1998; Klose, 1999), there is still no consensus on the best question format to use in that sector.

In order to improve the validity of the CVM, the choice of a question format to be used in a particular context should be hinged on mimicking the context for purchasing decisions with which the respondents are familiar. Familiarity of respondents with question format will act as a catalyst to activate their usual thinking process for price taking for them to yield valid responses. The use of unfamiliar formats may signal to them that the exercise is just research for research sake and they may thus play along and give invalid responses. Individuals can be expected to “respond differently depending on the question format” (Kealey & Turner, 1993). Furthermore, context-specific question formats will increase the realism of the CVM and thus, have practical relevance to the respondents and make it mentally and psychologically easier for them to state more valid WTP amounts.

Studies that have compared the criterion validity of different question formats have tended to produce mixed results, without favouring any one format. For instance, no significant difference has been found between the open-ended and binary formats (Brown, Champ, Bishop, & McCollum, 1996; Frykblom, 1997; Loomis et al., 1997). Another study found no difference between the criterion validity of the bidding game and the binary-with-follow up technique (Onwujekwe, 2001). Follow-up questions have been proposed as one way to improve the efficiency of the binary question format (Herriges & Shogren, 1996). The binary method improves if a follow-up question is used (Alberini, 1995), but findings of other studies did not support the use of open-ended follow-up question (Kartman, StAlhammar, & Johannesson, 1997).

The inconclusive evidence about the criterion validity of different question formats could arise because the content validity of existing question formats has neither been established in the health care sector, nor possibly in other sectors. Content validity refers to whether the instrument of measurement adequately represents the object of measurement (Klose, 1999). An important aspect of content validity is that the hypothetical contingent valuation scenario presents valuation tasks and choices that are realistic (O’Brien & Gafni, 1996). It also depends on the extent to which an empirical measurement reflects a specific domain of content (Streiner & Norman, 1995). Bowling identified a number of points to be considered in determining content validity (Bowling, 1997).

A gap in knowledge is how context may influence the relative content validity of different question formats, because context-specificity can improve the criterion validity of responses. The criterion of mimicking price-taking in the market as the basis for choosing an appropriate question format was posited by the NOAA panel in recommending the binary (dichotomous choice) question format (NOAA, 1993). As argued by Hanemann (1991), “to better mimic price-taking in market behaviour, it is now common to ask respondents whether they would pay a given dollar amount that varies (randomly) from respondent to respondent and then statistically infer the maximum WTP”.

Price-taking in many sub-Saharan market situations differs considerably from the context in which NOAA (1993) recommended the binary question format, and it may therefore not be the best format to be used in those areas. Question format is one technology that is unlikely to be transferred successfully, but which needs to be indigenous to the area in which it will be used (Onwujekwe, 2001). The context in which transactions take place in Nigerian markets is different from the European or American pricing system, which usually has fixed prices in open markets (Balogun, 1991). In Nigerian markets, people usually haggle before deciding on the final prices to pay for goods and services.

The content and criterion validity of a novel willingness to pay question format that closely mimics the haggling behaviour found in market transactions in Southeast Nigeria, called the ‘Structured Haggling Technique’ (SH), were compared with those of the bidding game (BG) and binary-with follow-up (BWFU) formats. Among all the existing CVM question formats, the BG and BWFU most closely resemble SH. The three question formats were first compared for content validity, and the findings used to improve them further before the test of criterion validity was undertaken. This involved determining the stated and actual WTP for insecticide-treated nets (ITNs) among the Igbo people of rural Southeast Nigeria. The nets are malaria control tools, and under current malaria control policy in Nigeria, people are expected to pay for them. ITNs have been shown to provide children with a remarkable degree of protection against malaria, and more generally against death before the age of 5 years (Binka, Kubaje, & Adjuik, 1996) and also have been found to be cost-effective for the prevention of malaria (Goodman et al., 2001).

SH has more iterations than the other two formats, so as to enable the respondents to better state their maximum WTP amounts (see below). In the BWFU format, a price for the good is presented to the respondent and s/he is expected to be either willing or not willing to pay the price. No matter the initial response, there is a follow-up open-ended question to determine the maximum amount the respondent is willing to pay. The BG also presents the respondent with an amount and asks whether s/he would be willing to pay that amount. Depending on the response s/he is bid up or down using a pre-determined bidding iteration until the maximum number of pre-determined bidding iterations is reached. Another format that presents the respondent with price choices is the payment card format (Klose, 1999), but this format was not evaluated in this study.