When using QMRA techniques to aid in the development of health-based quality targets for waterborne pathogens it is important that input data have validity. One of the inputs required for QMRA is accurate exposure information, which raises questions about how this information is best collected. Validity of a survey method can be quantified by comparing survey responses obtained with a gold standard. For example, in health-related studies where exposure to a particular agent is being assessed, responses to questions about exposures may be compared with results of tests for relevant biological markers. For studies such as this one where water used by householders inside and outside the home for non-potable purposes was the subject of investigation, the true gold standard is the volume of water householders are exposed to during specific water-using activities. Whilst sophisticated water meters are available that can, in theory, measure the volume of in-house and outside house water usage at individual taps, their use is somewhat problematic in an extensive household water usage survey. This is because of cost and logistical considerations associated with meter installation to a large number of households, meter calibration (e.g. to match water usage events such as toilet flushing or showering with recorded data 'spikes') and the lack of sensitivity of meters to detect single tap usage [
9]. In addition, the volume of water used at a particular tap is not a true measure of exposure because factors such as human behaviour and the type of water-using equipment also determine the exposure volume.
As a consequence of the barriers associated with accurately measuring individual domestic water exposure, this study sought to collect water usage information using two alternative data collection tools, a telephone interview and water-activity diary, both previously used for Australian household surveys [
6,
9,
10] and compare results. In this study, the comparison of these data collection tools was based not only on estimates of the duration and frequency of water-using activities but also on the cost and logistics of administration of each survey type.
Exposure estimates
The highest level of agreement between CATI and diary responses was obtained for: number of washing machine loads per week (Table
3); automatic system watering session duration (Table
2); use of high and automatic washing machine levels and use of cold and warm washing machine wash temperatures. Lower levels of agreement between CATI and 7-day diary results were obtained for less frequently performed water-using activities with the exception of toilet flushing (frequently performed but also showing poor agreement) or where standard durations (e.g. as might occur with automatic pre - programmed systems) or frequently applied settings were not used. 'Poor' agreement was obtained for: number of garden watering sessions; hand held hose watering duration; fixed manual water system duration; hose and sprinkler watering duration (Table
2); washing machine hot water wash; washing machine low water level and washing machine medium water level. Overall, when considering all individual water uses and the relative performance of the diary and CATI, diary responses were not consistently higher or lower than CATI responses.
The poor agreement between diary and CATI responses for the total number of toilet flushes per household per week (weighted kappa = 0.29) (Table
4) and for the frequency of use of half flush as a percentage of total toilet flushes (weighted kappa = 0.36) (Table
5) may be attributed to a number of factors. Considering that the number of toilet flushes is not insignificant per week per individual, it is likely that the exact number at home is difficult to recall. In addition, the CATI respondent was asked about individual behaviour, yet the diary recorded household behaviour. Accordingly, the number of toilet flushes per week estimated by the CATI respondent was multiplied by the number of persons in the households and compared with the tallied entries for toilet flushing recorded in the diary. A comparison of diary and CATI results (Figure
3) showed differential bias namely, that the CATI estimate (single respondent) combined with an assumption of identical toilet flushing behaviour by all persons in the household, overestimates the total flushes in large households.
The higher estimates of the frequency of use of half flush as a proportion of all toilet flushing for the CATI as compared with the diary (Table
5) indicates that the CATI respondent believed that the use of half flush was more prevalent in the household than was the case for the 7-day diary recording period. The CATI responses may represent the desired target behaviour of the respondent rather than actual household practice or, may truly reflect the respondent's behaviour, which is different to the rest of the household.
Many of the activities showing 'poor' agreement between data collection tools, as measured by the weighted kappa statistic, were not only less frequently performed activities, but also outdoor water-related activities. A possible reason for poor agreement between results for outdoor activities is that climatic conditions may have been different for each of the CATI and diary 7-day survey periods. Whilst the majority of diaries and CATIs administered were each compacted into separate seven week periods with no greater than a 3-week lag period between cessation of telephone interviews and commencement of diaries, the climatic conditions during diary and CATI survey periods may have been significantly different for a proportion of households (e.g. rain during the CATI 7-day survey period and no rain during the diary 7-day survey period or vice-versa).
The observation of greater concordance between CATI and diary responses for more frequent and standard water-related activities except toilet flushing may reflect the greater likelihood that standard frequencies, durations and settings are known to all adult household members. This observation is supported by results from other water-related studies where completion of a questionnaire also preceded the diary and referenced a different 7-day period [
11,
12]. These studies attributed lower correlations between questionnaire and diary responses to a greater potential for daily variation in some water-related activities compared with established routine ones.
When all results of this study are considered, they show that different estimates of the duration and frequency of water-related activities are obtained depending upon the survey method used. For estimates of garden watering frequency and the number of laundry machine loads per week, the diary, compared with the CATI, showed general positive bias (i.e. the diary response was consistently higher). Such differences between responses for CATI and diary may be associated with one or more factors including: recall bias of the CATI respondent [
13]; natural variation in water-related activity that occurs from week to week (CATI and diary responses 'referenced' different 7-day periods); the survey period may not have been over a sufficient period to 'capture' the exposure of interest (e.g. if garden watering was performed every 1.5 weeks it may not have been included in either (or both) of the diary or CATI survey periods); computations to convert single respondent results to household results (not all household members behave in the same way) and failure to complete diaries prospectively as intended or to record all activities and events [
14].
Compared with a telephone interview, the use of a household diary has a clear advantage in determining household exposure to a particular water supply. This is because the diary provides a collective measure of household exposure compared with an individual's estimate of household exposure. Acknowledging that the household diary cards may have in some instances been completed by one individual, it is more likely that they were completed by the householder performing the activity(ies) of interest. This assumption is based on the cards being placed in the location of water use (if instructions were followed) and/or that responsibility for diary card recording is likely to have been allocated within the household, to the household member with greatest familiarity with household water-using practices. In contrast, the telephone questionnaire may have been answered by an adult not fully familiar with the household water usage.
The prospective recording of water-related activities using a diary is also another advantage of using a diary collection method compared with a CATI. Diary responses are not subject to recall bias (assuming prospective completion) hence it is probable that the diary information provides the more accurate figure, compared with the CATI. Recall bias may be responsible for either under or overestimation as respondents may forget relevant episodes or they may report an episode from outside the period of interest as if it had happened within the period (forward telescoping) or vice versa (backward telescoping) [
13]. In this study, the CATI gave the higher figure for toilet flushing (this figure was also influenced by conversion from an individual estimate to a household one) and lower figures for garden watering frequency and number of machine washing loads. In addition, the diary format has the advantage that it provides continuous numerical, rather than categorical, data for use in QMRA modelling. In contrast, the CATI for some water-using activities provided only categorical data.
Logistics and Cost
Given advantages conferred by use of a diary, rather than telephone interview for obtaining household exposure information, this leads to questions about how to overcome obstacles such as cost and logistics, commonly associated with diary administration. In this study, the estimated number of input researcher hours per completed diary was 2-3 times greater than that for each completed CATI (Table
1). The time estimate for the completion of each was up until statistical analysis of data was performed but was not inclusive of the initial recruitment process which was common to both the CATI and the diary; diary recruitment being dependent upon CATI uptake. The researcher time input per CATI was approximately 30 minutes, based on the interview time (15-20 min), look-up of household details and telephone contact information prior to the telephone call and verification of data (data was checked immediately after the interview).
The CATI time estimate assumes that one telephone call will result in a completed CATI. In fact this was not the case as some households required more than one call to be made before a CATI (44%) was generated (or refused) and a percentage of calls did not yield a CATI (61%)[
15]. However, the assumption that one telephone call yields one CATI allows a more equitable comparison with the researcher time input for data preparation and handling following diary completion. This is because the starting point for the diary administration was agreement by the CATI respondent (70% agreed) that they wished to complete a water activity diary. Thus, each diary pack sent was expected to yield a completed diary that was returned to researchers. In fact this was not the case as the return rate (within the required 4 week time frame) of diaries was 63%. Based on this return rate and using a target of 200 completed diaries, the researcher time estimate per completed diary is approximately 85 minutes. This time estimate (expressed per unit diary) is based on: compilation of 320 diary packs, follow-up phone calls to 320 households to verify diary receipt; checking of completeness and legibility of 200 completed diary packs following receipt, tallying of data prior to entry onto data base, manual transfer of data to databases and data checking and verification.
A notable difference between the CATI and diary was that an incentive payment was offered for completion of the diary but not for completion of the CATI, adding to the budget costs for the diary. The use of a monetary incentive has been successful in other Australian studies [
10] and was considered appropriate to compensate householders for recording their water usage over a 7-day recording period (a significant time commitment for the respondent) but not for a 15-20 minute CATI.
In this study, manual entry of data was performed, adding to the total researcher time input required for data entry to the diary database. However, it is possible that this time input could have been reduced if data scanning of diary records had been employed, reducing both the time requirement for data transfers into the ACCESS database and for data checking. Clearly a change such as this would require custom design of data input sheets and the corresponding data base. Whilst this would add to the cost of diary data base design, it is likely that it would not exceed that of the CATI database design. However, even if data scanning was implemented, the overall cost of diary administration would nonetheless exceed that of a CATI based on the cost of incentive payments and remaining labour costs associated with the preparation, sending and sorting of diary recording sheets.
A notable observation of this study was the quality of diary card completion, indicating that the diary cards were well designed (based on cards used in a prior Australian study [
6]) and were easy to complete. The fact that there was no contact with households sent diary cards except for an initial phone call confirming diary card receipt and reinforcing that diary cards should be completed within a four week period was also testament to the clarity of diary instructions. This illustrates that it is possible to 'remotely' coordinate diary completion providing that adequate instructions are given and that diary recording forms are well designed. These observations are not only relevant to diary cards mailed to participants, but by extrapolation are also pertinent to web-based diary completion.
Limitations
This study was subject to a number of limitations including the sequential recruitment strategy in which householders were firstly recruited for the CATI and then (once the CATI was completed) were recruited to complete a 7-day water activity diary. This meant that the CATI 7-day period, always preceded the 7-day recording period for the diary. This limitation was countered as best as possible by confining the survey period to a maximum of 12 weeks (the summer period in which maximum household water consumption occurs). In addition, to reduce the time period between CATI and diary completion, a monetary incentive was offered to those households returning a completed diary within a 4 week period. Despite these measures, there was an approximate two to three week lag between CATI completion and diary commencement. The implications of this lag period are considered to be minimal for indoor water uses, which generally remain constant irrespective of weather conditions. However, this lag period potentially impacts on outdoor water uses such as garden watering where rainfall events and variations in temperature can influence garden irrigation frequency and duration. Rainfall in each of the CATI and diary 7-day periods was not tracked for individual households; hence the role of weather conditions on poor agreement between data collection tools for garden watering frequency was not able to be elucidated.
Another consequence of sequential recruitment was that only a subset of the household population that completed CATIs also completed diaries. It is therefore possible that the household population completing both the CATI and diary may have differed in some way from those that completed the CATI only. It is notable however that there was a high preparedness for householders to complete the diary (70% agreed) and that 63% of those agreeing to complete the diary also returned the diary within 4 weeks (a higher return rate may have resulted had the diary return been extended). The high rates of agreement to complete the diary support the view that financial reward was not a defining factor separating households that completed the diary from those that did not. Furthermore, the recycled water area in which householders were located is a niche housing development located in an area of high socio-economic status and is relatively homogenous with respect to household demographics and income.
Whilst it can be argued that it is likely that the diary population is representative of the larger household population completing the CATI only, a further question is whether the population completing the CATI are representative of the whole population of Rouse Hill dual reticulation households, including those that did not complete the CATI. In this study, CATI response rates were dependent upon the rate of matching of EWP and AEC records. The limiting factor in relation to household coverage was that only 57% of households were listed in the EWP, despite AEC records providing 87% of household coverage [
15]. While it is possible that the water-using behaviour of listed and non-listed households in the EWP does vary, we consider that EWP listing is unlikely to be a primary determinant in the volume of water used by households. The relative uniformity in Rouse Hill dual household characteristics that drive water usage such as garden area, garden age and household size in the entire target survey area independent of EWP listing supports this assumption. Nonetheless, the possibility that the subset of households completing the diary and/or CATI were different from other households, based on a greater interest in water sustainability and advocates of higher recycled water use (and thus with a greater preparedness to devote time to diary and/or CATI completion), cannot be discounted.
Another limitation of this study was that there was no independent measure of the frequency and duration of individual water-using activities; hence the validity of both data collection tools is uncertain. It is therefore possible that both questioning via the CATI and recording of water usage using the diary were potentially subject to respondents reporting aspirational, rather than actual, behaviour. This is somewhat unlikely as this study was undertaken independently of the water authority supplying recycled and drinking water to householders and there was no disincentive for householders to report their actual water behaviour. Also, even though there were water restrictions relating to the use of drinking water outside the home during the study period they did not apply to inside use or to recycled water use; encompassing those water uses under investigation in this study.
Even though there were no adverse ramifications for the householder associated with reporting actual water usage, it is possible that drinking water restrictions may have influenced recycled water use despite lack of restrictions for recycled water. Such behavioural modifications are not predictable and are best addressed by the collection of contemporaneous water usage data. In doing this, the impact on exposure profiles, of behavioural changes associated with changing community attitudes and newly introduced water-using appliances, can be assessed.