Bias of health estimates obtained from chronic disease and risk factor surveillance systems using telephone population surveys in Australia: results from a representative face-to-face survey in Australia from 2010 to 2013

Many established population-based, continuous chronic disease and behavioural risk factor surveillance systems worldwide utilise Computer Assisted Telephone Interviewing (CATI) [1‐9]. Since the 1990s, CATI surveys have been seen as an ideal tool since they are effective, relatively inexpensive, flexible and timely [6, 8‐12]. However, over the past 15 years vast changes have occurred in the telecommunication industry (mobile telephone and internet) and society’s acceptance of, and engagement with, these new technologies [13, 14]. The new communication technologies have had an impact on population-based telephone surveys, specifically, the diminishing coverage of traditional sampling frames and declining response rates [11, 15] resulting in increased costs [16, 17] and potential bias in survey estimates [18, 19].

In the early 1990s, 95–97 % of Australian households had a landline telephone connected [20] and response rates of around 70–80 % were the norm [20‐24]. For population health surveys in Australia, two sampling methodologies were used: directory-listed telephone numbers, referred to as Electronic White Pages (EWP) and random digit dialling (RDD) of landline telephone numbers [3, 20, 22]; both methods having the ability to target geographical areas (state, suburbs or postcodes) which has contributed to the utility and efficiency of telephone surveys [25, 26]. EWP consists mainly of listed landline telephone numbers with name and address details for a household or business which the sampling frame can be easily stratified by state, suburb or postcode. EWP has mobile and Voice over Internet Protocol (VOIP) telephone numbers but only as a small proportion of the total sample. One drawback of EWP is that it does not include unlisted (silent) telephone number; that is, households which have opted, at a cost, to exclude their landline telephone number from the EWP. RDD methods have been developed to include silent landline telephone numbers based on the prefixes of the landline telephone numbers. Some of these methods use the EWP, known as list-assisted RDD (LA-RDD), to make the sampling frame more efficient by removing blocks of numbers that have a high chance of not being connected or are assigned to large businesses [3, 27]. These RDD methods do not include mobile or VoIP telephone numbers. Since the turn of this century, there has been a trend of households moving away from traditional landline telephones with the emergence of mobile-only households [11, 13, 15, 28]. This is due to increasing portability, flexibility, affordability and broadening internet capability of mobile telephones including smartphones and other telecommunications, such as VoIP [11, 15, 26, 29‐32].

As a result of the increasing use of mobile telephones, conducting telephone surveys has become increasingly problematic in Australia and other countries [15, 33]. This is because of the difficulty in obtaining a representative sampling frame of mobile telephones numbers since are they are rarely listed (7.3 % of mobile telephone owners in South Australia are listed [26]). Unlike the structure of landline telephone numbers, the Australian mobile numbers do not provide details of geographical location and the common methods used to generate a RDD sample of landline telephone numbers geographically are not applicable to mobile telephone numbers [34, 35]. In 2011–12, approximately 20 % of households in Australia were mobile-only [14, 29], 34 % of USA households in 2012 were mobile-only [30] with countries in Europe reporting 50–70 % [32]. More notably, studies have found that mobile-only households are demographically different to traditional landline households: they are generally younger people, unrelated, never married, and socioeconomically disadvantaged [26, 30]. These issues suggest that by excluding mobile-only households biased estimates may be produced from chronic disease and behavioural risk factor surveillance systems.

This study presents the most up-to-date estimates available on the current status and possible sample biases of the current telephone survey methodology in South Australia, a state of Australia. Data from an annual representative face-to-face (non-telephone) population survey that included questions about telephone ownership were used to assess the population that were non-contactable by current telephone sampling methods. This included both household landline and mobile telephone ownership and listings in the telephone directory. This study will 1) explore trends of landline and mobile telephone ownership between 2006 and 2013; 2) describe the socio-demographic characteristics of respondents living in mobile-only households between 2010 and 2013; and 3) investigate the coverage bias of the two telephone samples (directory-listed numbers (EWP), consisting mainly of landline telephone numbers and a small proportion of mobile and VoIP telephone numbers; and a RDD sample of landline telephone numbers which excludes mobile-only households) by examining the prevalence estimates of health status and health risk behaviours between 2010 and 2013. This is one of the few studies to assess the potential bias of health estimates due to coverage bias from telephone sampling frames in terms of health indicators and socio-demographics, using a unique data source with telecommunication information on people who would be excluded from the hypothetical telephone samples [26, 30]. This study uses relatively current data, which is important since telecommunications technologies have rapidly changed and evolved over the last 10 years, with increased uptake and saturation of mobile telephones and associated changes in the way people communicate [36]. Methodological studies therefore need to continually assess sample coverage and potential bias in health-related estimates [26].

Figure 1 shows the household landline and mobile telephone status from 2006 to 2013. Mobile telephone ownership was consistently around 90 % during the last eight years, rising from 89.7 % (95 % CI 88.5–90.9) in 2006 to 96.3 % (95 % CI 95.5–97.0) in 2013 (7.4 % increase). The proportion of households that are mobile-only has increased by 431 % over the eight year period from 5.2 % (95 % CI 4.4–6.0) in 2006 to 27.6 % (95 % CI 24.7–30.7) in 2013. In contrast, the proportion of landline ownership (households with landline telephone only, and households with both landline and mobile telephones) has decreased by 24.1 % from 94.4 % (95 % CI 93.2–95.) in 2006, 87.3 % (95 % CI 85.7–88.8) in 2009 to 71.7 % (95 % CI 68.6–74.6) in 2013. Descriptive statistics for the participants for 2010 to 2013 are provided in Additional file 1: Table S1.

Table 1 shows the proportion of respondents living in mobile-only households by socio-demographic variables across the four years. Generally, respondents living in mobile-only household were more likely to be male, younger, of Aboriginal or Torres Strait Islander descent, born in Asia or countries other than Australia, UK, Ireland or Europe, never married, or separated or divorced, unemployed, fulltime employed, or home duties, renting privately or from the government, and to reside in the most disadvantaged areas. Largest percentage increases over the four years occurred amongst females (86.6 %), people in the older age groups (86.2–159.4 %), people living in rural areas of South Australia (80.4 %), people born in the United Kingdom or Ireland (118.3 %), people living in single parent households or shared-care parenting households (77.7 %), or couples with no children (72.6 %), widowed (131.4 %), married or in a defacto relationship (75.2 %), people with at least secondary schooling (81.0 %), people living in households on low income levels (89.4 %) or very high income levels (97.7 %), and people who are retired (134.4 %) or who are currently students (98.6 %).

Additional file 1: Tables S2 and S3 show the proportion of respondents living in a household with a landline connection and the proportion of respondents living a household with at least one mobile telephone by socio-demographic variables for 2010 to 2013. The proportion of respondents living in households with directory-listed mobile or landline telephone (EWP) has been steadily decreasing from 73.8 % (95 % CI 72.2–75.4) in 2006, to 60.4 % (95 % CI 58.1–62.7) in 2010 and 49.6 % (95 % CI 46.2–53.0) in 2013. This proportion by socio-demographic characteristics for 2010 to 2013 is listed in Additional file 1: Table S4. In 2013, 4.6 % (95 % CI 3.8–5.5) of mobile numbers were listed in the telephone directory compared to 62.7 % (95 % CI 59.2–66.1) of landlines.

The prevalence estimates of various health conditions and behavioural risk factors for all households, for people who live in households with a landline connection (hypothetical landline RDD sample) and for people who live in a household with a directory-listed landline or mobile telephone number (hypothetical directory-listed sample) are shown in Table 2. The RCB for the prevalence estimates derived from the two hypothetical samples are also in Table 2. There were small absolute differences in the prevalence estimates for current asthma, arthritis and obesity between the hypothetical telephone samples and the overall sample. The prevalence estimates for diabetes by the two hypothetical samples did not differ in 2010 and 2011, however, the prevalence estimate was slightly underestimated (RCB value of −0.077) in 2013 for the directory-listed sample. Even though the prevalence estimates for arthritis were similar for both hypothetical samples, the prevalence estimate for arthritis in 2010 was underestimated for the directory-listed sample (RCB value of −0.083) compared to the overall sample (prevalence of 20.7 vs. 21.4 %). The prevalence of having a mental health condition showed mixed results for both hypothetical samples and over time: the prevalence of having a mental health condition was underestimated for both samples with estimates from the directory-listed sample having larger RCB (ranging from −0.102 to −0.242) with the exception of 2011, which had the opposite result of overestimating mental health conditions (RCB value of 0.056). Current smoking prevalence was lower for both hypothetical telephone samples with absolute differences ranging from 2.9 to 3.4 percentage points for RDD landline samples and 3.3 to 5.3 percentage points for directory-listed samples, and associated large RCB values: −0.136 to −0.191 for RDD landline samples and −0.129 to −0.313 for directory-listed samples.

This paper presents estimates and trends of telephone coverage in Australia from 2006 to 2013. Continual assessment of methodological issues around conducting population health telephone surveys is essential due to the rapid technological changes in telecommunications and the different ‘user culture’ associated in use of these new and old telecommunication technologies. Even though telephone (landline and mobile) coverage in South Australia is very high (97 %), nearly a third of households are mobile-only (27.8 %) and only half of the households (49.0 %) have either a mobile or landline number listed in the White Pages telephone directory. Our results show that mobile-only respondents are different across a range of socio-demographic indicators, which is similar to international studies [13, 15, 30]. Using hypothetical sampling frames (RDD landline and EWP directory listing) that were weighted to the age and sex structure of the South Australian population produced contradictory results for health prevalence estimates when compared to all households in the face-to-face survey. Prevalence estimates of diabetes, current asthma, arthritis and obesity had very minor differences and biases, but the prevalence estimates for mental health condition and current smoking indicates biases using either RDD landline or EWP directory listing sampling frame. Even though our results show that mobile-only respondents are demographically different across a range of socio-demographic indicators, appropriately weighted data can produce reliable prevalence estimates for some health indicators, but not for others. These findings suggest landline-based sampling frames used in Australia are potentially biased for some health indicators, such as current smokers and having a mental health condition, particularly where conditions or risk factors are higher amongst those living in mobile-only households. Researchers using either RDD or directory-listing landline sampling frames need to be aware of their limitations and know of the potential biased estimates because of the groups that are excluded from the sampling frames.

This study is important because it quantifies the potential biases from the various landline-based telephone sampling frames used in Australia and the groups that are potentially excluded. Even though the data are limited to South Australia, the conclusions may be generalizable to the Australian population. This study is unique since the same questions have been asked annually for eight years and, using the face-to-face methodology in which all types of households are included (mobile-only, landline-only or both), it had the ability to examine, over time, the prevalence estimates of various health indicators by telephone status. Very few studies like this are known to exist nationally [14] and internationally [15, 30] and even fewer examine the assessment on health indicators [30].

The trends and demographic differences found in this study are similar to national and international studies [11, 14, 15, 30, 42, 43] and support findings from our previous research [26]. Our estimate of mobile-only households in 2012 (23.9 %) was higher than the estimate reported by the Australian Communication and Media Authority (19 %) [14]; the proportion of households with a landline telephone in 2010 was 82.5 % which was slightly higher than the 80.3 % estimate from the 2010–11 Australian Health Survey (AHS); and our estimate of 68.7 % of landline telephone numbers listed in the telephone directory was slightly lower than the 70.1 % from the AHS 2010–11 survey [44]. Between 2006 and 2008 the trend of mobile-only households remained low, however since 2009, the trend has steadily increased, following international patterns [30]. Similarly for landline ownership, up to 2011 the proportion was over 80 %, however, this has steadily decreased to 71.9 % in 2013. These changes are mainly due to the increasing popularity of greater flexibility and affordability offered by mobile technology. People are using landlines less frequently because they are able to have a single device with multiple communication and media services, which is less expensive than having a landline connection [13].

In our previous study [26], nearly 10 % of the population in 2008 lived in mobile-only households, and we showed that with appropriate weighting, the sampling methodology used for telephone surveys produced reliable health estimates with the exception of smoking prevalence in South Australia being underestimated. In contrast, with more recent data and up-to-date analyses, this study has estimated that close to 30 % of the Australian population now live in mobile-only households and these analyses have demonstrated the impact of the vast changes in the telecommunication over the eight year study period on the coverage of the sampling frames. Excluding a distinct subpopulation from the landline sampling frames, namely mobile-only households, resulted in under- or over-estimation in some health estimates, although with appropriate weighting most health estimates (except smoking and mental health) were very similar to the overall population. Even though the results in the health estimates (absolute differences and RCB values) between the overall population and the two hypothetical landline sample groups showed no clear pattern over time, the results do highlight that for specific health indicators, such as current smokers and mental health, the direction of the bias was consistently under-estimated for both RDD and directory-listed landline hypothetical samples. The other conditions (diabetes, current asthma, arthritis and obesity) had little absolute differences in health estimates and an inconsistent pattern, but relatively low, RCB values over time, which may suggest that the differences could be due to the random nature of the sample or other sampling errors. Our findings for current smokers, asthma and obesity are similar to other USA studies [30] using similar methodology, and are consistent with studies using dual-frame telephone surveys for mental health [45], current smoking [30, 46, 47], asthma [47], and obesity [30]. This suggests that perhaps an alternative sampling, surveying or statistical methodological approach may need to be considered to include groups of the population to remove the coverage biases in landline-based sampling frames.

Many studies have explored various methods to include the mobile-only group into chronic disease and risk factor surveillance systems [12, 48]. The favoured method is an over-lapping dual-frame design which involves two independent samples: a sample of mobile telephones and a landline-based sample [34, 35, 46, 49]. These studies showed an improvement in the representativeness, in particular for men, the younger and middle age groups, and people who were never married. However, obtaining a sample of mobile telephone numbers does have drawbacks, including low response rates and two to four times the costs of landline-based samples [34]. More importantly, the mobile sample that is currently available and used in Australia is of randomly generated mobile telephone numbers with no geographical marker. From a South Australian perspective, only 8 % of all mobile telephone numbers in Australia were estimated to be owned by South Australians [34, 35, 46, 49], which is almost the same proportion of the state’s population (7.4 %). This means a much larger initial sample is required for screening, and with the additional problem of low response rate, the feasibility of including mobile numbers using these methods in a chronic disease and behavioural risk factor surveillance system in South Australia would be costly. Even though 98 % of South Australians have a mobile telephone and it is perceived that people can be reached anytime, it does not mean that they are willing or able to use it to complete a survey. Receiving mobile telephone calls can happen at unpredictable moments when it is not suitable for the owner to respond, such as driving (safety issue), travelling overseas (which can incur a large cost to the researcher or participant), or during a meeting or in a restaurant (privacy issue); all have an impact on response rates [43].

Mixed-mode methods have also been suggested as a way to complement the traditional landline telephone survey by combining face-to-face, mail, and internet surveys [50]. These alternative modes introduce other methodological issues and the design of each mode need to be taken into consideration. The questionnaire design for CATI surveys, for example, complicated skips patterns or data range checks, needs to be careful considered in other modes such as mail survey [51]. Face-to-face, mail and internet survey can have the option of longer worded questions, explanations, and visual or prompt cards which is not recommended or possible with CATI surveys. Therefore, the wording of the questions in telephone surveys needs to be clear, concise and short [52]. Operational differences can have an impact on how the questions are answered. Telephone surveys are mainly interviewer administered whereas mail or internet surveys are self-administered which can lead to different responses [50, 51]. In telephone surveys, the interviewer has control over who is the selected respondent within the household whereby in the mail or internet surveys any member of the household determines who is the selected [12]. The level of privacy can vary by survey modes which is high with mail or internet surveys compared to moderate level of privacy with telephone (others listening in, or answering sensitive questions) [53]. Mail surveys require a longer data collection period compared to the allocated time period for telephone surveys. In an attempt to include respondents from mobile-only households, a study examined the possibly of using two modes, telephone and mail, with a single database that consisted of residential addresses. However, they found that the groups that were under-represented in telephone surveys were also under-represented in the mail surveys [48]. Another consideration for surveillance systems that used the telephone to collect data, is the challenge of how to incorporate alternative modes but still maintain the timeliness, flexibility, low non-response and low cost of the system [12]. Other methodological studies have used statistical approaches such as alternative weighting strategies, such as raked weights, which incorporate a wider range of socio-demographic variables, can improve the health estimates and are more in line with face-to-face surveys [54‐56].

The study design used in this research is robust due to the large representative state-wide samples used and is unique in that the data were collected over eight years using the same or similar questions, and by one organisation, thus minimising interviewer biases. These data are also very recent and it is one of the few face-to-face studies conducted in Australia and worldwide that included questions on landline and mobile telephone status that also had questions on health status and behavioural risk factors [30] so the biases in health estimates can be assessed. However the results could be biased due to the moderately acceptable response rates (median = 59.3 %) which is following the trends observed interstate and overseas. This study only analysed a few health-related variables and additional questions such as health service usage, quality-of-life or alcohol consumption would have provided a more comprehensive description of telephone sampling biases.

Telephone surveys have become a standard and accepted method of collecting health information in Australia and are widely used to monitor chronic disease and behavioural risk factors. Such surveillance systems provide evidence to inform interventions and service planning with the aim of reducing the impact of chronic diseases and their associated costs to the health system. Analyses like those presented here are important to demonstrate that the health estimates obtained are not biased due to sampling methodology. This study has shown that the proportion of mobile-only households is increasing and this does not appear to have reached a plateau. This corresponds with the decrease in landline telephone coverage. Even with appropriately weighted data, using landline-based sampling frames in Australia are potentially biased for some health indicators. This implies that the landline sampling frames that are currently used in most Australian chronic disease and risk factor surveillance systems (RDD landline or directory-listed telephone numbers) are not sufficient on their own because of the exclusion of the mobile-only households. Other methodologies need to be considered for small states like South Australia that are timely, cost-effective and efficient.