Can colorectal cancer survivors recall their medications and doctor visits reliably?

During the economic evaluation of health services, health care resources are required to be quantified and valued in monetary terms. The accurate estimation of these costs is crucial within economic analyses, however there are no standardized methods for collecting the quantities of health service use at the patient level. This is mostly because every study setting is unique, and the collection of this data is often constrained by the study design as well as participant burden considerations [1, 2]. Health resource use may be collected through a variety of means including patient chart reviews, administrative records at hospitals, clinics or insurance companies or by participant-reported methods, (self-reported using surveys, interviews or diaries). By necessity, the linkage of various sources is often required to capture the full extent of resources used [3].

Each method comes with its own set of advantages and limitations. Self-reported methods are open to poor recall or memory decay, interpretation difficulties, and increased participant burden. However, these methods are often cheaper, more efficient, and useful when multiple health services and locations are used for a particular health condition. Medical chart review is a relatively labour-intensive method. There may be difficulty in accessing medical charts and special expertise may be required for interpreting the data, however a wide range of clinical data is available. Administrative records are not designed for research projects and therefore not all relevant data is likely to be available and there is the problem of timely access, cost, legal and privacy limitations and an unknown degree of programming or coding errors. There is currently no gold standard or benchmark although many researchers suggest administrative records are more likely to be accurate for encounters with health services [2, 3]. Automated administrative data sets also allow large numbers of individuals to be analysed and compared with other data sources. Moreover, the availability of data linkage systems and electronic health records will facilitate large-scale, population-based research and health service evaluation, subject to the issues of missing data, confidentiality and errors [4].

Many studies have investigated the extent of agreement between at least two of the available methods in different populations, with or without certain health conditions, to decide on their exchangeability. The discordance between different methods has been attributed to the types of services studied, the recall period, socio-demographic profile of the target population, poor health status and symptoms, high stress or high treatment needs, errors in administrative records, incorrect recording of records and patients misspecification of a treatment or service [2, 5]. Importantly, the reliability and validation of readily available instruments for researching self-reported health care utilisation is an area still developing with standardized methods gradually improving, see http://​www.​dirum.​org. The task is particularly challenging when there is a high volume of resource types and quantities and considering adequate coverage of resource types across many different diseases and patient groups.

In general, studies have shown poorer agreement for participant-supplied data against routinely collected administrative data sources among elderly participants (due to high service needs and potential memory decline), males, those experiencing high frequency of visits, health service contacts of a non-serious nature, poor health status and collecting data over longer recall periods (e.g., 12 months) [2, 5]. Several reports have also shown good reliability of self-reported data in relation to administrative data [6‐9] and particularly for hospitalisations [2, 6, 8]. It has been suggested that since generalisation across studies is problematic due to differences in populations, diseases and recollection periods, health service researchers should address the internal validity of their own health service use data, uncover any systematic bias, its potential uncertainty and characterise the ensuing implications [5].

To this end, in the context of a larger cost-utility analysis for assessing the costs and benefits of a lifestyle intervention for colorectal cancer survivors (CanChange) [10], we tested the consistency and validity of self-reported GP and specialist visits over six months, and medication use against linked administrative data from the national Medicare Benefits Scheme among a sub-sample of participants (n = 76). In addition, we assessed factors that potentially influenced disagreement in responses across the two methods.

In our sample of 76 participants, the mean (SD) age was 65 (9.8) years and 32% were aged between 71-80 years (Table 1). Men and women were represented equally, 68% lived with a partner, 70% had private health insurance, 47% had higher education qualifications and 49% were retired from the work force. Twenty-four percent had Duke’s stage C cancer (severe disease), 54% had received open surgery versus 43% for laparoscopic surgery. Participants were also commonly treated with adjuvant chemotherapy (59%) and/or radiotherapy (84%) (Table 1).

In a sample of colorectal cancer survivors, the overall quality of self-reported doctors’ visits and medication use was mixed with reasonable agreement for doctors’ visits but poor agreement for medications. Although the ICCs and kappa statistics showed ‘substantial’ agreement between the two methods for doctors’ visits, considerable under-reporting occurred, the proportion of absolute agreement was small and the 95% confident limits ranged from poor to substantial. Nevertheless, when costs were applied to the survey data and compared with MBS fees, the cost differences were not substantially different. Medication types were considerably over-reported in our sample with the exception of psychological medicines that were under-reported.

Our results showed a tendency for males and those with frequent contacts to under-report actual visits, coinciding with the findings from other studies [3, 5]. However, unlike other reports, we did not show disagreement was linked with older age, more severe disease and income or education levels for both doctors’ visits and medication use. Forty-three percent of our participants received laparoscopic surgery rather than radical open surgery and this was a factor associated with perfect agreement of specialist visits. Although the numbers involved are small, this may be a marker for participants having better general well-being and/or fewer side effects after surgery, and subsequently fewer services to recall, compared with participants undergoing other surgeries.

While our results point towards self-reported medication use being somewhat unreliable, it is important to understand the research context. Our participants have colorectal cancer and high treatment needs, they have multiple treatment modalities and they are likely to experience ongoing side-effects consistent with the evidence [12]. Treatment involves a protracted and sometimes complex navigation through the health system at different service locations [13]. Therefore, recall of health service utilisation over a six month period may be difficult for this group given their high usage and poor health status resulting in ‘poor agreement’ between self-reported and administrative data.

The agreement between the two methods for prescribed medication responses was disappointing despite efforts to ask participants to prepare records ahead of the interview time and clarify with the researcher any misunderstandings they had. However, this method assumed participants would be highly motivated which may have been unrealistic. Participants may have confused some prescription medications with over-the-counter medicines, participants may have included inexpensive medications not subsidised by the PBS or it is possible that hospital-acquired medications or those prescribed earlier in the recollection period were forgotten. It is clear from our findings that administrative data is necessary for collecting medication data in our participants receiving polypharmacotherapy. PBS data showed the total number of prescriptions (including repeats) was 1523 at a value of $244,231 to the government and $16,484 in out-of-pocket expenses for the participant. We did not test the reliability of frequency or repeats of specific prescriptions but it is clear from the survey responses for general groupings that this would have been too onerous for our participants to accurately recall.

Ultimately, the purpose of our study was to investigate whether self-reported health care utilisation via telephone surveys was an acceptable method for collecting data within a larger cost-utility analysis. If under-reporting service use can cancel out over-reporting and this occurs randomly across intervention arms, then minor discrepancies in resource use may be acceptable. However, our data suggested the extent of discrepancies was more serious however, there were no systematic differences in agreement categories across the CanChange intervention arms. When costs were applied to the frequency of doctor visits, MBS costs were significantly higher than costs from self-reported use, 18% higher for GP visits and 29% higher for specialist visits. Therefore, if the self-reported data is relied on for the cost-utility analysis, the extent of under-reporting may bias the final results. In this particular case, it may be preferable to use the mean cost and standard error from the Medicare records for use in modelling the larger study. Alternatively, the self-reported data collected on the full sample in the larger study could be increased by a compensating factor representing the extent of disagreement seen in our 76 participants.

Our study has several limitations. The small sample was not powered to detect significant differences in agreement category within logistic regressions. The framing of the survey questions may have been too broad and open to interpretation, particularly the medication question. To overcome this limitation we could have asked a closed question about medication usage including a list of specific drugs of interest (e.g., anti-depressants, anti-hypertensives) with their commonly available trade names. Finally, we assigned average costs of MBS attendance items to the survey data during our cost comparisons which may have obscured the variation in doctor services provided. However, the unit costs reflect the most common MBS items for GP and specialist visits and were considered reasonable estimates. Finally, important health service items collected by self-reports for allied health services, community services and home care were unable to be verified with Medicare records as they are not billable items and were therefore excluded from the analyses.

Research has suggested that cancer support services may promote downstream cost-savings to the health system from avoided GP visits and lower medication use [14]. The strength of this claim is not yet supported by the research evidence and the issue is further complicated by the measurement challenges for health service use. Due to having Medicare data on only a small sample of colorectal cancer survivors, we are not able to contribute meaningfully to this topic, however forthcoming research on a larger sample of men with prostate cancer receiving a psychological intervention [15] will inform this growing literature.

There is an increasing demand for data on health resource use at the patient level among researchers seeking reliable and high quality data. In comparative intervention research, the types and extent of resources used is important to evaluators to inform service delivery, future service planning and to contribute to assessments of cost-effectiveness.

In conclusion, our findings suggest that self-reported data for doctors’ visits is reasonably reliable for use in economic studies but self-reported medication use is sub-optimal. Administrative data on doctor visits and medications are preferable to self-reported data due to substantial disagreement in self-reported telephone interviews. If self-reported data is the only available option, very specific closed questions for doctor visits and medication usage are recommended.