Figures
Abstract
Background
Verbal autopsy (VA) is one method to obtain valid estimates of causes of death in the absence of valid medical records. We tested the reliability and validity of a VA questionnaire developed for a cohort study in Golestan Province in northeastern Iran.
Method
A modified version of the WHO adult verbal autopsy was used to assess the cause of death in the first 219 Golestan Cohort Study (GCS) subjects who died. The GCS cause of death was determined by two internists who independently reviewed all available medical records. Two other internists (“reviewers”) independently reviewed only the VA answers and classified the cause of death into one of nine general categories; they repeated this evaluation one month later. The reliability of the VA was measured by calculating intra-reviewer and inter-reviewer kappa statistics. The validity of the VA was measured using the GCS cause of death as the gold standard.
Results
VA showed both good validity (sensitivity, specificity, PPV, and NPV all above 0.81) and reliability (kappa>0.75) in determining the general cause of death independent of sex and place of residence. The overall multi-rater agreement across four reviews was 0.84 (95%CI: 0.78–0.89). The results for identifying specific cancer deaths were also promising, especially for upper GI cancers (kappa = 0.95). The multi-rater agreement in cancer subgroup was 0.93 (95%CI: 0.85–0.99).
Citation: Khademi H, Etemadi A, Kamangar F, Nouraie M, Shakeri R, Abaie B, et al. (2010) Verbal Autopsy: Reliability and Validity Estimates for Causes of Death in the Golestan Cohort Study in Iran. PLoS ONE 5(6): e11183. https://doi.org/10.1371/journal.pone.0011183
Editor: Cesar V. Munayco, Direccion General de Epidemiologia, Peru
Received: March 30, 2010; Accepted: May 25, 2010; Published: June 17, 2010
This is an open-access article distributed under the terms of the Creative Commons Public Domain declaration which stipulates that, once placed in the public domain, this work may be freely reproduced, distributed, transmitted, modified, built upon, or otherwise used by anyone for any lawful purpose.
Funding: This work was supported in part by intramural funds from the collaborating institutions, including Digestive Diseases Research Center of Tehran University of Medical Sciences (www.ddrc.ac.ir), Cancer Research United Kingdom (www.cancerresearchuk.org), and the Division of Cancer Epidemiology and Genetics of the National Cancer Institute (http://dceg.cancer.gov). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Introduction
Accurate information on overall and cause-specific mortality is essential to prioritize the activities of health systems and to efficiently invest scarce public health and medical care resources [1], [2], [3]. The availability of such information is also important for epidemiologic studies. The standard method to determine the cause of death is certification by an attending physician, based on valid medical documents, but this approach may yield unreliable results in many low- and middle-income countries, especially in rural and suburban areas. This is mainly due to the lack of infrastructure and the high cost of collecting the data, which limit access to information from diagnostic tests and post-mortem pathology services. Mortality data from these countries are therefore limited and potentially biased [3], [4], [5]. One relatively simple and low-cost alternative for determining a person's cause of death which is available in most low-resource countries is the so-called verbal autopsy (VA) [5], [6].
The VA methodology was first developed for investigating epidemics [7] and was later used for evaluations of outcomes of specific interventions [8], [9] and national mortality surveillance systems, principally in low-income countries such as India [10]. Several studies have shown that VA gives more valid causes of death than routine death certificate data in many developing countries [11], [12], [13], [14], [15]. In VA, a trained interviewer ascertains the symptoms, signs and events during the period leading up to death from family members or primary care givers of the deceased [6], [16]. This information is analyzed to derive a probable cause of death. The most commonly used method for analysis of the collected information is a “physician's review”, generally performed by more than one physician [6], [16]. Other methods, such as algorithms that can be applied by computer, have been tried but found to lack validity [17], [18], [19].
During 2004–2008, the Golestan Cohort Study (GCS) enrolled more than 50,000 adults in Golestan Province, in northeastern Iran [20], following a pilot study [21]. Golestan is a low-resource area of the country, and consequently, reliable clinical data are not available to determine the causes of death of the residents. Thus we have applied the VA method as a tool to identify the causes of death in the GCS. It is estimated that about 60% of the GCS participants will die at home, and some of them will not have any medical records accurately documenting their cause of death. VA represents an appealing approach to determine the cause of death in this group of subjects. However, it is necessary to validate the VA questionnaire in this adult population. The majority of VA validation studies have focused on neonatal and childhood mortality [12], [22], [23], [24], [25], [26], [27], [28], [29], [30], [31], [32], [33]. Only a few studies have investigated the validity of VA in adults [12], [25], [28], [32]. Although VA is prone to erroneous estimates of cause-specific mortality rates due to misclassification [34], several studies have demonstrated its ability in valid identification of the most common causes of death in many settings [12], [26], [27], [29], [32], [35], [36]. And even those who think VA is an imprecise tool for detecting the leading causes of death suggest that in the absence of other more reliable methods, VA may be useful as a secondary tool to determine causes of death in rural areas [6]. Our study is the first attempt to validate an adult VA questionnaire to be used in a longitudinal study in a medium income country.
Materials and Methods
The Golestan Cohort Study
The methods of the Golestan Cohort Study (GCS) have been previously described in detail [20]. In brief, 50,045 adult middle-aged individuals were enrolled in eastern Golestan Province, Iran between January 2004 and June 2008. Participants are actively followed through annual telephone contact to ascertain their vital and health status. If a participant cannot be reached, family members, friends, or local health workers are contacted. Moreover, local health workers in rural areas, called “Behvarz”, are contacted monthly to inquire about any possible outcomes, including death. In the event of death, the follow-up team performs two main tasks in parallel. First, a trained general practitioner goes to the homes of the family members or primary care givers of the deceased and conducts a VA interview. Second, the team determines which physicians or hospitals were visited by the decedent and obtains all medical documents (charts, X-rays, pathology reports, etc) that could be used to identify the cause of death. These documents may be available in Golestan or in neighboring provinces.
The GCS follow-up team uses the adult VA questionnaire originally developed by World Health Organization (WHO) and the International Network of field sites with continuous Demographic Evaluation of Populations and Their Health in developing countries (INDEPTH) [37], [38], with some modifications to adapt to the local situation in Golestan. We tailored the standard VA questionnaire based on cultural background and education of study population. We made special attention to the most common disease and causes of death in the study area. We added some disease-oriented questions for specific diseases (cardiovascular, stroke, cancer (esophageal and gastric), diabetes, hypertension, tuberculosis and asthma) to collect more information by VA. Since we have already collected the data of life style and personal habits of the study participants at the enrollment phase of GCS, we excluded this part of VA questionnaire to save time. Local terms for some signs/symptoms such as “dysphagia” were applied when we translated the VA questionnaire to Farsi.
After the VA interview and medical document search are completed, the results are given to two internists to ascertain the cause of death. The two internists who review the VA and other documents are unaware of each other's diagnosis. When they disagree on the cause of death, a third senior internist reviews the VA, the available documents, and also the diagnoses of the first two internists and makes the final decision. All causes of death are coded according to the core three digit codes of the International Classification of Diseases, Tenth Revision (ICD-10) [39]. The cause of death obtained by this method was considered as the gold standard for the current validation study. Seventy cases (32%) had no medical documents, so in these cases the VA-based diagnoses confirmed by the above method were used as the gold standard.
Validation study
This validation study was conducted on all 219 deaths reported in GCS participants by the end of January 2005. Copies of all 219 completed VA questionnaires were given to two trained internists, henceforth referred to as the “reviewers”, who were different individuals from the internists who made the first GCS cause of death determinations. The reviewers studied the completed VA questionnaires independently, and made their decisions on the cause of death based on the VA questionnaire alone, without having any other medical documents. In order to get an estimate of within-reviewer reliability of the VA diagnoses, the same two reviewers were asked to review the VA's a second time one month later, without the knowledge that this was a repeat review.
For the purpose of this study the causes of death were categorized into one of nine major categories. To estimate the reliability, kappa statistics were calculated for the agreement between the reviewers' diagnoses. Both within-reviewer reliability (comparing the first and second diagnoses of the same reviewer) and between-reviewer reliability (comparing the diagnoses made by the two reviewers) were calculated. Multi-rater agreement was calculated and its confidence interval was calculated using bootstrap technique. To estimate validity, the VA diagnoses made by the two reviewers were compared to the gold standard diagnoses and the sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV) and kappa statistics were calculated for each reviewer-gold standard comparison. As a sensitivity analysis, the VA validity was once calculated in the subgroup of 149 cases (68%) who had both VA and medical documents available for the gold standard cause of death determinations, and then in all 219 cases.
All study participants had signed written informed consent at enrollment phase of GCS and ethical approval for the present study was obtained from the ethics committee of Digestive Disease Research Center of Tehran University of Medical Sciences.
Results
Of the 219 deceased participants, 133 (60.7%) were male and 86 (39.3%) were female. The mean age (±standard deviation) at death was 64.4±10.7 years. Among the deceased, 91 (41.6%) were urban and 128 (58.4%) were rural dwellers. In most cases (85%), the respondent lived with the deceased at the time of death. Of the 219 deaths in the validation study, 70 (32%) had no medical record other than the completed VA.
Table 1 presents the major causes of deaths according to the gold standard diagnoses, among the total study population and the subgroup of 149 subjects (68%) who had both VA and medical documents available. Ischemic heart disease, cancers, cerebrovascular events, and transportation accidents were the most common causes of death, respectively, and were responsible for, approximately 80% of deaths.
Table 2 shows the results of kappa statistics for the within and between reviewer diagnoses and the comparison of the VA diagnoses with the gold standard, based on the 149 deaths with documentation available.
The overall multi-rater agreement across four reviews was 0.84 (95%CI: 0.78–0.89). Most pairwise kappas were higher than 0.80, indicating good within-reviewer and between-reviewer reliability, the within-reviewer reliability being somewhat better than between-reviewer reliability. Agreement between each reviewer and the gold standard was also good (kappa>0.75).
Sensitivity, specificity, and predictive values for the four most common causes of death are presented in Table 3. To analyze sensitivity, these were calculated for the A1 review which had the lowest agreement with the gold standard and then for the one with the highest agreement (A2). All estimates were higher than 0.81 which indicate good validity. As expected transportation accidents had the highest validity.
Since the main goal for Golestan Cohort Study is to study the causes of upper GI cancers in particular and other cancers in general, we also tested the validity of VA for different types of cancer. Of 41 cancer deaths (in 149 deaths), 13 were due to esophageal cancer, the others being due to gastric cancer (n = 5) liver cancer (4), lymphoma (4), lung cancer (3) leukemia (3), breast cancer (2) and other cancers (7). In the comparison between A1, A2, B1 and B2 review results versus GS, the kappas were 0.82, 0.85, 0.78, and 0.85, respectively for all types. The multi-rater agreement for four reviews was 0.93 (95%CI: 0.85–0.99). In addition, the validity of VA in detecting upper GI cancers was 0.95 for all reviews.
To check the differences between documented and non-documented causes of death, we did the same analysis on 219 VA (Tables 4, 5). The numbers are comparable to those in Table 2, 3.
Discussion
Verbal autopsy seems to be a reliable and valid supplemental method to assess causes of death in the Golestan Cohort Study with comparable results in men and women and for patients from both rural and urban areas. One major reason for the usefulness of VA in the GCS may be the appropriate modifications made in the adult questionnaire prepared by WHO and the INDEPTH [37], [38] to adapt it to the local setting. Our results are consistent with those of most previous studies, showing that the VA is a reasonably valid tool to ascertain causes of death [12], [26], [27], [29], [32], [35], [36]. Some other studies are less supportive of the VA [34], and some even suggest that VA is not a very precise tool for detecting the leading cause of death among adults [6]. The reason for inconsistency in results of VA validation studies may be that VA is a developing method itself [40]. Thus, there are several variations of VA methodology and questionnaires, and some studies have not use the ICD coding system for their cause of death diagnoses. WHO has recently published instructions to improve the quality and standards for use of this method [41].
We used the VA method in the GCS, which is the first large-scale prospective population-based cohort study of cancer in the Middle East, to improve the accuracy of diagnosing the causes of death of cohort members. The majority of families in the rural area of Golestan Province prefer their family members to die at home after a diagnosis of end-stage cancer. About 60% of the decedents in the current study died at home, and only half of these had a prior hospital-based diagnosis; for the other half, the VA seems to be a promising approach to identify at least a general cause of death.
The kappa statistics obtained in the current study show that VA generates highly reliable results, at least among the 9 major categories of causes of death used in our study. Our results showed both high within–reviewer and between-reviewer reliability. We also found this method to be valid, with high sensitivity and specificity when compared to the gold standard diagnoses. Our results for making the diagnosis of different cancer subtypes also seem promising. This is especially true for upper gastrointestinal (UGI) cancers, the main focus of the GCS. Dysphagia, the main symptom of esophageal cancer, is very characteristic of this disease, and the availability of at least 10 UGI endoscopy clinics in the region, three providing free-of-charge endoscopy services to the GCS subjects, has made it possible to have accurate histologic diagnoses for almost all UGI cancers.
There are of course several caveats and methodological considerations related to this method. The gold standard was set by a combination of diagnoses made by two internists, or a third internist when the first two internists did not concur. These physicians used both VA and other clinical documents to adjudicate the results, but in 32% of the cases both the original internists and the reviewers in our study had only the VA answers to review. This lack of additional clinical documents in a third of the cases might raise concern that our validity estimates were falsely elevated, but this does not seem to have been the case, since these estimates were essentially identical in the full group of cases and in the subgroup which had additional clinical documents. On the other hand, our results may underestimate the potential sensitivity and specificity of the VA method because they tested the diagnosis made by each reviewer separately. In the actual GCS, at least two internists and perhaps a third one decides on the final diagnosis, and therefore the results may be more accurate than the judgment of just one physician.
In conclusion, our results suggest good reliability and good validity of verbal autopsy in determining the causes of death in a large-scale adult cohort study in a predominantly rural area in a developing country. These results add to the current literature on the use of VA for cohort studies in adult populations.
Acknowledgments
The authors wish to thank all the study participants and Behavarz for their cooperation. We also would like to show our appreciation to all follow-up team. We received special support from the Social Security Organization of Iran, Golestan branch.
Author Contributions
Conceived and designed the experiments: HK AE MN BA AP PB SMD RM. Performed the experiments: HK BA AP MB AH RM. Analyzed the data: HK AE FK FI CCA PDP PB PB SMD. Contributed reagents/materials/analysis tools: HK AE FK MN RS BA MB AH RM. Wrote the paper: HK AE FK MN RS AP MB AH FI CCA PDP PB PB SMD RM.
References
- 1. Lopez AD, Mathers CD, Ezzati M, Jamison DT, Murray CJ (2006) Global and regional burden of disease and risk factors, 2001: systematic analysis of population health data. Lancet 367: 1747–1757.
- 2. van Eijk AM, Adazu K, Ofware P, Vulule J, Hamel M, et al. (2008) Causes of deaths using verbal autopsy among adolescents and adults in rural western Kenya. Trop Med Int Health.
- 3. Franca E, de Abreu DX, Rao C, Lopez AD (2008) Evaluation of cause-of-death statistics for Brazil, 2002-2004. Int J Epidemiol 37: 891–901.
- 4. Arudo J, Gimnig JE, ter Kuile FO, Kachur SP, Slutsker L, et al. (2003) Comparison of government statistics and demographic surveillance to monitor mortality in children less than five years old in rural western Kenya. Am J Trop Med Hyg 68: 30–37.
- 5. Soleman N, Chandramohan D, Shibuya K (2006) Verbal autopsy: current practices and challenges. Bull World Health Organ 84: 239–245.
- 6. Yang G, Rao C, Ma J, Wang L, Wan X, et al. (2006) Validation of verbal autopsy procedures for adult deaths in China. Int J Epidemiol 35: 741–748.
- 7. Etard JF, Le Hesran JY, Diallo A, Diallo JP, Ndiaye JL, et al. (2004) Childhood mortality and probable causes of death using verbal autopsy in Niakhar, Senegal, 1989–2000. Int J Epidemiol 33: 1286–1292.
- 8. Genton B, al-Yaman F, Beck HP, Hii J, Mellor S, et al. (1995) The epidemiology of malaria in the Wosera area, East Sepik Province, Papua New Guinea, in preparation for vaccine trials. I. Malariometric indices and immunity. Ann Trop Med Parasitol 89: 359–376.
- 9. Meddings DR, Ronald LA, Marion S, Pinera JF, Oppliger A (2004) Cost effectiveness of a latrine revision programme in Kabul, Afghanistan. Bull World Health Organ 82: 281–289.
- 10. Setel PW, Sankoh O, Rao C, Velkoff VA, Mathers C, et al. (2005) Sample registration of vital events with verbal autopsy: a renewed commitment to measuring and monitoring vital statistics. Bull World Health Organ 83: 611–617.
- 11. Murray CJ, Lopez AD (1997) Global mortality, disability, and the contribution of risk factors: Global Burden of Disease Study. Lancet 349: 1436–1442.
- 12. Chandramohan D, Maude GH, Rodrigues LC, Hayes RJ (1998) Verbal autopsies for adult deaths: their development and validation in a multicentre study. Tropical Medicine & International Health 3: 436–446.
- 13. Gajalakshmi V, Peto R, Kanaka S, Balasubramanian S (2002) Verbal autopsy of 48 000 adult deaths attributable to medical causes in Chennai (formerly Madras), India. BMC Public Health 2: 7.
- 14. Gajalakshmi V, Peto R (2004) Verbal autopsy of 80,000 adult deaths in Tamilnadu, South India. BMC Public Health 4: 47.
- 15. Setel PW, Rao C, Hemed Y, Whiting DR, Yang G, et al. (2006) Core verbal autopsy procedures with comparative validation results from two countries. PLoS Med 3: e268.
- 16. Fantahun M, Fottrell E, Berhane Y, Wall S, Hogberg U, et al. (2006) Assessing a new approach to verbal autopsy interpretation in a rural Ethiopian community: the InterVA model. Bull World Health Organ 84: 204–210.
- 17. Quigley MA, Chandramohan D, Rodrigues LC (1999) Diagnostic accuracy of physician review, expert algorithms and data-derived algorithms in adult verbal autopsies. Int J Epidemiol 28: 1081–1087.
- 18. Winbo IG, Serenius FH, Dahlquist GG, Kallen BA (1997) A computer-based method for cause of death classification in stillbirths and neonatal deaths. Int J Epidemiol 26: 1298–1306.
- 19. Lawn JE, Wilczynska-Ketende K, Cousens SN (2006) Estimating the causes of 4 million neonatal deaths in the year 2000. Int J Epidemiol 35: 706–718.
- 20. Pourshams A, Khademi H, Malekshah AF, Islami F, Nouraei M, et al. (2010) Cohort Profile: The Golestan Cohort Study–a prospective study of oesophageal cancer in northern Iran. Int J Epidemiol 39: 52–59.
- 21. Pourshams A, Saadatian-Elahi M, Nouraie M, Malekshah AF, Rakhshani N, et al. (2005) Golestan cohort study of oesophageal cancer: feasibility and first results. Br J Cancer 92: 176–181.
- 22. Benara SK, Singh P (1999) Validity of causes of infant death by verbal autopsy. Indian J Pediatr 66: 647–650.
- 23. Snow RW, Armstrong JR, Forster D, Winstanley MT, Marsh VM, et al. (1992) Childhood deaths in Africa: uses and limitations of verbal autopsies. Lancet 340: 351–355.
- 24. Dowell SF, Davis HL, Holt EA, Ruff AJ, Kissinger PJ, et al. (1993) The utility of verbal autopsies for identifying HIV-1-related deaths in Haitian children. AIDS 7: 1255–1259.
- 25. Kamali A, Wagner HU, Nakiyingi J, Sabiiti I, Kengeya-Kayondo JF, et al. (1996) Verbal autopsy as a tool for diagnosing HIV-related adult deaths in rural Uganda. Int J Epidemiol 25: 679–684.
- 26. Mobley CC, Boerma JT, Titus S, Lohrke B, Shangula K, et al. (1996) Validation study of a verbal autopsy method for causes of childhood mortality in Namibia. Journal of Tropical Pediatrics 42: 365–369.
- 27. Quigley MA, Armstrong Schellenberg JR, Snow RW (1996) Algorithms for verbal autopsies: a validation study in Kenyan children. Bull World Health Organ 74: 147–154.
- 28. Chandramohan D, Rodrigues LC, Maude GH, Hayes RJ (1998) The validity of verbal autopsies for assessing the causes of institutional maternal death. Studies in Family Planning 29: 414–422.
- 29. Rodriguez L, Reyes H, Tome P, Ridaura C, Flores S, et al. (1998) Validation of the verbal autopsy method to ascertain acute respiratory infection as cause of death. Indian J Pediatr 65: 579–584.
- 30. Kalter HD, Gray RH, Black RE, Gultiano SA (1990) Validation of postmortem interviews to ascertain selected causes of death in children. Int J Epidemiol 19: 380–386.
- 31. Coldham C, Ross D, Quigley M, Segura Z, Chandramohan D (2000) Prospective validation of a standardized questionnaire for estimating childhood mortality and morbidity due to pneumonia and diarrhoea. Trop Med Int Health 5: 134–144.
- 32. Kahn K, Tollman SM, Garenne M, Gear JS (2000) Validation and application of verbal autopsies in a rural area of South Africa. Trop Med Int Health 5: 824–831.
- 33. Marsh DR, Sadruddin S, Fikree FF, Krishnan C, Darmstadt GL (2003) Validation of verbal autopsy to determine the cause of 137 neonatal deaths in Karachi, Pakistan. Paediatr Perinat Epidemiol 17: 132–142.
- 34. Anker M (1997) The effect of misclassification error on reported cause-specific mortality fractions from verbal autopsy. Int J Epidemiol 26: 1090–1096.
- 35. Chandramohan D, Maude GH, Rodrigues LC, Hayes RJ (1994) Verbal autopsies for adult deaths: issues in their development and validation. Int J Epidemiol 23: 213–222.
- 36. Datta N, Mand M, Kumar V (1988) Validation of causes of infant death in the community by verbal autopsy. Indian J Pediatr 55: 599–604.
- 37.
International Network of field sites with continuous Demographic Evaluation of Populations and Their Health in developing countries. Vision, Objectives, INDEPTH DSS sites. www.indepth-network.org.
- 38.
INDEPTH NetworkINDEPTH Standardized Verbal Autopsy Questionnaire. Available from: www.indepth-network.org/core_documents/indepthtools.htm.
- 39.
(1993) Statistical presentation. International Statistical Classification of Diseases and Health Related Problems–Tenth Revision.
- 40. Fottrell E, Byass P (2010) Verbal Autopsy: Methods in Transition. Epidemiol Rev.
- 41. Baiden F, Bawah A, Biai S, Binka F, Boerma T, et al. (2007) Setting international standards for verbal autopsy. Bull World Health Organ 85: 570–571.