Cost-effectiveness analysis of three algorithms for diagnosing primary ciliary dyskinesia: a simulation study

Primary Ciliary Dyskinesia (PCD) is a genetically heterogeneous disorder that affects one in approximately 15,000 live births [1]. PCD is characterized by chronic sinopulmonary symptoms and development of bronchiectasis, recurrent otitis, male infertility and situs inversus [2]. Defective components of the ciliary axoneme (e.g. dynein arms) as well as dysfunctional regulatory or transport proteins have been implicated in the etiology of PCD and to date more than 40 genes have been found to be causative for PCD [3]. This genetic heterogeneity translates into a wide spectrum of ciliary structural and beating abnormalities and a diverse diagnostic and clinical phenotype. Patients with PCD usually present with chronic cough and rhinorrhea as well as recurrent infections of unknown aetiology. Some of them also present with situs abnormalities and in the case of older patients, with infertility or subfertility [2]. Bronchiectasis may develop already in childhood in some patients [4] and it is usually present in most adult PCD patients [5]. Late diagnosis is associated with a worse clinical picture and reduced lung function [6, 7], while several patients undergo surgical resection of lung segments to control lung infection, even before diagnosis is established [8]. Situs inversus is the only characteristic manifestation associated with PCD. With the exception of chronic cough and rhinorrhea, all other manifestations may not always be present and may be characterized by considerable variability in their severity [9‐11]. As a result, heterogeneity in the clinical picture presents a challenge to the clinician who needs to decide when to test for PCD and with which diagnostic test(s). Diagnostic approach is further perplexed by heterogeneity in the diagnostic features of the disease as respiratory epithelial samples from PCD patients exhibit diverse ciliary ultrastructure [12] and motility pattern [13] especially in the presence of infection [14].

Up to date diagnostic testing for PCD relies on a combination of tests which primarily includes nasal Nitric Oxide (nNO) [15], High Speed Video Microscopy (HSVM) [16, 17] and Transmission Electron Microscopy (TEM) [8, 18]. Measurement of nNO is considered the simplest and fastest among the PCD diagnostics tests as it only involves air suction from the nasal passage via an olive while the subject preferably maintains velum closure through active mouth exhalation against resistance [19]. The other two tests require brushing of the inferior nasal turbinate and the collection of an adequate sample of respiratory epithelial cells in order to allow for the assessment of ciliary motility using HSVM and ciliary ultrastructure using TEM [20]. As no single test has 100% sensitivity and specificity [21], which is further complicated by the fact that many centers lack either the equipment or expertise to perform all required tests, some of which are quite laborious and time consuming, different diagnostic algorithms for diagnosis of PCD have been adopted by diagnostic centers across the world [22]. Recently, nNO has been proposed as the screening test of choice in cohorts of patients with PCD-suspect manifestations due to its high ability to discriminate between PCD and non-PCD subjects [15, 23]. Although the cost of a (validated) chemiluminescence NO analyser is quite high (approximately €40,000 per piece), the recent development of handheld and cheaper electrochemical NO analysers [24] and publication of relevant technical guidelines by the American Thoracic Society (ATS) and the European Respiratory Society (ERS) [19] may further enhance the potential of nNO measurement to be used as a screening test in the clinical setting and especially in countries with limited resources or in areas that lack, or are distant from, PCD-specialist centers [25]. However, the use of a non-perfect screening test such as nNO in isolation may allow for some PCD patients with false negative results to be missed entirely or some non-PCD patients with false-positive results to undergo further diagnostic tests. For this reason, the diagnostic algorithm described as part of Standardized Operating Procedures for PCD diagnosis developed by the EU-funded Seventh Framework Program project BESTCILIA, in 2016, proposed standardized operating procedures for PCD diagnosis and a diagnostic algorithm which recommended that nNO should be performed in parallel with HSVM and confirmatory TEM assessment should follow in case of conflicting results (Additional file 1). Similarly, the recent ERS guidelines for the diagnosis of Primary Ciliary Dyskinesia also recommend a diagnostic algorithm which includes as a first step the parallel performance of both nNO and HSVM and confirmation with TEM in a second step [26]. The rationale of employing a diagnostic algorithm which proposes parallel performance of nNO and HSVM, is to take advantage of the ability of the one test to identify cases that the other test may have missed. Consequently, a positive result in both tests provides evidence that PCD is “highly likely” while a negative result in both tests, especially in the absence of very strong clinical suspicion, provides evidence to consider PCD diagnosis as “extremely unlikely” [26]. Nevertheless, such algorithms require the performance of a significantly higher number of nasal brushings for HSVM and result in higher costs compared to algorithms that only require the performance of a confirmatory test (HSVM or TEM) following a positive screening test.

To better illuminate the decision-making process, the overall diagnostic accuracy of each algorithm, the associated costs as well as the resulting health benefits for PCD patients, need to be addressed and compared. This study aimed to evaluate the diagnostic accuracy, the cost-effectiveness and incremental cost-effectiveness of three distinct diagnostic algorithms for patients referred for PCD diagnostic testing across the European Union through a probabilistic decision analysis framework.

The model output for TP, FN, TN and FP and estimates of net sensitivity, net specificity, net positive predictive value and net negative predictive value for the application of each diagnostic algorithm in a hypothetical cohort of 1000 patients suspected of PCD is presented in Table 2. Table 3 compares mean diagnostic costs with the number of PCD cases identified and reports relevant CERs and ICERs. Deterministic comparison for mean costs and effects demonstrated that the nNO/HSVM+TEM was the most effective algorithm but also the costliest (313 PCD cases identified/year, 209 thousand €/year). nNO + HSVM was the second most effective (273 PCD cases identified/year, 136 thousand €/year) while nNO + TEM was the least effective (198 PCD cases identified/year, 150 thousand €/year). The most cost-effective algorithm was nNO + HSVM with a CER of €653/PCD case identified, followed by nNO/HSVM+TEM (€678/PCD case identified) and nNO + TEM (€975/PCD case identified). The cost effectiveness frontier in presented in Fig. 3 and the resulting ICER for nNO/HSVM+TEM compared to nNO + HSVM, the second most effective algorithm, is €2097 per additional PCD case identified. The nNO + TEM algorithm is dominated (simple dominance) by nNO + HSVM as it is more expensive but less effective compared to nNO + HSVM. Figure 4 presents the cost-effectiveness acceptability curve (CEAC) for nNO/HSVM+TEM. The CEAC demonstrates the uncertainty in the estimation of ICER and provides information about the probability of nNO/HSVM+TEM being more cost effective compared to nNO + HSVM for a range of potential monetary amounts (termed willingness to pay (WTP) thresholds) that a decision maker might be willing to pay to correctly diagnose an additional PCD case. For a WTP threshold equal to €2500 the probability of nNO/HSVM+TEM being cost effective is over 70% and for a WTP threshold equal to €3500 the probability is over 97%. The results of one-way sensitivity analyses demonstrated that the modelled mean ICER for nNO/HSVM+TEM is primarily affected by changes in the input value for HSVM sensitivity, followed by the changes in input values for the prevalence of PCD among suspect patients. Changes in the input values of other modelled parameters had smaller effects on the ICER (Fig. 5). Results of secondary analysis are presented in Additional file 3.

The high genetic heterogeneity that characterizes PCD and the resulting inability to rely on a single test to confirm or exclude diagnosis of the disease has led to increased research interest in specialized diagnostic testing for PCD in recent years. This study compares three diagnostic strategies currently in use for diagnosing PCD and reports on their effectiveness and cost-effectiveness under a societal costing perspective. Data were drawn primarily from meta-analyses of diagnostic effectiveness or published estimates from large studies and were synthesized in a probabilistic cost effectiveness model.

The results presented here demonstrate that when the effectiveness outcome is defined as the number of PCD patients identified, nNO/HSVM+TEM is the most effective diagnostic algorithm followed closely by nNO + HSVM. Both nNO/HSVM+TEM and nNO + HSVM are significantly more effective compared to the third diagnostic strategy evaluated, nNO + TEM. Mean estimates of CERs demonstrate that nNO + HSVM was the most cost-effective option and a decision maker should expect to pay on average an amount equal to €2097 per additional case identified if nNO/HSVM+TEM is implemented. Whether the effectiveness outcome is defined as the number of PCD patients identified or as the number of QALYs saved nNO/HSVM+TEM was still the most effective algorithm followed by nNO + HSVM and nNO + TEM. Nevertheless, the results of the extended model, which are expressed in Euros per QALY saved, demonstrate that all three diagnostic algorithms appear to be very cost-effective. Compared to no screening, the cost per QALY gained for the three diagnostic algorithms examined here ranged from €6674 to €12,930, an estimate which is lower than WTP thresholds commonly used by regulatory authorities around the world. Such WTP thresholds range between £20,000 and £30,000 per QALY saved in the UK [37] or the more conventional WTP threshold of $50,000 per QALY saved, commonly used in the US [38] or even more recently suggested WTP thresholds in the range of $100,000 to $200,000 per QALY [39].

Diagnostic algorithms including nNO measurement during VC as an initial screening could be cost-effective. However, our results demonstrate that nNO screening is more effective when the confirmatory test is HSVM and not TEM. Although in the past TEM was considered the gold standard [13], it is now known to miss an important fraction of PCD patients [32], mainly those with biallelic mutations in DNAH11 gene [40] and those with specific ultrastructural abnormalities (nexin link defects) that are not easily detectable by standard TEM [41]. Furthermore, it requires access to a specialized lab with personnel experienced in staining and interpretation of TEM micrographs and consequently involves considerable resource allocation [42]. At the same time, TEM studies are usually time consuming and results are frequently obtained and communicated to patients considerably later than results of other tests thus contributing to patient distress [43]. HSVM is easier, considerably faster and cheaper than TEM as it is usually performed on the same day following nasal brushing and the equipment required consists of standard microscope, a high speed video camera and a standard computer loaded with specialized software. It has also been reported to be a highly sensitive and specific test [35] thus it significantly outperforms TEM as a confirmatory test both in terms of overall effectiveness and cost. However, extra caution is required with HSVM as it may be affected by observer subjectivity and non-PCD specific findings which may interfere with the motility interpretation [22]. Overall, the parallel performance of two highly specific and sensitive tests such as nNO and HSVM during the first step of the diagnostic algorithm, followed by confirmatory TEM in only the few cases of conflicting findings, results in the identification of most PCD patients and does not require the performance of the more expensive and time consuming TEM analysis for the largest part of the cohort of suspect patients.

In this study we did not include diagnostic algorithms that included immunofluorescence (IF) and/or genetic testing for PCD. Although a recent study has reported the first diagnostic accuracy and cost estimates for immunofluorescence testing in PCD [44], the use of this test is still very limited (as it is performed only in a small number of few highly specialized centers around the world). Genetic testing, on the other hand, is available in many centers around the world. However, as yet, there is little standardization of procedures for the conduct and interpretation of results. Different centers may use different technologies and may not test for the same number of genes [45, 46]. Thus estimation of the effectiveness or the cost of genetic testing as diagnostic for PCD was not possible at this stage and it was not included in the diagnostic algorithms considered in our analysis. This approach is in line with the recent guidelines published by the ERS where genetic testing was recommended as a last step following abnormal TEM primarily for further characterization of the underlying defect or as a final diagnostic test if all other tests were inconclusive. For immunofluorescence there was no ERS recommendation towards its use as a diagnostic test given the scarcity of evidence [26].

The main strength of this study is that it makes use of evidence-based estimates and individual good quality studies on the diagnostic accuracy of nNO, TEM and HSVM and the prevalence of PCD among cohorts of referred suspect patients. With the use of Bayes’ Theorem, it was possible to estimate the diagnostic effectiveness of sequential tests and to compare the effectiveness of diagnostic algorithms instead of simply comparing the effectiveness of isolated tests, as had been done in the past. In addition, our analysis of the costs involved in diagnostic testing followed standard approaches for economic analysis of healthcare procedures [28] and made use of the extensive literature on the effort, equipment and consumables involved in the performance of nNO [47, 48], HSVM [13, 35] and TEM [18]. Based on this evidence, we were able to calculate effectiveness and economic outcomes (number of PCD patients identified, total diagnostic costs) as well as robust CERs, ICERs and identify the cost-effectiveness frontier.

Nonetheless, this study also has some limitations. In the main analysis, although our data on diagnostic accuracy are mostly based on meta-analyses of well conducted studies, these are characterized by a degree of heterogeneity [32, 33]. On the other hand, our data on diagnostic cost parameters are primarily based on realistic estimates of current market values, although these may not be uniform across all EU countries. The one-way sensitivity analyses for the diagnostic ICER for NO/HSVM+TEM demonstrates that our results are most sensitive to variations in HSVM sensitivity and PCD prevalence among suspect patients. A recent, large study on diagnostic accuracy of HSVM reported a sensitivity of 100%, which is in line with the value used in our model [49]. Nevertheless, it is possible that PCD prevalence among referred suspect patients varies considerably between countries, as different countries may utilize different diagnostic protocols and referral patterns [20, 50, 51]. Even so, these disparities between countries are expected to be reduced in the future with the increasing use of clinical scoring tools [52], the intercalation between PCD clinicians in international networking projects such as the BEAT-PCD COST project [53] and the establishment of European Reference Networks for rare diseases including PCD (ERN-LUNG) [54].

Most limitations of this work however, relate to the considerable uncertainty of the parameters used in the secondary analysis and for this reason the results of the basic and extended model are presented separately. As a result, caution is advised before generalizing the results of this study, especially those regarding the extended model. Another limitation of the extended model is that despite empirical evidence about various approaches for the treatment of PCD, at the moment there are no widely recognized PCD-specific treatment protocols. The efficacy of a few treatment approaches are now under investigation through randomized control trials, for example, those now underway on the effect of azithromycin for antibiotic prophylaxis [55]. Furthermore, there are no published estimates of the annual (or lifetime) cost of various options for treatment of PCD. Although we used credible sources to estimate patient associated cost [56] and cost of each procedure (resource cost) [57‐59], we had to rely on our own experience with the disease to characterize the typical frequency of treatment (resource use). To address this limitation, the underlying uncertainty in each parameter was characterized and included in the model. Through Latin Hypercube sampling and Monte Carlo analysis, these uncertainties in individual parameters were propagated through the model and are reflected in the uncertainty in final model outputs.

Evidence about treatment costs is especially weak. We found no evidence of the cost of treatment of PCD patients who remain undiagnosed; and only limited evidence about the cost of treatment of PCD patients who are properly diagnosed. A sensitivity analysis was conducted to determine whether differences in the overall costs of treatment of diagnosed and undiagnosed PCD patients affected the estimates of cost-effectiveness from the extended model. The overall order of diagnostic algorithms was not affected and nNO/HSVM+TEM was the most cost efficient algorithm in all scenarios. However, the magnitude of the difference in the cost effectiveness of the three algorithms was significantly affected, with nNO/HSVM+TEM becoming relatively more cost-effective when it was assumed that the cost of treating undiagnosed PCD patients was at least 3 times greater than the treatment cost for properly-diagnosed PCD patients. This highlights the importance of future studies which address the economic cost of treatment in PCD patients before and after diagnosis.

We encountered a similar lack of data on the impact of PCD on life expectancy and the patients’ valuation of health status (health utility). Currently, PCD is considered a disease characterized by normal or near-normal life span, although cases of premature mortality among PCD patients are reported in the literature [8, 60]. To date, no study has reported on patients’ life expectancy and this lack of information could be attributed to the fact that PCD has been studied primarily in small cohorts in the pediatric setting. The recently established prospective international PCD registry [61], which now includes several thousands of pediatric and adult patients, is expected in the next few years to provide data on disease progression and life-expectancy. Likewise, to date no study has reported on health state utilities in PCD and thus we used in our calculations data on health utilities from mild Cystic Fibrosis patients that have been previously reported to have similar clinical severity with PCD [62]. The one-way sensitivity analyses in the extended model, which included treatment costs and outcomes, demonstrated that the most important parameter impacting the CER of nNO/HSVM+TEM was PCD health utility followed by loss of productivity, reduction in life expectancy and antibiotics cost. In order to further improve our understanding of the disease and better inform the development and improvement of guidelines for PCD diagnosis and treatment, future studies aiming to assess the real value of cost-of-illness, healthcare utilization estimates and health state utilities are urgently needed.

Across the world, many PCD diagnostic centers follow a variety of algorithms for diagnosing PCD and, most likely, in some low income countries, there is a complete lack of specialized diagnostic testing. The results of this study suggest that a diagnostic algorithm which includes nNO during VC as a screening test followed by confirmatory HSVM identifies approximately 85% of PCD patients with a mean CER of €653per PCD case identified. The algorithm which maximizes the number of PCD patients identified involves parallel performance of nNO and HSVM as the first step, followed by TEM as a confirmatory test for the few cases where nNO and HSVM yield conflicting results, with a corresponding ICER of €2097 per additional PCD patient identified. Decision analysis methods and the evidence from this study can inform the dialogue on evidence-based guidelines for PCD diagnostic testing. Future studies in understudied aspects of PCD relating to quality of life, treatment efficiency and associated costs are urgently needed to help the better implementation of these guidelines across various healthcare systems.