A systematic approach to evaluate practice-based process- and outcome data applied to the treatment of neovascular age-related macular degeneration

Change in VA in the treated cohorts is shown in Table 2. Visual acuity improved in eyes treated with the anti-VEGFs ranibizumab and bevacizumab in the first months, to slightly decline in long-term follow-up.

We compared outcomes in treated groups with VA in the no-treatment group. Table 2 shows VA in non-treated eyes declining over time. We noted no difference between PDT and no treatment. There was a benefit of > 0.4 LogMAR in the anti-VEGFs over no treatment (at 6 to 24 months). We also compared the outcomes to those in RCTs and other daily-practice outcome studies, shown in Table 3.

Table 1 shows a difference in the type of CNV lesions between the PDT and anti-VEGF groups. After adjusting for lesion type as a confounder, there was no significant effect of PDT over no treatment, and anti-VEGFs remained equally effective.

Disease severity is prognostic for treatment success. In nAMD, severity can be derived from size of the lesion and the presence of blood, shown in Table 1. Diagnostic accuracy is an important process variable. In re-evaluating FAs, we found that a total of 104 patients had not received an appropriate diagnosis of nAMD, or did not have an appropriate indication for treatment. Examples of inappropriate diagnosis were CNV without other hallmarks of AMD such as drusen (e.g. CNV in myopia) and central serous chorioretinopathy. The number of treatments is another important process variable. The number of initial injections was lower, and the time between injections was longer than in current strategies. This may worsen treatment outcomes.

Relevant outcomes can be measurements, expected to change due to treatment. Visual acuity and quality-of-life are examples in the case of nAMD. For comparison purposes, it is useful if the outcome was used in trials. Intermediate outcomes can also be relevant. These reside in the causal pathway from treatment to outcome. Outcomes can then be improved by improving an intermediate outcome. Examples of intermediate outcomes in the case of nAMD are the presence of CNV on FA, edema on OCT, or the presence of blood on the macula.

We used VA as the relevant outcome. It significantly impacts quality-of-life and guides treatment decisions. It is frequently recorded in medical visits, and allows for comparison with RCT results. OCT was not in clinical use during the follow-up of the no-treatment cohort. FA was not commonly repeated in the time-periods before the introduction of effective treatment, and after OCT was introduced. Quality-of-life was not useful in this retrospective analysis, as it is rarely measured in daily practice. There is a risk of bias that is introduced when converting the clinical measure, Snellen VA, to a scale that is suitable for parametric analysis, such as LogMAR. Poor agreement between Snellen and ETDRS charts has been shown previously [28]. The difference tends to be more pronounced in patients with poor vision.

Outcomes need to be compared, to identify differences which may indicate opportunities to improve diagnosis and treatment. Relevant references and their outcomes depend on the research question. To assess intervention effect, a reference outcome is needed from an untreated cohort. Disease severity and prognosis without treatment should be similar between the treated and the untreated cohort. Selecting an untreated historical cohort from the same setting, probably reflects the characteristics of treated nAMD patients well. This historical cohort should be selected from a period as recent as possible because, for instance, changes in the general population may affect patient characteristics such as average age.

Non-treated patients were selected from the 1992 to 1997 period. We assumed that this group reflected the treated patients’ disease severity and prognosis (had they not been treated). However, there were differences in age, lesion subtype, presence of blood, and fellow eye status (Table 1). These are prognostic factors for nAMD, and therefore may affect outcomes. Importantly, there was no difference in mean VA between groups. As expected, VA decreased over time in the no-treatment group (Table 2). There was no difference in outcome between PDT and no treatment. There was a marked difference in outcome with the anti-VEGFs. A lower prevalence of classic CNV in the no-treatment group may be an explanation. This lower prevalence may imply that the no-treatment group does not reflect the theoretical PDT groups’ prognosis (without PDT). However, even after adjusting for baseline characteristics such as CNV type and baseline VA, there was no difference in outcome between PDT and no treatment, and the anti-VEGFs still had statistically significant better outcomes compared to the no-treatment group at 10–12 months.

Comparing a treated group with an untreated group from the same time period may seem attractive. However, this introduces a high risk of confounding by indication, inducing interpretation errors. In nAMD, patients with mild disease and good prognosis may be left untreated. Patients with severe disease with poor prognosis, such as severe macular hemorrhage, may also be left untreated. Comparing anti-VEGF-treated patients with a no-treatment group that includes such patients will not produce meaningful results, as disease severity and prognosis without treatment are not comparable. We decided not to compare with an untreated group from the same time period.

Outcomes can also be compared between treatments. An advantage is that the indication for treatment is more likely to be similar. However, in nAMD, PDT and anti-VEGF have different indications. Therefore, we compared the anti-VEGFs and PDT, and adjusted for baseline differences in, for example, CNV type. We found that bevacizumab and ranibizumab had better outcomes. There was no difference in outcome between bevacizumab and ranibizumab (Table 2).

Daily-practice outcomes can be compared to RCT outcomes. A difference in outcome may provide indications for improving clinical practice. There are important differences between clinical practice and RCTs. For instance, VA may not be measured as accurately in daily practice as in RCTs, while such measurements do determine the decision-making by clinicians. In RCTs, patient selection is generally strict. This can yield a population with better established diagnosis, which could lead to a better treatment outcome. Inclusion criteria may select patients that are more sensitive for improvement with a specific intervention, or with a more severe disease status. Follow-up can be more frequent, with better equipment and higher-quality assessment of re-treatment criteria. Compliance-enhancing strategies could have been implemented. To assess for differences in patient populations’ disease status, the untreated daily-practice group can be compared with the RCT’s placebo group (Table 3). The VA change at 12 months in our no-treatment group was worse than in RCT placebo groups. This implies that prognosis was worse in our patients. Next, the outcomes of treatment in daily-practice and in the RCTs can be compared. For PDT, the daily-practice outcome appeared worse than that in RCTs (Table 3). However, offsetting PDT outcomes to the corresponding untreated groups (yielding the ‘effect size’) produces benefits in VA in both daily practice and RCTs. However, outcomes in daily practice were still worse than in RCTs and not statistically different from no treatment. This suggests a need for further assessment of differences in patient and treatment characteristics. This could indicate possibilities to improve patient selection or improve the treatment protocol.

Regarding anti-VEGF treatments, outcomes appeared worse in daily-practice than in RCT treatment arms (Table 3). However, the effect size over the non-treated group is greater in our study than in RCTs. Relevant prognostic factors were taken into account. Thus, effectiveness of anti-VEGF appeared as good as in RCTs. The outcomes of our study align more with the treated patients in the PIER and SAILOR study, which showed a lower gain in VA from baseline compared to earlier RCTs. In these studies, patients received a reduced number of treatments due to extended follow-up intervals of 3 months. A reduced number of injections by itself will not lead to a lower VA if monthly follow-up takes place and injections are given pro re nata based on adequate criteria [8, 29‐31]. However, to achieve optimum outcome, 3 months follow-up period is too long [32].

Finally, outcomes in daily practice can also be compared to outcomes of other retrospective outcome studies. Those studies found VA to stabilize or decline after 1 year of anti-VEGF treatment [33‐35]. Tao et al. found a VA decline after 2 years [36]. In contrast, a majority of outcome studies found a VA improvement of at least − 0.1 logMAR after 1 year treated with anti-VEGFs (Table 3). A very large outcome study in over 11,000 patients showed VA stabilization over a period of 3 years with a yearly median number of 4–5 injections and 8.2 to 9.2 outpatient visits [13].

In comparing outcomes between cohorts, it is important to consider the cause of the observed difference. For example, is a difference in VA after treatment with PDT or anti-VEGF is really caused by the difference in treatment? To answer this question, confounding variables need to be taken into account. Relevant variables are prognostic factors, or factors that modify the effect of treatment. In comparing PDT and anti-VEGF, we needed to adjust for CNV lesion type, since PDT was not indicated for all lesions, and we observed a difference in lesion types between PDT and anti-VEGF groups (Table 1). Furthermore, lesion type determines the prognosis [37‐39]. We found that, after adjusting for lesion type, occult or classic CNV, there was no effect of PDT.

Comparing the effect size of ranibizumab treatment over no-treatment in daily practice to the effect size in an RCT, effect-modifying variables need to be taken into account. Lesion type and size can be effect-modifying variables. The effect size is greater in classic CNV and in small lesions than in occult CNV and in larger lesions [38]. In considering effect modification, a stratified analysis is needed based on varying levels of the effect-modifying variable. This analysis shows whether the effect size differs between different levels of the variable. Lesion type as an effect-modifier was taken into account to assess the difference in outcome between PDT and anti-VEGF.

Confounding by indication is a special case of confounding. In comparing PDT or anti-VEGF-treated patients to untreated patients from the same period, a difference in outcome can be due to confounding by indication, even after taking other confounding variables into account. Ophthalmologists consider patients eligible for treatment based on prognosis or expected treatment effect. Therefore, a priori, the outcome is different between treated and untreated patients from the same period. Comparing two treatments with different selection criteria may introduce similar confounding. In our study, after taking confounding factors (lesion type) into account, there still was a difference in outcome between PDT and anti-VEGFs. To a high degree of validity, anti-VEGFs were more effective in daily practice than PDT, unless the difference is due to confounding variables that were not measured, or are difficult to measure, such as confounding by indication.

The ultimate goal of outcome- and process-analysis is identifying factors to improve in the process of patient selection, diagnosis and treatment. In the previous sections, we outlined the relevant factors that determine outcomes. We described methods to determine the effectiveness of the treatments, the effect sizes, corrected for confounding, with the major pitfalls. In the following we describe the next step of addressing several process factors that commonly affect the outcome: severity of the disease at first presentation; physician-dependent process variables; and patient-dependent process variables.

The severity of the disease determines the prognosis, the appropriate intervention, and the outcome. We identified several nAMD severity factors: lesion type; lesion size and presence of blood in the lesion (see Table 1). These variables determine the prognosis without treatment, and the effect of treatment with e.g. PDT and anti-VEGF [37, 38]. Earlier referral to an ophthalmologist may lead to a lower severity of the disease at baseline. Literature indicates that delay access to a general practitioner or late referral to an ophthalmologist occurs and influences the outcome of nAMD treatment [40]. Earlier access to an ophthalmologist is a focus for improvement if the outcome of nAMD is to be improved.

Assessing physician-dependent process-variables includes assessing whether the diagnosis and the indication for treatment were appropriate, and whether the treatment or intervention was correctly timed and applied. We re-assessed the diagnosis and the indication for treatment (subfoveal CNV due to nAMD) in our study. They were not correct in 104 patients. Classification of CNV type or assessment of retreatment criteria tends to differ between ophthalmologists [41, 42]. Treatment is unlikely to be effective without the presence of CNV. In addition, literature indicates that the timing of the intervention may impact the outcome [43, 44]. Improving diagnosis and indication appropriateness will improve outcomes. Measuring the incidence of diagnostic error in everyday practice is an essential requirement of a comprehensive quality management program [45]. Having diagnostic confirmation before commencing treatment is paramount. Treating a patient for a disease based on inaccurate diagnosis leads to unpredictable results and delays proper treatment.

Sometimes, treatment schedules in clinical practice may not be according to the “state-of-the-art.” In this regard, improving care requires a definition of the state-of-the art by guidelines or clinical trials. In our study, PDT was given according to the TAP protocol with a follow-up period of 3 months and a maximum of four PDT treatments per year [2]. Early after the introduction of anti-VEGF treatments, some repeated treatments with anti-VEGF were applied with longer intervals as compared to early RCTs. In the RCTs, monthly treatments or PRN treatment were applied [4, 7, 11, 46‐50]. Using a different treatment protocol may not necessarily worsen outcomes. Ophthalmologists may individualize treatment to optimize outcomes, and they may take into account a strategy preferred by the patient, or may prioritize avoiding adverse events over other potential benefits of treatment. However, we now know that three-month intervals between anti-VEGF injections of bevacizumab as a standard is too long and adversely affects the outcome [11].

Patient-dependent process variables involve the timely reporting of symptoms by the patient, and adherence to prescribed medication, to follow-up appointments and to life-style advice. We did not specifically address these factors in our study, as it was based on a medical record review. In nAMD, timely reporting of symptom recurrence and timely follow-up are known to be important patient related-process variables that impact the outcome [43, 51, 52].