Interobserver variability of ventilatory anaerobic threshold in asymptomatic volunteers

Cardiopulmonary exercise testing (CPET) is a key method in clinical diagnostics, in assessment of illness severity, in determination and monitoring of therapy, and in prognostic stratification. The performance of CPET and interpretation of the results are generally well standardized internationally [1, 2]. Among other CPET parameters, the ventilatory anaerobic threshold (AT) has been used in pre-operative risk assessment for many years [3], and recently the predictive value of AT determination has expanded from valvular and thoracic surgery to pancreatic and liver resection [4]. An increase of AT is a decisive response to exercise prescription in patients with chronic heart failure [5, 6], pulmonary disease [6] or stroke [7]. The importance of AT and its exact determination is acknowledged in recent guideline updates [8, 9].

AT is defined as the exercise level at which ventilation (VE) begins to increase exponentially relative to the increase in oxygen uptake (VO₂) [10]. There are wide differences in AT detection procedures and terminology [11, 12]. From a strict physiological viewpoint, there are two ventilatory thresholds, the first reflecting the transition from aerobic to anaerobic metabolism, and the second one from anaerobic metabolism to metabolic acidosis. Throughout this paper, we use the term ‘AT’ to refer to the first threshold. Methodological basics and practical guidelines for AT determination have been thoroughly reviewed elsewhere [9, 13].

The American Heart Association (AHA) Scientific Statement on CPET in adults [10] notes that confidence in determining AT may be increased by having two or three independent, experienced observers perform the calculation. If AT is calculated by software-based algorithms, it should be checked by an individual experienced in CPET and its assessment [10]. The repeatability of CPET parameters in healthy individuals [14] and intraindividual determination of the AT (test-retest comparison) in chronic disease cohorts [15, 16] is generally high. However, studies of interobserver variability of AT are limited. They refer to different populations, including healthy volunteers [17] and patients with heart failure [16, 18‐20], congenital heart failure [21], pulmonary arterial hypertension [22], chronic obstructive pulmonary disease (COPD) [23] and mixed etiologies [24, 25]. Furthermore, the published studies vary substantially in terms of sample size (from n = 6 to n = 428) and statistical interpretation. Studies comparing software-based AT determination with visual determination of AT by clinically experienced readers have produced heterogeneous results [23, 25].

Therefore, the aim of our study was to determine the interobserver variability of AT using CPET data from a large population-based epidemiological study (Study of Health in Pomerania [SHIP]).

Out of the 1,079 available data sets, 10 (0.9%; all showing an unstable cyclic breathing pattern) had non-detectable AT according to at least one physician reader and were therefore excluded from the analysis (7 [0.6%] were deemed uninterpretable by both physicians). Thirteen data sets with technical problems were also excluded. Thus, the final interobserver agreement analysis contained 1,056 data sets. The study participants represented a typical and asymptomatic but not strictly healthy population with a median body mass index of 27.3 kg/m²; 82.4% self-reported being physically active, 15.7% were active smokers and 5.0% had chronic bronchitis. In total, 18.9% had pulmonary obstruction (forced expiratory volume in 1 s/forced vital capacity < 70%). The median RER at AT was 0.82 (0.77–0.86), consistent with the reported training status and sure beyond exercise to exhaustion.

Our epidemiological study contained over 1,000 participants, and experienced readers were able to determine an AT in > 99% of cases. This proportion is consistent with the literature: despite adequate exhaustion and experienced readers, AT determination is not always possible [10], especially in individuals with periodic breathing patterns (as seen in our study) and in patients with chronic heart failure (AT was indeterminate in 16% of 1,679 tests in the Fix-Heart Failure-5 trial [16], 17% of 398 tests in the Heart Failure-ACTION trial [15], and 29% of 331 tests in a multicenter trial by Cohen-Solal et al. [19]). In contrast to these studies, we intended to determine AT in a typical population, and the demographic characteristics of our study population matched this criterion.

Our data show that after a training period with 400 exercise tests, the subsequent analysis of > 1,000 data sets by two physicians resulted in an ICC of 0.95 (95% confidence interval [CI]: 0.95–0.96), and VO₂@AT values within the 95% LOA had a difference of ±161 mL/min (SD_d ± 8%). In phase 1 (the first 522 data sets), the consensus discussion increased the ICC to 0.99 and lowered the difference within the 95% LOA to ±90 mL/min (SD_d ± 4%). A comparably high ICC of 0.98 (95% CI: 0.97–0.99) was shown in a study of 23 healthy volunteers undergoing cycle ergometer testing [23]. By contrast, analysis of 92 patients with COPD in the same study resulted in an ICC of 0.72 (95% CI: 0.60–0.81) [23]. Studies of AT in patient populations undergoing exercise testing with a treadmill ergometer revealed a range of ICCs, from 0.64 in 16 patients with chronic heart failure [20] to 0.85 in 445 tests of patients before vascular operation [25] and 0.88–0.97 (with three readers) in 13 children with congenital heart failure [21].

The software-based algorithm used in our study showed substantial differences in all statistical parameters when compared with manual assessment by physicians or medical assistants (Table 2). However, comparison with previously published studies is difficult, because the software-based AT determination was performed using different algorithms. Dubé et al. [23] used the LAB Manager version 5.3.0.4 (Cardinal Health, Höchberg, Germany) and Vainshelboim et al. [25] used the COSMED system (Rome, Italy) and special analytic software, whereas our study used VIASYS software (JLab Labmanager V5.32.0).

The absolute differences in the AT values calculated by the physician readers deserve special interest from the clinician’s viewpoint. Our study assessed AT in asymtomatic volunteers and showed a difference of 5 ± 82 mL/min (mean ± SD_d) between the two readers, which corresponds to a 95% LOA of ±161 mL/min. The study of patients with COPD mentioned above [23] showed a mean interobserver difference of 189 ± 115 mL/min (95% LOA: − 35-413 mL/min). In patients with chronic heart failure the mean interobserver difference was 13 ± 105 mL/min (95% LOA: − 194-220 mL/min) [16]. Other authors reported median interobserver differences, expressed in relation to body weight (mL/kg/min) [18], in percent [24], or in absolute terms (mL/min) [22]. The latter study analyzed 42 patients with pulmonary hypertension and showed differences according to the readers’ experience, with median differences in AT ranging from 20 mL/min (for very experienced readers) to 60 mL/min (for less experienced readers). The median interobserver difference overall was 36 mL/min (6.4%) [22].

Although interobserver differences between physicians were low in our study, larger differences were seen when comparing medical assistants’ readings with those taken by physicians (95% LOA: approx. 400 mL/min) and particularly when comparing software-based versus manually-derived readings (95% LOA: ±719–806 mL/min). This is partly contradictory to the recent recommendations of the AHA which include AT determination in the pre-surgical risk assessment algorithm for non-cardiac surgery [8]. Possible consequences of our calculated 95% LOA can be shown on the basis of this recommendation: if the AT of a patient weighing 70 kg was determined by a physician to be 10.0 mL/kg/min, which is lower than the accepted cut-off for mortality risk in abdominal surgical procedures (11 mL/kg/min) [29], it could also be determined as 13.4 mL/kg/min (+ 34%, upper LOA) by a medical assistant or even 15.1 mL/kg/min (+ 51%) by a software-based algorithm. In this context, at least the software-based AT calculation could lead to a patient being classed (perhaps wrongly) as having an acceptable mortality risk. The determination of AT before exercise training in rehabilitation may be less critical; however, physicians should be aware of the LOA before prescribing aerobic exercise training. The benefit of these individual exercise doses has recently been shown in cardiac [30] and pulmonary diseases [6]. Exact determination of the AT and an awareness of the variability in AT calculation will enhance the application of AT in future pre-operative risk assessment, rehabilitation and study design.

In summary, our analysis of CPET data from > 1,000 asymptomatic volunteers shows varying degrees of interobserver variability and supports the need for independent assessment of essential CPET parameters by more than one reader. Furthermore, the data presented here may inform the calculation of statistical power in future clinical studies. Interobserver variability should be considered when determining AT values for pre-operative risk assessment and before prescription of aerobic exercise training.