Diagnostic accuracy of lung ultrasound for SARS-CoV-2: a retrospective cohort study

A pandemic of severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2), which causes the syndrome known as COVID-19 has led to over 75 million infections and over 1 million deaths as of December 2020 [1]. Early diagnosis is vital to enable early isolation, reduce ongoing transmission, and facilitate clinical decision-making. SARS-CoV-2 RT-PCR is the current diagnostic gold standard but has an estimated sensitivity of 75% [2, 3], and may take multiple days to result in some high-demand areas [4]. In the absence of sufficient RT-PCR availability, thoracic computed tomography (CT) has been used to detect the characteristic peripheral ground-glass opacities of COVID-19 pneumonia [5‐8]. However, CT is not an optimal screening tool as it exposes patients to ionizing radiation, requires extensive decontamination, and is not readily available in many resource-limited situations [9‐11].

Lung point-of-care-ultrasound (POCUS) has been proposed as a screening tool for COVID-19, and may offer several advantages. For non-SARS-CoV-2 interstitial syndromes, lung ultrasound has better sensitivity and specificity than chest radiograph and CT [5, 12, 13]. In confirmed cases of SARS-CoV-2 infection, lung POCUS evaluation of disease severity has correlated well with CT chest [14]. In addition to superior performance, lung POCUS is cost-effective [15, 16], portable, provides real-time data, and does not require ionizing radiation. POCUS supports infection control efforts by minimizing the number of healthcare workers exposed to patients under investigation (PUI) for COVID-19, and decontamination of ultrasound machines is relatively quick and easy compared with other imaging modalities [17]. Thus, if proven reliable, lung POCUS would allow for expedited and cost-efficient diagnosis of COVID-19 in hospitals, in the community, in resource-limited settings, and in surge situations when RT-PCR or CT chest availability is limited.

Preliminary studies have prompted the World Health Organization (WHO) to endorse ultrasound for use in the diagnosis of COVID-19, although this endorsement is noted as being based on weak preliminary evidence [18‐20]. Characteristic findings of the COVID-19 syndrome on lung POCUS include a thickened or irregular pleural line, confluent B-lines, and subpleural consolidations [19, 21‐23]. These findings correlate closely with those observed on CT [22], and demonstrate promise in trending clinical progression from onset to peak to resolution [21, 24]. Descriptive studies [25‐31] have revealed the potential utility of lung POCUS but are limited by small sample sizes, lack of in-depth statistical analysis, and limited evaluation of patient characteristics that impact the utility of lung POCUS. To facilitate optimal application of lung POCUS for the diagnosis of COVID-19, we report the diagnostic accuracy of lung POCUS compared the criterion standard of SARS-CoV-2 RT-PCR test.

A convenience sample of COVID-19 PUI > 18 years old evaluated between March 16, 2020 and May 16, 2020 who had lung POCUS recorded as part of their routine emergency department or inpatient care were included in this retrospective cohort study conducted at two urban academic tertiary care centers. PUI designation was identified by the presence of a SARS-CoV-2 RT-PCR order by the treating physician. Criteria for SARS-CoV-2 testing during the study period included either exposure to COVID-19 or report of at least two of the following symptoms: fever, acute cough, sore throat, dyspnea, myalgias, or loss of taste or smell. Details of all patients included in the study as well as alternate diagnoses are in Additional file 1: Figure S1. This study was performed in accordance with the Declaration of Helsinki. This human study was approved by Johns Hopkins Institutional Review Board—approval: IRB00255571. Adult participant consent was not required because this was a retrospective study.

The institutional POCUS database (Qpath Ultrasound Manager, Telexy Healthcare, Blaine WA) was queried to identify all lung POCUS examinations performed on COVID-19 PUI during the study period. All lung examinations performed during routine evaluation for COVID-19 by residents and faculty credentialed in the use of lung POCUS in the departments of Emergency Medicine and Internal Medicine were included. Study team members performed 89.1% of ultrasound studies, and all studies were evaluated for image adequacy by study team members blinded to clinical information. No dedicated training in lung POCUS or COVID-19-specific ultrasound was provided. A 12-field protocol was encouraged for POCUS users across the institution, including views of the bilateral anterior lung (L1–L2/R1–R2), lateral lung (L3–L4/R3–R4), and posterior lung (L5–L6/R5–R6) (Fig. 1). Ultrasound scans were acquired using equipment from Sonosite (Bothell WA), GE Healthcare (Waukesha WA), Philips (Bothell WA), and EchoNous (Redmond WA) (Additional file 2: Table S1).

Demographic characteristics including age, gender, race, duration of symptoms before POCUS examination, duration of symptoms before RT-PCR, RT-PCR result, body mass index (BMI), and history of comorbid conditions were recorded in a REDCap database (Vanderbilt University, Nashville TN). Two study team members credentialed in lung POCUS and blinded to all clinical information interpreted the lung POCUS studies. Reviewers did not evaluate POCUS examinations that they acquired. The blinded reviewers recorded their assessments in a separate REDCap tool based on a Lung Ultrasound Score described for use in COVID-19 pneumonia [21]. Each lung field was assessed for abnormal findings including pleural irregularity (0 points), multiple discrete B-lines (1 point), confluent B-lines (2 points), subpleural consolidations (3 points), and pleural effusion (0 points). Examples of these findings are presented in Additional file 3: Video S1, Additional file 4: Video S2, Additional file 5: Video S3, Additional file 6: Video S4, Additional file 7: Video S5, Additional file 8: Video S6. The reviewers also provided a summary assessment of whether the POCUS examination was consistent with COVID-19. A third blinded reviewer adjudicated any disagreements between reviewers. The reviewer whose ultrasound COVID-19 determination was concordant with the adjudicator was retained in the analysis.

The test characteristics and discriminative power of the Lung Ultrasound Score relative to the SARS-CoV-2 RT-PCR result were calculated. The point totals for all lung fields were summed to yield a total Lung Ultrasound Score for each reviewer. Mean Lung Ultrasound Score was calculated as the mean score between the two ultrasound reviewers. POCUS examinations were compared with the institutional gold standard RT-PCR test. A secondary analysis of complete examinations (12 acquired lung fields) was also performed. Means and proportions were compared using Student’s T test and Chi-squared testing. Inter-rater reliability of lung POCUS interpretation was assessed using percent agreement, percent positive agreement, and kappa statistics. The sensitivity, specificity, negative predictive value, and positive predictive value associated with overall COVID-19 status determination by ultrasound and individual lung fields were also calculated. Agreement between SARS-CoV-2 RT-PCR testing and lung POCUS was evaluated using kappa statistics. The area under the receiver operator curve was then calculated for the Lung Ultrasound Score, and a cutoff proposed based on the Youden J statistic [32]. Scoring frameworks assigning negative points for pleural effusions were assessed, and the framework with the highest discriminative power (pleural effusion − 3 points) was included for analysis.

Patients included in the cohort (n = 174) had mean age of 53.1 years, were 52.9% male, and 53% had positive RT-PCR testing for SARS-CoV-2 (Table 1). 77% were admitted to the hospital, 26.4% were admitted to the intensive care unit, 5.7% required high-flow nasal cannula oxygen support, and 14.4% required intubation (Table 1, Additional file 2: Table S2). Other details of hospital admission and level of care are reported in Additional file 2: Table S2. Pathological lung findings were identified more frequently in all lung fields in patients who tested positive for SARS-CoV-2 RNA compared to those who tested negative (p < 0.001). Bilateral lung involvement was more common among those with a positive RT-PCR result (78%), compared to those with a negative RT-PCR result (26%).

Test characteristics for lung POCUS in the diagnosis of COVID-19 are detailed in Table 2. Compared to the standard of RT-PCR testing, lung POCUS had a sensitivity 86.0% and specificity 71.6%, with negative predictive value (NPV) 81.7% and positive predictive value (PPV) 77.7%. Test characteristics for the involvement of more than one lung field, posterior lung field involvement, and bilateral lung field involvement are also reported in Table 2. Examinations with multiple discrete B-lines had a sensitivity 86.0%, specificity 54.3%, NPV 77.2%, and PPV 68.4% while examinations with confluent B-lines had a sensitivity 43.0%, specificity 98.8%, NPV 60.2%, PPV 97.6%. Lung POCUS was more accurate in patients with more significant oxygen requirement or requiring higher level of care (Table 3).

Patients with positive SARS-CoV-2 RT-PCR testing had higher Lung Ultrasound Scores than those with negative tests (9.4 ± 5.5 versus 2.6 ± 3.2). There was high discrimination (AUC 0.84, 95% CI 0.78, 0.90) of the Lung Ultrasound Score for COVID-19 syndrome (Fig. 2). The test characteristics associated with various cutoff points are reported in Fig. 2. A Lung Ultrasound Score of 2 points maximized sensitivity (sensitivity 88%, specificity 55%, positive likelihood ratio (+ LR) 1.98, negative likelihood ratio (− LR) 0.21, PPV 80.4%, NPV 69.5%). A Lung Ultrasound Score of 8 points maximized specificity (sensitivity 59%, specificity 91%, + LR 6.84, − LR 0.45, PPV 66.1%, NPV 88.7%). A Lung Ultrasound Score of 6 points optimized a balance of sensitivity and specificity (sensitivity 77%, specificity 84%, + LR 4.82, − LR 0.27, PPV 76.4%, NPV 84.7%) are also reported in Fig. 2. A grey zone analysis in Fig. 3 demonstrates that Lung Ultrasound Scores under 4 have sensitivity over 90% of SARS-CoV-2 infection, and that scores over 6 have over 90% specificity for the detection of SARS-CoV-2 infection. A modified version of the Lung Ultrasound Score that scored the presence of pleural effusion as − 3 points had improved discrimination compared to the standard Lung Ultrasound Score (Fig. 2; AUC 0.90, 95% CI 0.84, 0.96), and improved test characteristics (Fig. 2). Using pleural effusions as a negative prognostic factor improved the performance of the Lung Ultrasound Score in the grey zone analysis, narrowing the area of uncertain diagnosis (Fig. 3).

Patient-related factors associated with decreased lung POCUS accuracy were also investigated. Patients with false-negative lung POCUS scans (n = 13) had higher average BMI compared to patients with true-positive scans (36.5 kg/m² ± 9.4 versus 29.3 kg/m² ± 8.1; p < 0.001) (Table 4). Patients with false-positive lung POCUS scans (n = 23) were more likely to have pre-existing ILD than patients with true negative scans (26% versus 0%, p < 0.001; Table 4). Patients with false-positive lung POCUS scans were also more likely to have systolic heart failure (57% versus 24% p < 0.001) and heart failure with ejection fraction (EF) under 35% (30% vs 3% p < 0.001; Table 4). There was no difference in the prevalence of asthma, COPD, HIV with CD4 < 200, immunosuppression, or end-stage renal disease in patients with discordance between lung POCUS and RT-PCR result (Table 4).

Exclusion of patients with these comorbid conditions associated improved the diagnostic performance of lung POCUS for COVID-19 (Table 2). In patients without these confounding conditions, the sensitivity was 89.3%, specificity was 83.3%, NPV 84.7%, and PPV 88.2%. These exclusions also improved the discriminative ability of the Lung Ultrasound Score (Fig. 2; AUC 0.91, 95% CI 0.85, 0.98) and narrowed the grey zone area of uncertain diagnosis (Fig. 3). The test characteristics associated with cutoff points maximizing sensitivity (2 points: PPV 84.0%), specificity (8 points: NPV 97.1%), or statistically optimizing a balance of sensitivity and specificity (6 points: PPV 81.6%, NPV 91.7%) are also reported in Fig. 2.

A secondary analysis including only complete lung POCUS examinations with all 12 lung fields was performed. Participants with complete POCUS lung evaluation (n = 100) had a mean age of 51.3 years, were 54% male, and 55% had positive RT-PCR testing for SARS-CoV-2 (Additional file 2: Table S3). Complete examinations had improved test characteristics and better discriminative ability (Additional file 9: Figure S2) compared to examinations that did not include all lung fields (AUC 0.89, 95% CI 0.83, 0.96. Sensitivity 90.9%, specificity 75.6%, NPV 87.2%, PPV 82.0%) (Fig. 2).

These data demonstrate that lung POCUS provides rapid information regarding COVID-19 status that is consistent with results of the current gold standard RT-PCR test. In our patient cohort, there was also anecdotal evidence that lung POCUS could outperform the RT-PCR test. At least three patients in this study with initially negative RT-PCR testing but lung POCUS examinations suggestive of COVID-19 were subsequently diagnosed with SARS-CoV-2 infection through more invasive testing.

While descriptive studies of lung POCUS findings in COVID-19 are numerous [19, 22, 25], this is most thorough investigation of a quantitative lung POCUS score to diagnose COVID-19 with data allowing maximization of sensitivity, specificity, and discriminative ability. Other studies do provide information on the test characteristics of lung POCUS in the diagnosis of COVID-19, but are limited by much smaller cohorts and unable to demonstrate the performance in populations with various comorbidities [26, 27, 29, 31]. The present study provides a more comprehensive assessment of the diagnostic power of lung POCUS in a large heterogenous population, and provides vital information for properly applying lung POCUS in the diagnosis of COVID-19.

These data suggest that a Lung Ultrasound Score cutoff of 2 points maximizes sensitivity for use in screening of symptomatic patients. Using this approach, any POCUS examination with three or more discrete B-lines in two distinct lung fields, or any examination with confluent B-lines or subpleural consolidation in any single lung field is concerning for COVID-19, requires isolation, and may benefit from additional testing. Any lung POCUS examination with a Lung Ultrasound Score of 0 points or 1 point is very unlikely to be associated with COVID-19, and can be triaged out of PUI workflows and investigated for other etiologies of their symptoms. At any point during the examination, if the cumulative Lung Ultrasound Score is equal to or greater than 2 points, the clinician can stop and order confirmatory testing, and move on to the next patient. This approach deliberately maximizes the sensitivity and negative predictive value at the expense of the specificity and positive predictive value, and will need to be externally validated. Additionally, while this approach may be advantageous for efficiency, the secondary analysis strongly suggests that complete examinations with 12 lung fields have improved diagnostic power compared to abbreviated examinations. Other cutoffs maximizing specificity (8 points) or balancing between sensitivity and specificity (6 points) can be considered in the appropriate clinical contexts.

The accuracy of lung POCUS may be impacted by patient factors. Elevated BMI reduced the sensitivity of lung POCUS for COVID-19, consistent with prior studies that have reported similar effects of obesity [33]. Pre-existing interstitial lung disease reduced the specificity of lung POCUS for COVID-19. This confounder may be due to the pre-existing pathology of interstitial inflammation, scarring, and thickening leading to a similar ultrasonographic appearance. It is also challenging to differentiate between pulmonary edema due to heart failure [34] or end-stage renal disease and the interstitial inflammation caused by COVID-19. In patients with B-lines on lung POCUS, a concurrent cardiac POCUS may help differentiate between COVID-19 and cardiac etiologies [35]. Other tools such as M-mode evaluation for pleural irregularities [35] and the presence of "spared areas" [36] have been used to differentiate interstitial syndrome from pulmonary edema in the past, but have not been validated for use with COVID-19. Avoiding the use of lung POCUS in patients with these confounding comorbidities (morbid obesity, interstitial lung disease, heart failure) improves the diagnostic performance for COVID-19.

Lung POCUS is a rapid, inexpensive tool that provides results that are concordant with RT-PCR testing in patients under investigation for COVID-19. The low cost, rapid assessment, lack of ionizing radiation, and applicability to a variety of practice environments make it an appealing option for use when other diagnostic tests such RT-PCR or CT chest are unavailable.