Continuous respiratory rate monitoring during an acute hypoxic challenge using a depth sensing camera

The clinical importance of respiratory rate (RR) is well known as it provides important information regarding many aspects of a patient’s respiratory status. Changes in RR are often one of the earliest and more important indicators that precedes major clinical manifestations of serious complications such as respiratory tract infections, respiratory depression associated with opioid consumption, anaesthesia and/or sedation, as well as respiratory failure [1‐3]. A wide range of methods have been proposed for the determination of respiratory rate using non-contact means including RGB video camera systems [4, 5], infrared camera systems [6], laser vibrometry [7], piezoelectric bed sensors [8], doppler radar [9], thermal imaging [10] and acoustic sensors [11]. The determination of respiratory information from depth data has received relatively less attention than RGB video methods, although such systems are well suited to this task.

The inflation and deflation of the lung during a respiratory cycle is a phenomenon measurable with a depth camera. Many studies have compared tidal volume, as measured by a reference system (e.g. a spirometer), with the tidal volume extracted by a depth camera system based on morphological changes in the chest wall. One of the earliest tidal volume measurements extracted in this way was carried out by Yu et al. [12]. They assessed a Kinect V1 based system against a spirometer and achieved a correlation coefficient, R = 0.97 (p < 0.001), based on 12 healthy subjects undertaking a range or respiratory activities including shallow, middle and deep breathing as well as isovolume maneuvers. Aoki et al. [13] performed a similar study using a Kinect V1 system comparing the tidal volumes to that measured using a gas analyser. Four healthy subjects were monitored over a 180 s acquisition period and they obtained an R = 0.99 correlation coefficient. A more recent study by Seppänen et al. [14] evaluated depth camera extracted tidal volume over a range of respiratory rates. Part of the data was used to train a bank of FIR filters from which the depth data volume was extracted. Various filter configurations were tested. Their best model produced R² = 0.932 (R = 0.965) and a tidal volume accuracy error of 9.4%. In another study, Soleimani et al. [15] obtained volume and flow signals by processing Kinect V2 depth data to produce parameters typically associated with a pulmonary function testing. They evaluated their system on 40 COPD patients. Using a spirometer as a reference, they demonstrated that the forced vital capacity and slow vital capacity test both produced correlation coefficients of 0.999, (although they included a rescaling based upon the spirometry measurements to achieve this). Harte et al. [16] used four Kinect V1 cameras arranged in a cross configuration pointing towards the subject located in the centre. Their study included thirteen healthy subjects and nine cystic fibrosis patients. Based upon a one-way ANOVA test (F[1,51] = 7.5783; p = 0.0082), they concluded there was a significant difference in the sample means of the two groups. Transue et al. [17] evaluated a Kinect V2 based system against a spirometer for tidal volume accuracy. Their system performed a general volume estimation of the chest and then required per-subject training against a spirometer as reference to offer more accurate individual results. The dataset consisted of multiple trials of 20 s duration with participants in a standing position. They obtained accuracies ranging from 92.2 to 94.2% (corresponding to absolute errors in tidal volume from 0.055 to 0.079 l) when assessing four healthy subjects against a spirometer.

A number of research groups have focused specifically on determining a respiratory rate using depth sensing camera equipment. Bernacchia et al. [18] assessed ten healthy young adult subjects and found good agreement between the breath periods derived from a Kinect depth sensing system and a spirometer reference. They achieved a 9.7% RMSD for the breath periods between the two devices. During the tests, which lasted only 40 s per acquisition, the subjects were asked to maintain ‘regular respiratory activity’. In a study of young children (between 1 and 5 years) Al-Naji et al. [19] found excellent agreement between depth-sensing RR and a piezo-belt reference with correlation coefficients ranging from 0.97 to 0.99 depending on whether bed sheets were used and the background lighting levels. The study was, however, limited to five healthy volunteers in relatively benign conditions. Rezaei et al. [20] studied the respiratory rate of restrained rodents when subjected to fear-inducing predatory odours. They found they could measure respiratory rate with an accuracy of 94.8% using a reference RR from visual observation. Martinez and Stiefelhagen [21] utilised a depth camera to extract respiratory rate data from 94 sleep analysis sessions from 67 patients in a sleep clinic. They found their system to be 85.9% accurate when compared to a reference thermistor placed at the nose. However, their depth system achieved similar results to that of a contact chest band sensor. Monitoring the respiratory rates of three preterm infants was the focus of a study by Cenci et al. [22] where each infant was assessed in five 30 s intervals. They found an overall correlation coefficient of R = 0.95 between their system output and the respiratory rate derived from ECG impedance pneumography.

Much of the early work involving depth sensing cameras focused on the determination of tidal volume. More recent studies which have considered respiratory rate are, in general, limited to relatively benign conditions, short periods of time, limited numbers of subjects/patients and/or poor reference measures. The work reported here extends current research in this area by studying a cohort of healthy volunteers exhibiting a wide range of respiratory rates resulting from being subjected to a rigorous, protocolized hypoxic challenge.

Figure 4 contains the comparative RR signals for all fourteen subjects. The primary protocol did not state any specific breathing paradigm other than to breathe normally and to take deep breaths while the air supply was returned to room air at the end of the session.

Figure 5a contains the scatterplot of RR_depth against RR_cap aggregated over all subjects. Here each point in the plot represents a respiratory rate calculated from an individual breath of one of the subjects. The bias and accuracy (RMSD) for all subjects pooled together was 0.04 bpm and 0.66 bmp respectively and with a Pearson’s correlation coefficient of 0.99 (p < 0.001) The corresponding Bland–Altman plot is shown in Fig. 5b for comparison and shows that 95% of calculated RR_depth values occur within 1.32 bpm of the reference system. The distributions of the individual respiratory rates across all subjects for the depth sensing and reference capnograph devices are shown in Fig. 5c. We can see that both distributions look very similar.

The data was also analysed on a per-subject basis. The box plots of Fig. 6a, b show the spread of the individual mean biases, which range from − 0.30 to 0.22 bpm, and the RMSD accuracies, which varies from 0.45 to 0.94 bmp. The spread of the individual Pearson correlation coefficients is shown in Fig. 6c which ranged from 0.91 to 0.98.

Uptime was calculated to be 100% for every case.

Excellent agreement was found between the computed depth sensed respiratory rates and the reference capnograph values, with an overall RMSD of 0.66 bpm. This was achieved with a near zero bias and with 100% uptime. (Even on a per-subject basis, individual RMSDs were all less than 1 bpm.) In addition, the results were achieved over a considerable range or respiratory rates exhibited by the volunteers, (from 6 to 27 breaths per minute). This wide range is indicative of the physiological stress associated with an acute hypoxic challenge and provides a valuable test of the technology. In fact, marked cyclical variations in the respiratory volume signal were observed for some subjects. Figure 7 provides an example this phenomenon. This breathing pattern is a result of hypoxia and the pattern can change as the hypoxia increases. One of the challenges with hypoxia studies is hyperventilation in response to hypoxic gas mixture as the body attempts to maintain as high a PO₂ as possible in the lungs by reducing CO₂. The chemoreceptors in the brain detects the decrease in PaCO₂ and signals to slow down breathing and/or reduce tidal volume, however, the hypoxic drive overrides this signal and make the subject breath deeper. This results in this pattern which is similar in form to Cheyne–Stokes breathing.

A relatively small data set (N = 14) was used in developing the algorithm with no data withheld for separate blind testing. Over-training is a therefore possibility, and the results must be viewed in this context. However, the respiratory signal was manifestly obvious across all patients and a relatively simple algorithm was sufficient to identify individual breaths and produce a respiratory rate. In addition, off-the-shelf depth sensing technology was employed and the equipment is simple to set up with minimal instruction provided to the clinical research staff for its use. (They were asked to set up the camera at chest height within 1 to 2 m of the subject, ensure that the subject was in the frame and turn the acquisition on; with no other intervention or calibration required.)

Motion was relatively restricted as the subjects were seated, attached to a face mask and also had pulse oximeter probes attached to each hand. The participants therefore remained relatively immobile during the acquisition. In clinical practice, however, an algorithm would have to cope with more significant patient motion, including change of posture or position in bed, hand and limb movements, (including hand movements within the line of sight) and interactions with clinical staff. Recently, our group proposed a novel motion protocol for RGB video monitoring including yaw, pitch and roll maneuvers of the head [24]. Although this is at a very early stage and specific to idealized maneuvers for the subject’s head it could perhaps be extended to whole body maneuvers for testing depth-based respiratory technology. However, it is in general very difficult to synthesize the wide variety of complex activities observed in clinical practice and exposing the algorithm to large amounts of patient data acquired from across the spectrum of patient care is ultimately the best way to develop a robust technology.

It is an open question whether depth sensing monitoring could replace other modalities for monitoring respiratory rate in the clinical environment. The technology may be adopted more rapidly in its current state for specific use cases including post-anesthesia respiratory depression indications, neonatal monitoring (where there is a need to avoid excessive contact with the neonatal skin), sleep monitoring and home monitoring of, for example, elderly, post-surgical and/or respiratory patients. The technology also has the advantage of having additional physiological and patient contextual information available from the same modality. These include tidal volume trending, apnea detection, patient activity monitoring, fall detection and bed posture monitoring. A final positive attribute of the technology which should be commented on is its ease of use. Ultimately, the technology should require nothing more than to be aimed at the subject and turned on and it works through patient clothing and bed sheets and with the lights turned off (both of which RGB methods cannot do).

The results strongly indicate the potential for a robust respiratory rate monitoring technology based on depth sensing camera equipment. Future work should attempt to fully test this technology in a more rigorous fashion through a range of confounders typically exhibited in the clinical setting including motion and interference.