Abstract
Contactless measurements during the night by a 3-D-camera are less time-consuming in comparison to polysomnography because they do not require sophisticated wiring. However, it is not clear what might be the diagnostic benefit and accuracy of this technology. We investigated 59 persons simultaneously by polysomnography and 3-D-camera and visual perceptive computing (19 patients with restless legs syndrome (RLS), 21 patients with obstructive sleep apnea (OSA), and 19 healthy volunteers). There was a significant correlation between the apnea hypopnea index (AHI) measured by polysomnography and respiratory events measured with the 3-D-camera in OSA patients (r = 0.823; p < 0.001). The receiver operating characteristic curve yielded a sensitivity of 90% for OSA with a specificity of 71.4%. In RLS patients 72.8% of leg movements confirmed by polysomnography could be detected by 3-D-video and a significant moderate correlation was found between PLM measured by polysomnography and by the 3-D-camera (RLS: r = 0.654; p = 0.004). In total, 95.4% of the sleep epochs were correctly classified by the machine learning approach, but only 32.5% of awake epochs. Further studies should investigate, if this technique might be an alternative to home sleep testing in persons with an increased pre-test probability for OSA.
Similar content being viewed by others
Introduction
In medicine, contactless motion analysis using the Kinect 3-D-camera and subsequent VPC is an effective diagnostic instrument for measuring gait impairment in patients suffering from multiple sclerosis (gait speed, stride length, stride time), tremor in Parkinson patients, fall risk assessment of elders, gesture commands, body sway and posture, anthropomorphic measurements for example for surgical planning, epileptic seizure semiology and even contactless heart rate measurement1,2,3,4,5,6,7,8,9,10,11,12,13,14,15. The Microsoft Kinect includes different sensors. At first, it includes a high definition 30 Hertz camera, similar to most of modern webcams and a microphone array. Moreover, the Kinect emits infrared light from three internal sources. This light is reflected by the environment and captured by an infrared sensor. Based on these infrared data, it can be calculated for each pixel, how long the light needs to be sent and captured again, and a 3-D-image of the environment can be built. Although the technical characteristics are ideal to examine sleep disorders, only few studies used the Kinect during the night. One study has monitored the sleep position of elder persons in the bed during the night by VPC in order to investigate the risk of falling16. Krüger et al. investigated one person, Procházka et al. four persons, and Garn et al. 10 persons simultaneously with the Kinect and polysomnography (PSG)17,18,19. Procházka et al. found a very good correlation of breathing features, and Garn et al. a good correlation of periodic limb movements (PLM)18,19.
Vision-based and hence contactless movement analysis using a 3-D-camera was originally developed for game consoles. However, the use of 3-D-cameras is not limited to the entertaining industry; they have been employed also with promising results in medicine1,2,3. 3-D-cameras have been originally developed by Microsoft, who indeed discontinued the production of the so called “Kinect” sensor and adapter, but continued to provide a free Software Development Kit (SDK) (https://developer.microsoft.com/en-us/windows/kinect). Almost all study groups using 3-D-cameras for motion analysis are using the Microsoft Kinect and SDK for 3D video capture16. Subsequently, after recording and after signal extraction computational algorithms are needed to interpret the flood of data. Therefore, most study groups program their own visual perceptive computing (VPC) software tools.
The aim of this study was to investigate whether VPC could i) be a home-based measurement for obstructive sleep apnea (OSA) comparable to home sleep testing (HST), ii) detect PLM, and iii) differentiate between sleep and wake.
OSA and RLS patients aged 18 to 70 years were consecutively recruited between September 23, 2015 and November 20, 2016 from our outpatient sleep center and healthy volunteers through personal contact. All OSA patients were newly diagnosed and untreated and underwent a nocturnal HST prior to enrollment. Complete data sets of simultaneously recorded PSG and Kinect measurements were obtained from 59 persons: in the OSA subgroup 11 men and 10 women aged 27 to 64, in the RLS subgroup 10 men and 9 women aged 20 to 70, and in the healthy volunteer (HV) subgroup 9 men and 10 women aged 18 to 57.
Results
Apnea hypopnea index in the polysomnography versus respiratory events measured with the Kinect
Kinect-respiratory-event-index (KREI) versus apnea-hypopnea-index (AHI): Fig. 1 shows a Box-Whisker-Plot comparing respiratory events measured with the Kinect (KREI) versus apneas and hypopneas measured with polysomnography (AHI). It indicates the variability outside the upper and lower quartiles in the different subgroups.
In healthy volunteers and RLS patients, the measurement with the Kinect overestimates respiratory events and in OSA patients the opposite is the case. (Ten RLS patients suffered from comorbid OSA, for which reason the AHI is also increased in this subgroup). Moreover, there is a linear relationship between respiratory events measured with the Kinect and the PSG which is shown in Fig. 2.
In the OSA subgroup there was a highly significant and strong correlation between the AHI and the KREI (r = 0.823; p < 0.001), and a significant moderate correlation in HV (r = 0.647; p = 0.013) and in RLS patients (r = 0.616; p = 0.025).
KREISLEEP versus AHI: Analysing the relationship between the AHI and KREIsleep there was quite similar a highly significant and strong correlation in the OSA subgroup (r = 0.826; p < 0.001), and a significant moderate to strong correlation in HV (r = 0.666; p = 0.009); whereas in RLS there was only a trend (r = 0.530; p = 0.063).
KREI versus AHIobs: After adjusting for central apneas and analyzing exclusively respiratory events with continued breathing efforts (obstructive sleep apnea) similar results were found. In the OSA subgroup there was a similarly significant correlation between the AHIobs and the KREI (r = 0.887; p < 0.001), a significant moderate correlation in HV (r = 0.701; p = 0.005), but no correlation in RLS patients (r = 0.445; p = 0.128).
The Bland–Altman plot in Fig. 3 below shows, that measurements with the Kinect tend to underestimate respiratory events in OSA patients.
To test the validity of the Kinect as a diagnostic device a receiver operating characteristic (ROC) curve was generated. It shows, comparing the OSA subgroup with the HV subgroup, a sensitivity of 90% for detection of OSA and a specificity of 71.4% - applying a KREI cut-off of 6.6 per hour (area under the curve 0.857).
In order to investigate whether the difference between the KREI and the AHI is caused by the sensitivity of the Kinect sensor (modifications of the breathing signal) or the computing algorithm, we performed retrospectively a simultaneous visual scoring of the row data of the Kinect and the PSG. By visual scoring, only 77% of the apneas and hypopneas from the PSG could be seen in the Kinect data which suggests that the problem has not been caused by the computing algorithm, but rather by the sensor itself and it is unlikely that optimization of VPC will improve the results.
Periodic limb movement index in the polysomnography versus Kinect measurements
By visual scoring of the raw data of the leg movements (Kinect versus PSG), 72.8% of leg movements in the PSG could be seen in the Kinect data. In HV and RLS patients there was a significant correlation between the PLMI and the KPLMI (RLS: r = 0.654; p = 0.004/HV: r = 0.686; p = 0.002), whereas no correlation was found in OSA patients (r = 0.323; p = 0.207).
Sleep detection by visual perceptive computing and subsequent machine learning
The aim of the machine learning concept was to develop algorithms which can automatically detect sleep based on a motion analysis – independent from the electroencephalography. For technical reasons the Kinect data were stored in different sections of two hours and in most cases a single night included four parts of Kinect recordings. For the following machine learning approach, only artefact free parts were used.
The Kinect data of the first 12 participants were segregated in analogy to the PSG data into 30 second epochs and we performed a motion analysis of the different polysomnographically confirmed sleep stages. Subsequently, we developed an algorithm of sleep stage classification based solely on the Kinect data, what means that these new machine learning algorithms can classify sleep only by an analysis of movements (Kinect sleep stages) without needing any PSG data. The data of the first 12 participants served exclusively as a training set and were not examined further.
In a second step, we used then these new algorithms to predict sleep in the remaining participants whose data have not been used for the development of the above-mentioned new algorithms.
In a third step, the obtained Kinect sleep stages were compared to the PSG sleep stages. In total, 95.4% of the PSG sleep epochs were correctly classified by the machine learning approach, but only 32.5% of awake states. This results in a positive predictive value of sleep detection of 77.9% and a negative predictive value of sleep detection of 74.1%.
In particular in the sleep onset period (calm awakening) when participants were lying in bed without moving and tried to fall asleep, the machine learning algorithm classified often wrongly sleep instead of awake – probably due to the lack of movements in this period.
Discussion
This is the largest prospective study using VPC to measure respiratory events, PLM and sleep stage classification based on a motion analysis during the night. If applying a cut-off of 6.6 respiratory events per hour (KREI), the Kinect discovered OSA with a sensitivity of 90% and a specificity of 71.4%. These findings suggest that 3-D-video could perhaps become an alternative to HST.
Reichert et al. investigated 51 consecutive adults referred to the sleep lab for suspicion of OSA, and they found OSA using HST in 91% and specificity was 83%20. Against this background it is especially notable that in our study only approximately one-third of persons had an increased pre-test probability for OSA - in contrast to most studies investigating the sensitivity of HST, mostly predominantly in patients suspected of having OSA21,22. The scoring of hypopneas requires according to the AASM rules the measurement of flow and arousal and oxygen saturation23. Although 3-D-video measures neither arterial oxygen saturation nor arousal or flow, the Kinect can help probably in a comprehensive evaluation for sleep disordered breathing. It seems to detect apnea and hypopnea with a sensitivity and specificity comparable to that of three channel HST. (It should be noted that HST measures flow and oxygen saturation, but nevertheless does not measure arousal which is why arousal-related hypopneas (type B) are consequently overlooked in HST; arousal can only be detected by EEG and the measurement of arousal requires a PSG. Hence, both methods, 3-D-video and HST cannot detect arousal-related hypopneas). This might partially explain the differences between the AHI measured with the Kinect and the PSG.
The diagnosis of OSA requires a clinical evaluation and an assessment of breathing patterns. Both, HST and 3-D-video measurements can fail to identify mild OSA which may result in misdiagnosis and lack of therapy24. Therefore, OSA can only be excluded by PSG and not by HST. In case of suspicion of OSA a PSG is in most cases required in order not to misdiagnose mild OSA - even if a prior HST has been normal. This is important in order to describe the limitations of the HST. Measurements with 3-D-video seem to detect apnea and hypopnea with a sensitivity and specificity comparable to that of three channel HST.
It should be noted that the Kinect system includes a microphone array. An audio analysis of snoring might improve additionally diagnostic accuracy. However, in this study we did not investigate the combined parameter of simultaneous changes in breathing amplitude and snoring; but this should be done in further studies.
The Portable Monitoring Task Force of the AASM recommended for the diagnosis of OSA home diagnosis with portable monitoring devices in conjunction with a comprehensive sleep evaluation by a sleep specialist as an alternative to PSG in patients with a high pretest probability of moderate to severe OSA and without significant comorbid medical conditions or other sleep disorders24. Our results indicate that the Kinect may be appropriate for these purposes and may become a new diagnostic tool in timely home-based diagnosis of OSA and for therapy control in patients under CPAP therapy. The results do not depend on the sleep stage classification. Even if analyzing only the time spent sleeping (KREIsleep) the results did not change significantly.
This is especially important considering the background that in our study the sleep stage classification based on the motion analysis was not successful. Krüger et al. and Procházka et al. found a good correlation of the sleep stage classification by the Kinect compared to PSG17,18. But it should be noted, that Krüger et al. investigated only one person, and Procházka et al. investigated (apart from the small sample size with four persons) breathing patterns for the sleep stage classification and that in our study approximately the half of the patients suffered from OSA (21 OSA patients and 10 RLS patients with comorbid OSA).
There are several possibilities to explain why the motion analysis was not successful. One the one hand, it must be said that for that purpose larger samples are required and our study should be seen as a preliminary study. It may well be the case that further studies will achieve the objective to classify sleep based on movements during sleep. Moreover, as a further methodical limitation, the algorithm could not be accurate enough. All in all, it could be that a machine learning approach with a bigger sample size could be able to classify sleep only by a motion analysis. On the other hand, it is possible, that a sleep stage classification by motion analysis might be generally not successful. Up to date, very few data exist on the frequency and number of movements in different sleep stages25,26. Stefani et al. investigated movements during physiological sleep in the head, neck, trunk, and in the extremities in a single night PSG25. They found a median number of 10.2 movements per hour sleep, most frequent in N1 and to a lesser extent in REM, and lowest in N2 and N3. But these results can be influenced by the so-called first-night-effect due to cables and the unusual environment. Moreover, to our knowledge, there are no valid data about the amount of physiological movements during the time awake before sleeping and during nocturnal awake phases. This situation is complicated by the fact, that in patients suffering from RLS and in persons with periodic limb movements in sleep (PLMD) the beginning of sleep is not necessarily associated with a decrease of movements. Finally, we do not have normative data about the amount of movements during the bed-time in healthy subjects (awake and asleep). The situation appears even more complicated if we consider, that in OSA patients movement arousal due to desaturation can be associated with or without sleep stage changes or awakenings. Finally, a sleep stage classification by a 3-D-video-camera and subsequent motion analysis uses the parameter “motion” for the sleep stages classification and for long there has been a debate, if actigraphies are able to detect sleep and in which extent. Our approach to classify sleep by a motion analysis does not substantially differ from previous studies using actigraphy. The only difference is that we used a 3-D-video for motion caption, but the principles are the same. Previous sleep studies using actigraphy showed indeed only a moderate correlation between the TST measured concurrently with PSG and actigraphy, due to an overestimation of TST by actigraphy27. Moreover, several studies suggest some systematic misclassification of sleep and wake by actigraphy27. It must be highlighted, that our machine learning approach classified sleep correctly but misclassified the calmer wake periods during the phase before falling asleep as sleep - and these calmer periods before falling asleep were not distinguishably from wake, because patients did not move during this period. This is in line with previous studies showing, that actigraphy seems to underestimate sleep onset latency (SOL) and wakefulness after sleep onset (WASO)28. However, previous studies using actigraphy showed that actigraphy is appropriate for the assessment of and stability of treatment effects of anything from hypnotic drugs to light treatment to continuous positive airway pressure29. Against this background, it remains to be analyzed whether measurements with 3-D-video might also be suitable to assess treatment effects. Actigraphy has also been shown to be very good for identifying rhythms29. Here, too, it is conceivable that 3-D-video measurements may be suitable for the contactless investigations of rhythms (which should be investigated by further studies).
The Kinect detected approximately three-fourths of PLM measured with the PSG. This might be due to the fact, that a certain number of PLM measured by bipolar EMG are not accompanied by a visible movement. On the other hand, the Kinect can detect movements from the whole body. PLM are mostly caused by a periodic innervation of the tibialis anterior muscle with subsequent extension of the big toe and often with a flexion in the knee and sometimes also in the hip30. Apart from the chin EMG, the AASM 2012 rules recommend solely the EMG of both tibialis anterior for detection of movement23. Apart from some sleep laboratories with particular expertise in REM sleep behavior disorder and other movement disorders, the routinely done standard video-PSG contains only the EMG of both tibialis anterior for detection of motor activity of the limbs and trunk. However, the sleep specialist can use the video in order to analyze movements what is very time-consuming and not used on a regular basis.
We have conversely not explored how many movements are visible in the Kinect, but not in the tibial EMG. Provini et al. investigated PLM in detail by PSG and extended surface EMG (biceps brachii, triceps brachii, rectus abdominis, thoracolumbar paraspinales, rectus femoris, biceps femoris, tibialis anterior and gastrocnemius muscles). Nearly three-fourths of PLM involved the tibialis anterior, but PLM also involved the gastrocnemius (antagonist of the tibialis anterior), biceps femoris, rectus femoris and even in axial muscles (rectus abdominis and thoracolumbar paraspinales)31. This is not surprising, given that the RLS is a sensimotor disorder with an altered dopaminergic neurotransmission in the brain with activation of the ascending arousal system and loss of the inhibitory influence on the spinal cord32,33,34.
Even if the exact mechanism of idiopathic RLS remains unclear, low brain iron despite normal peripheral iron due to a decreased iron acqusition by brain cells in genetically predisposed persons is one of its key findings35. The brain iron insufficiency is associated with an altered dopaminergic neurotransmission which is considered the final mechanism underlying restlessness and PLM32. Autopsy data showed a decreased brain iron in the substantia nigra and putamen with an increased activity of tyrosin hydroxylase (tyrosine hydroxylase is catalyzing L-tyrosine into levodopa)32,36. All in all, brain iron insufficiency seams to lead to an increased dopamine synthesis. Therefore, it is not surprising, that PLM involve more muscles than just the tibialis anterior and more EMG should be recorded. But this is often not possible because many PSG systems have not enough free channels in order to record more EMG and only sleep laboratories with special expertise in REM sleep behavior and movement disorders are recording EMG from all four limbs and from the trunk.
Provini et al. investigated the time delay and pattern of activation between the first and the other activated muscles. The tibialis anterior was the most frequent starting muscle with approximately the half of PLM starting in the tibialis anterior31. However, some PLM started also in the gastrocnemius, biceps or rectus femoris, in the upper limb muscles and 1.1 per cent even in axial muscles (rectus abdominisa and thoracolumbar paraspinales). Moreover, detailed analysis of the pattern of activation showed no constant recruitment pattern from one to another PLM, even in the same patient and the time delay between the first and the last activated muscles was up to two seconds. A major advantage of the 3-D-video is, that it is contactless and only a minimum of time is required for starting and stopping of the records so that movements from the whole body can be investigated and also repeated investigations can be performed with a minimal investment of time. Further investigations of other sleep disorders as for example REM sleep behavior should be performed.
In sum, the Kinect seems to be a valuable device for home-based measurements and might be an alternative to home sleep testing in patients with a high pretest probability of moderate to severe OSA and without significant comorbid medical conditions or other sleep disorders. However, future prospective and controlled studies are required to examine whether this new technology is a suitable diagnostic tool for home sleep testing at home.
Moreover, this new VPC-based technology has the potential to become also a valuable diagnostic instrument to measure PLM and other movements during sleep. Considering the background of current technical developments and a shift toward home-based investigations, normative values about the frequency and nature of movements during the bedtime are recommended and future studies should be performed.
Method
Subjects
Figure 4 displays the sampling process. OSA and RLS patients aged 18 to 70 years were consecutively recruited between September 23, 2015 and November 20, 2016 from our outpatient sleep center and healthy volunteers through personal contact. All OSA patients were newly diagnosed and untreated and underwent a nocturnal home sleep testing prior to enrollment.
The OSA diagnosis was based on the AASM criteria and we used the cutoffs for OSA severity described there (AHI: 5–15 mild, 15–30 moderate, and >30 severe)37. If OSA was not confirmed by the subsequent performed in-laboratory PSG, patients were not included. All RLS patients were consecutively included and fulfilled the four essential RLS criteria38,39.
Twenty-three OSA patients, 27 RLS patients and 28 healthy volunteers were enrolled. From the 28 healthy volunteers 7 showed an increased apnea hypopnea index (AHI) ≥ 5/h (of these, one patient showed moderate OSA with an AHI of 15/h who was subsequently included in the OSA subgroup) and one patient a low sleep efficacy. Therefore, 20 healthy volunteers were included in this subgroup.
All participants with a known or suspected diagnosis or history of heart failure, home ventilation, continuous positive airway pressure (CPAP) therapy, respiratory insufficiency, mental disorder or class III obesity were excluded. Moreover, pregnancy was an exclusion criterion. By definition, all OSA patients showed apneas or hypopneas.
Drugs were not systematically tapered in RLS patients, but in some patients, dopamine agonists were reduced the evening before the examination. The absence of PLM in the PSG was considered as an exclusion criterion for the RLS subgroup, since the main interest in the RLS subgroup was the analysis of PLM.
Ethics statement. Permission for the study was obtained from the Charité University Medicine Berlin Ethics Committee on 21 September 2015 (EA1/199/15). All methods were carried out in accordance with the relevant guidelines and regulations. All patients gave prior written informed consent.
Clinical trial. Clinical Trial Name: Pilot study on contactless sleep monitoring of patients with different sleep disorders based on three-dimensional video signal and audio signal analysis (“Kinect”); http://www.germanctr.de; DRKS-ID: DRKS00009347; registration date: 27/11/2015.
Procedures
Polysomnography. The PSG with audio-video recording were performed and scored with the Embla system (N7000 or S7000; Natus; Software Remlogic 3.4, Embla Systems, Ontario, Canada) according to the AASM Manual for the Scoring of Sleep and Associated Events23. We used respiratory inductive plethysmography and nasal pressure transducer23. Sleep stages were scored manually according to the AASM Manual by the same board-certified sleep specialist (B.D.)23. Subsequently, the AHI was calculated (AHI).
Whereas obstructive apneas are associated with continued or even increased respiratory effort, central apneas are associated with an absent inspiratory effort. Based on these considerations, and moreover taking the fact into account that the Kinect is not measuring the flow, it would be possible that obstructive apneas with continued respiratory effort will be overviewed by the Kinect which might detect mostly central apneas, so that a shift to central apneas will occur. After an exploratory analysis during the first measurements, we found that this was not the case and that even a minor modification of the breathing amplitude was enough to detect an obstructive apnea by the Kinect. However, in order to evaluate the diagnostic accuracy of the detection of OSA and to better evaluate and compare the two methods (PSG and Kinect) we created in parallel an AHIobs containing only obstructive and mixed apneas and hypopneas.
The bipolar electromyogram (EMG) from the left and right anterior tibial muscle was used to record leg movements. The AASM Manual defines a leg movement as a period of 0.5–10 seconds with an increase in EMG voltage of 8 μV above resting EMG. PLM are defined as the consecutive sequence of at least four leg movements with intervals between 5 and 90 seconds. A PLM index (PLMI) was calculated.
3-D-video (Kinect). The patients were sleeping under a normal blanket. This means, that the 3-D-video captured mostly in the strict sense of the term the movement of the blanket. We used a Motognosis Labs System V1.0 (Motognosis, Berlin, Germany) equipped with a Kinect V2 for Windows sensor (9.8 × 2.6 × 2.4 inchs) and the Kinect Software Development Kit (SDK) version 2.0 (Microsoft, Redmond, WA, USA). The sensor and software was previously validated against a marker based system for medical motion analysis40. The software was adapted as recording software based on the programming language C#. We implemented a recording software based on the programming language C#. The recorded 3-D-video (30 frames per second) was digitized in the Microsoft XEF format and then processed with Matlab R2015b (MathWorks Inc., Natick, Massachusetts, United States).
We generated three-dimensional point clouds (see Fig. 5) which were rotated to remove perspective distortion and we employed several filters (flood-fill-operation, average-filter). After segmentation into a thoracic and a lower body segment in a multi-step procedure a circular section was placed on the thorax for the extraction of the breathing signal. The oscillating movements in this area reflect the respiratory effort.
For the signal of leg movements, a vertical was drawn from head to toe. All pixels below the vertical surface in the area of the legs form the signal for leg movements. As the Kinect measures neither the oxygen saturation nor the flow, only the changes of the respiratory effort, it cannot distinguish hypopneas from apneas.
Consequently, we calculated a Kinect-respiratory-event-index (KREI) including all hypopneas and apneas per hour from ‘lights off’ to ‘lights on’. Similarly, we calculated the Kinect-PLM-index per hour from ‘lights off’ to ‘lights on’ (KPLMI).
Both methods rely a variety of different thresholds for detection of respiratory and PLM events. The KREI is derived from the Kinect breathing signal by first calculating the signal envelope and afterwards using a sliding window approach to recognize local drops in the breathing amplitude. These drops are then classified as Kinect respiratory events if the drop in the amplitude is above a certain threshold over a certain period of time. Leg movements (also periodic leg movements) are derived from the Kinect movement signal by detecting peaks in the signal above a certain threshold. These peaks are then further filtered by various means (e.g. in close vicinity to whole body movements or according AASM close to a Kinect respiratory events). The values are oriented on the Embla parameters and fitted empirically on a subset of the data set and then kept fix for the remaining experiments. A Kinect PLM event is detected if enough remaining peaks fall into a long enough time interval. Figure 7 illustrates an example of the bipolar EMG on the right anterior tibial muscle and the Kinect leg movement detection.
In addition, to better compare the two methods (PSG and Kinect), due to methodical reasons we created the KREISLEEP which represents all apneas und hypopneas per hour of sleep. For this purpose, we added the information about sleep stage classification from the Embla system into the Kinect data. Similarly, we calculated the KPLMISLEEP which contains all PLM per hour of sleep measured with the Kinect. Regarding the sleep wake discrimination based on the Kinect data, we used artefact free periods from 12 OSA patients for a machine learning approach and designed an optimized algorithm which has been used in the further course of the study for the sleep wake discrimination based on a motion analysis.
With regard to the distinction between sleep and wake, the machine learning approach was applied as previously described17. At first, the average change between two consecutive frames for an interval of 30 seconds were calculated (all in all 900 frames). These timeframes include the information about the sequence of a movement. Statistical parameters such as mean value, median, minimum and maximum were calculated. Additionally, the changes in the average pixel value over the time were analyzed – indicating the local manifestation of a movement. For these changes mean value, median, minimum and maximum were calculated as well. A raster is superimposed onto the mean change per pixel. For each field of the raster the mean value, median, minimum and maximum were calculated. We used an ensemble classifier of decision trees that was trained using RUSBoost to tackle the class imbalance between sleep and wake samples41. For training, the statistic parameters of the Kinect data and the PSG-hypnogram of 12 patients were used. In a next step the validity of the classification was examined.
Statistical analysis
Following an exploratory analysis of the data we calculated the Pearson correlation coefficients to investigate the relationship between the AHI and the KREI (and between the AHIobs and the KREI and between the AHI and the KREIsleep respectively). Similarly, we investigated the relationship between the PLMI and the KPLMI. Bland–Altman plots were performed to evaluate the agreement between PSG and Kinect measurements. Statistical significance was established at p ≤ 0.05. Analysis was performed with SPSS software (IBM Corp. Released 2016. IBM SPSS Statistics for Windows, Version 24.0. Armonk, NY: IBM Corp).
Data availability
Raw data from the polysomnography study, protocols, and other metadata are available on request from the corresponding author (C.V.). Videos cannot be made publicly available due to ethic restrictions.
Code availability
The code used in this study is available by contacting the corresponding author (C.V.).
References
Grobelny, A. et al. Maximum walking speed in multiple sclerosis assessed with visual perceptive computing. Plos One 12, e0189281 (2017).
Behrens, J. et al. Using perceptive computing in multiple sclerosis - the Short Maximum Speed Walk test. J. Neuroengineering Rehabil. 11, 89 (2014).
Behrens, J. R. et al. Validity of visual perceptive computing for static posturography in patients with multiple sclerosis. Mult. Scler. Houndmills Basingstoke Engl, https://doi.org/10.1177/1352458515625807 (2016).
Amini Maghsoud Bigy, A., Banitsas, K., Badii, A. & Cosmas, J. Recognition of postures and Freezing of Gait in Parkinson’s disease patients using Microsoft Kinect sensor. in 731–734, https://doi.org/10.1109/NER.2015.7146727 (IEEE, 2015).
Stone, E., Skubic, M., Rantz, M., Abbott, C. & Miller, S. Average in-home gait speed: investigation of a new metric for mobility and fall risk assessment of elders. Gait Posture 41, 57–62 (2015).
Stone, E. E. & Skubic, M. Fall Detection in Homes of Older Adults Using the Microsoft Kinect. IEEE J. Biomed. Health Inform. 19, 290–301 (2015).
Ejupi, A. et al. Kinect-Based Five-Times-Sit-to-Stand Test for Clinical and In-Home Assessment of Fall Risk in Older People. Gerontology 62, 118–124 (2015).
Ruppert, G. C. S., Reis, L. O., Amorim, P. H. J., de Moraes, T. F. & da Silva, J. V. L. Touchless gesture user interface for interactive image visualization in urological surgery. World J. Urol. 30, 687–691 (2012).
Yeung, L. F., Cheng, K. C., Fong, C. H., Lee, W. C. C. & Tong, K.-Y. Evaluation of the Microsoft Kinect as a clinical assessment tool of body sway. Gait Posture 40, 532–538 (2014).
Xu, X., McGorry, R. W., Chou, L.-S., Lin, J.-H. & Chang, C.-C. Accuracy of the Microsoft Kinect for measuring gait parameters during treadmill walking. Gait Posture 42, 145–151 (2015).
Grunert, R., Kretzschmar, C., Rotsch, C., Werner, M. & Prietzel, T. Development of an optical measurement system for hip implant surgery to evaluate the leg length and the hip rotation center. In 151–154, https://doi.org/10.1109/MECBME.2014.6783228 (IEEE, 2014).
Henseler, H., Kuznetsova, A., Vogt, P. & Rosenhahn, B. Validation of the Kinect device as a new portable imaging system for three-dimensional breast assessment. J. Plast. Reconstr. Aesthetic Surg. JPRAS 67, 483–488 (2014).
Yang, C., Cheung, G. & Stankovic, V. Estimating heart rate via depth video motion tracking. In 1–6, https://doi.org/10.1109/ICME.2015.7177517 (IEEE, 2015).
Cunha, J. P. S. et al. A novel portable, low-cost kinect-based system for motion analysis in neurological diseases. in 2339–2342, https://doi.org/10.1109/EMBC.2016.7591199 (IEEE, 2016).
Wheat, J. S., Choppin, S. & Goyal, A. Development and assessment of a Microsoft Kinect based system for imaging the breast in three dimensions. Med. Eng. Phys. 36, 732–738 (2014).
Martinez, M., Rybok, L. & Stiefelhagen, R. Action recognition in bed using BAMs for assisted living and elderly care. 14th IAPR Int. Conf. Mach. Vis. Appl. MVA 329–332 (2015).
Krüger, B. et al. Sleep Detection using De-Identified Depth Data. Journal of Mobile Multimedia 10, 327–342 (2014).
Procházka, A. et al. Extraction of breathing features using MS Kinect for sleep stage detection. Signal Image Video. Process. 10, 1279–1286 (2016).
Garn, H. et al. 3D detection of periodic limb movements in sleep. In 427–430, https://doi.org/10.1109/EMBC.2016.7590731 (IEEE, 2016).
Reichert, J. A., Bloch, D. A., Cundiff, E. & Votteri, B. A. Comparison of the NovaSom QSG, a new sleep apnea home-diagnostic system, and polysomnography. Sleep Med. 4, 213–218 (2003).
Ragette, R., Wang, Y., Weinreich, G. & Teschler, H. Diagnostic performance of single airflow channel recording (ApneaLink) in home diagnosis of sleep apnea. Sleep Breath. 14, 109–114 (2010).
Chen, H. et al. Evaluation of a portable recording device (ApneaLinkTM) for case selection of obstructive sleep apnea. Sleep Breath. 13, 213–219 (2009).
Berry, R. B. et al. The AASM Manual for the Scoring of Sleep and Associated Events: Rules, Terminology and Technical Specifications, Version 2.0, www.aasmnet.org. (American Academy of Sleep Medicine, 2012).
Collop, N. A. et al. Clinical guidelines for the use of unattended portable monitors in the diagnosis of obstructive sleep apnea in adult patients. Portable Monitoring Task Force of the American Academy of Sleep Medicine. J. Clin. Sleep Med. JCSM Off. Publ. Am. Acad. Sleep Med. 3, 737–747 (2007).
Stefani, A. et al. A Prospective Video-Polysomnographic Analysis of Movements during Physiological Sleep in 100 Healthy Sleepers. SLEEP, https://doi.org/10.5665/sleep.4994 (2015).
Frauscher, B. et al. Normative EMG Values during REM Sleep for the Diagnosis of REM Sleep Behavior Disorder. Sleep 35, 835–847 (2012).
Blackwell, T., Paudel, M., Redline, S., Ancoli-Israel, S. & Stone, K. L. A novel approach using actigraphy to quantify the level of disruption of sleep by in-home polysomnography: the MrOS Sleep Study. Sleep Med. 32, 97–104 (2017).
O’Hare, E. et al. A comparison of radio-frequency biomotion sensors and actigraphy versus polysomnography for the assessment of sleep in normal subjects. Sleep Breath. 19, 91–98 (2015).
Ancoli-Israel, S. et al. The role of actigraphy in the study of sleep and circadian rhythms. Sleep 26, 342–392 (2003).
International classification of sleep disorders, 3rd ed. (2014).
Provini, F. et al. Motor pattern of periodic limb movements during sleep. Neurology 57, 300–304 (2001).
Ferré, S., García-Borreguero, D., Allen, R. P. & Earley, C. J. New Insights into the Neurobiology of Restless Legs Syndrome. Neurosci. Rev. J. Bringing Neurobiol. Neurol. Psychiatry 25, 113–125 (2019).
Veauthier, C. et al. Periodic limb movements during REM sleep in multiple sclerosis: a previously undescribed entity. Neuropsychiatr. Dis. Treat. 2323, https://doi.org/10.2147/NDT.S83350 (2015).
Rijsman, R. M., Stam, C. J. & de Weerd, A. W. Abnormal H-reflexes in periodic limb movement disorder; impact on understanding the pathophysiology of the disorder. Clin. Neurophysiol. Off. J. Int. Fed. Clin. Neurophysiol. 116, 204–210 (2005).
Connor, J. R. et al. Profile of altered brain iron acquisition in restless legs syndrome. Brain J. Neurol. 134, 959–968 (2011).
Allen, R. P., Connor, J. R., Hyland, K. & Earley, C. J. Abnormally increased CSF 3-Ortho-methyldopa (3-OMD) in untreated restless legs syndrome (RLS) patients indicates more severe disease and possibly abnormally increased dopamine synthesis. Sleep Med. 10, 123–128 (2009).
Sleep-related breathing disorders in adults: recommendations for syndrome definition and measurement techniques in clinical research. The Report of an American Academy of Sleep Medicine Task Force. Sleep 22, 667–689 (1999).
Allen, R. P. et al. Restless legs syndrome: diagnostic criteria, special considerations, and epidemiology. Sleep Med. 4, 101–119 (2003).
Allen, R. P. et al. Restless legs syndrome/Willis–Ekbom disease diagnostic criteria: updated International Restless Legs Syndrome Study Group (IRLSSG) consensus criteria – history, rationale, description, and significance. Sleep Med. 15, 860–873 (2014).
Otte, K. et al. Accuracy and Reliability of the Kinect Version 2 for Clinical Measurement of Motor Function. PloS One 11, e0166532 (2016).
Seiffert, C., Khoshgoftaar, T. M., Van Hulse, J. & Napolitano, A. RUSBoost: A Hybrid Approach to Alleviating Class Imbalance. IEEE Trans. Syst. Man Cybern. - Part Syst. Hum. 40, 185–197 (2010).
Acknowledgements
This work was supported by Deutsche Forschungsgemeinschaft (Exc 257 to FP) and by grants from the German Federal Ministry for Economic Affairs and Energy (BMWI ZIM KF3315301KJ4 to Motognosis).
Author information
Authors and Affiliations
Contributions
The corresponding author had full access to all data, takes responsibility for its integrity and the accuracy of the data analyses, and had final responsibility for the decision to submit for publication. All authors have read, critically revised the manuscript and agree to the manuscript as written. Study design: C.V., A.U.B. Conduct of the study and data collection: C.V., J.R. Kinect Software: K.O., B.K. S.M.-M. Technical advice: M.G., C.S. Data analysis: J.R., K.O. Writing: C.V., J.R. Revision: F.P., T.P.
Corresponding author
Ethics declarations
Competing interests
S.M.-M. and A.U.B. are founder and hold shares of Motognosis GmbH. A.U.B. is founder and holds shares of Nocturne UG. A.U.B. is named as inventor on several patent applications describing serum biomarkers for MS, perceptive visual computing for tracking of motor dysfunction and retinal image analysis. K.O. and B.K. hold shares of Motognosis GmbH. Outside this work: C.V. has received grants from the German Federal Ministry of Education and Research (BMBF) and travel support from Bioprojet. F.P. serves on the scientific advisory board for Novartis; received speaker honoraria and travel funding from Bayer, Novartis, Biogen Idec, Teva, Sanofi-Aventis/Genzyme, Merck Serono, Alexion, Chugai, MedImmune, and Shire; consulted for Sanofi-Genzyme, Biogen Idec, MedImmune, Shire, and Alexion; received research support from Bayer, Novartis, Biogen Idec, Teva, Sanofi-Aventis/Genzyme, Alexion, Merck Serono, German Research Council, Werth Stiftung of the City of Cologne, German Ministry of Education and Research, Arthur Arnstein Stiftung Berlin, EU FP7 Framework Program, Arthur Arnstein Foundation Berlin, Guthy Jackson Charitable Foundation, and National Multiple Sclerosis of the USA. T.P. has received research grants from Heinen & Löwenstein, Itamar, Philips/Respironics, Resmed, Somnodent. T.P. received speaker fees and travel support from Bayer, Itamar, Inspire, Somnodent, UCB, Weinmann. T.P. is a shareholder of Advanced Sleep Research GmbH, The Siestagroup GmbH, Somnico GmbH. He was supported by the project no. LQ1605 from the National Program of Sustainability II (MEYS CR) and by the project FNUSAICRC no. CZ.1.05/1.1.00/02.0123 (OP VaVpI). All other authors report no conflicts of interest. The authors alone are responsible for the content and writing of the paper.
Additional information
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Veauthier, C., Ryczewski, J., Mansow-Model, S. et al. Contactless recording of sleep apnea and periodic leg movements by nocturnal 3-D-video and subsequent visual perceptive computing. Sci Rep 9, 16812 (2019). https://doi.org/10.1038/s41598-019-53050-3
Received:
Accepted:
Published:
DOI: https://doi.org/10.1038/s41598-019-53050-3
This article is cited by
-
Clinical applications of artificial intelligence in sleep medicine: a sleep clinician’s perspective
Sleep and Breathing (2023)
-
Schlafapnoescreening mit reduzierten Systemen
Somnologie (2021)
Comments
By submitting a comment you agree to abide by our Terms and Community Guidelines. If you find something abusive or that does not comply with our terms or guidelines please flag it as inappropriate.