Reliability and validity of a novel Kinect-based software program for measuring posture, balance and side-bending

First, we tested whether there were any systematic differences between the two occasions with the Paired Wilcoxon Rank Sum test, using p < 0.05 as level of significance. Thereafter, we examined the relative reliability using the Intraclass Correlation Coefficient (ICC_[3.1]),¹ where we used a two-way mixed effects model [13]. We also calculated two absolute reliability statistics through the standard error of measurement (SEM)² and the minimal detectable change (MDC),³ and these measures express the measurement error in the same unit as the original measurement for use on an individual level. The value of SEM reflects the uncertainty in the test score, while MDC represents the magnitude of change in an individual’s test score necessary to be sure that there is a real change in the construct and not a change due to measurement error. SEM and MDC are thus more clinically applicable measures than the ICC and are not affected by variability between subjects [14, 15]. The ICC-levels were classified as <20 = poor, 0.21–0.40 = fair, 0.41–0.60 = moderate, 0.61–0.80 = good, 0.81–1 = very good [16]. The SEM or MDC should preferably be as low as possible. Finally, we plotted Bland-Altman plots in order to detect any systematic difference or proportional bias [16].

Our hypothesis was that the posture, balance and side-bending performance could be repeatable, since video-based instructions were used (ICC > 0.6). On the other hand, the many different ways that a human being can perform the same functional movement suggests poorer reliability (ICC <0.4 and large SEM).

In the construct validity study, data from the third trial was used and, in case any data was missing, the second or first trial was used (in this order). The rationale for using three consecutive trials was to exclude a learning effect and to ensure that all subjects were performing the movements correctly. Each dataset was visually and statistically checked for normality by studying histograms and boxplots, comparing means and medians and by the Kolomogorov-Smirnov’s test and by the Shapiro-Wilk’s test. The data was not normal distributed, even after applying different logarithmic transformation and other transformations; hence non-parametric statistics were used. Another reason for not transforming the data is that the SEM and MDC should be expressed in their original units. The Paired Wilcoxon Rank Sum was used test to examine differences between the two datasets, using p < 0.05 as level of significance. We used scatterplots for visual examination for trends of any correlation between Qinematic™ and the motion capture system and if there were any outliers in the data. Spearman’s correlation tests were used to analyse strength of the correlation (22). Alpha level was set at p = 0.05 and the Spearman’s correlation coefficients were interpreted as follows: 0.0–0.3: negligible correlation, 0.3–0.5: low correlation, 0.5–0.7: moderate correlation, 0.7–0.9: high correlation, 0.9–1.0: very high correlation. Finally, Bland-Altman plots were also drawn to check for fixed and proportional bias (23).

Our hypothesis was that Qinematic was highly correlated (r > 0.7) with the motion capture system, as found in previous studies using the Kinect sensor [5, 6].

Discriminative validity concerns the ability of a measurement system to detect differences between two groups that are distinct differently from each other regarding the construct that is tested [11]. In the clinical study, the LBP group was compared with their healthy controls using Mann-Whitney U test using p < 0.05 as level of significance. Our hypothesis was that the variables measuring posture, balance and side-bending performance were poorer in the LBP group compared to the controls.

The analyses were carried out using Excel (Microsoft for Mac version 15.25.1) and SPSS Statistics (IBM version 23).

The Kinect device has in previous studies been found to have satisfactory reliability [6, 7], which is why the poor reliability found in our study for balance and posture seems to relate to the individual variation and our results concur with other observational studies on intra-tester reliability [2‐4], in which better results were found if video-analyses were used (i.e. excluding the individual variation). Higher reliability was found for side–bending, but only for the left side and it is difficult to know why only one side has acceptable reliability. The absolute measures of reliability were large for most of the variables, indicating that it is not possible to use these variables when monitoring a patient’s improvements over time.

In the laboratory study, we studied the construct validity, which is the type of validity that is applicable in situations in which there is no gold standard, and refers to whether the instrument provides the expected scores, based on existing knowledge about the construct (theory-based) [11]. We compared Qinematic™ data with the data obtained from the motion-capture system as done in previous studies [17, 18]. For the measures of postures, there were significant differences between the two measurement systems found for all variables except for the variable weight distribution, but for this variable the relative validity was very poor. Hence, the construct validity for variables measuring posture is low, and these results concur with a previous study in which high between device-differences (both fixed and proportional bias) were found in their Bland-Altman plots [18]. Despite this, they concluded, to our opinion incorrect, that the construct validity of the Kinect camera was excellent.

Concerning the balance variables, one force plate was used to measure postural sway as a “gold standard” for balance [19], but large sway does not always correspond to poor balance [20]. Hence, it could be difficult to measure balance capacity using sway only. The software program used in Qinematic™ calculates the position of the CoM, based on the body contours collected with the Kinect camera and uses the projection of this point to the ground to estimate the position of the CoP, while the force platform measures the CoP directly, and this measure contains also muscle forces applied to the floor. However, both systems aim to measure the construct “sway” and should give correlations above at least 0.6. This was not the case for the variables related to balance, and these results do not correspond to previous studies [6, 7, 18] which all reported higher correlations between CoM measured with a Kinect camera and a motion capture system. The reasons for this discrepancy between our and previous studies are several. In our study, we included subjects with ongoing pain, we used force plates for the calculation of CoP, we used a high sampling frequency, we used both absolute and relative statistics and we used standardized instructions, which are all factors that differ from most of the other studies. One important different is the short measurement time of the Qinematic™ system, which is much shorter compared to the Kinect systems used in previous studies. A systematic review of the test-retest reliability of CoP-variables concluded that the reliability of CoP varies depending on the design of the study and which variable of CoP to be evaluated. To achieve good reliability the sampling duration is proposed to be between 90 s to 120 s [21], and this is much longer compared to the software of Qinematic™, in which data is collected only for 3–5 s when balancing in one leg and 5–8 s during the standing task. In a previous study, good validity was obtained only for balance tests in which the subjects were testing their limits of stability and in balance tests performed with the eyes closed leading to a large sway area and sway velocity [18]. In Qinematic™, the tests are not challenging for balance to a large extent, leading to a small postural sway for all subjects, hence low between-subject variation. This result then in relative large within-subjects variation and low ICC’s; hence, poor validity.

For the measures of side-bending, an earlier study found acceptable correlations found between the Kinect camera and a motion capture system (Pearson correlations were over 0.89), but low ICC’s (less than 0.3) in both patients with early Parkinson and healthy controls [5]. These results are in concordance with ours as we found a nearly six-degree difference between the systems (29% of body height in BTS-Elite vs. 23% in Qinematic™) together with high correlations. They concluded that Kinect is not able to collect the spatial characteristics with the same precision as the timing characteristics, which seems to be a reasonable conclusion also for our results. Moreover, a recent study showed that KinectOne could not track small changes in trunk motion, which seems to be in accordance to our results [22].

We also tested the discriminative validity and we found that Qinematic™ was not able to detect differences between LBP and healthy controls in the posture and balance sections. On the one hand, large variation within the groups could have made it difficult to find significant differences (type 1 error). On the other hand, we found that the point estimates did not differ between the groups indicating no clinical difference between the groups. As previous studies have found differences in posture and balance between patients with chronic LBP and healthy controls [23, 24], our results showed that that Qinematic™ is not sensitive enough for measuring differences in posture and balance for patients with LBP.

Some of the strengths in this paper are its use of three different data collections with different study designs including both healthy subjects and patients, and different psychometric statistics and the fact that we obtained consistent results over the three studies. These results were both in concordance and contradicted previous studies that have investigated the Kinect camera [5, 17, 18, 22, 25]. We believe that the discrepancies between our study and these previous studies depend mostly on methodological differences. No other studies have previously used the Qinematic™ software program and tested these specific short static movements in which the subject was asked to stand still, and other studies used movements in which the subjects were moving through a wide range of motion or challenged in their balance, and this has a large effect on the ICC’s. Moreover, most of the previous studies have built their conclusions about validity and reliability only on correlation measures, and did not taken into account the occurrence of systematic differences, or discussed the absolute reliability measures such as SEM and MCD. These papers did not take into account the information obtained from Bland-Altman plots or follow the recommendations on how to design a reliability and validity study as suggested by the Cosmin groups [26], except for one recent study which provided both absolute and relative reliability statistics [22]. Another strength of this present study is that none of the authors had any financial relationship with the companies involved.

In the construct validity part of the laboratory study, it was possible for nearly all of the variables to be measured in the same way with the two systems systematically. However, due to the nature of the data in the one leg balance tasks, there was a need for a synchronization of the start and stop of the movements, and we used both marker data and a visual analysis for this process, which could have introduced errors into the measurement. However, it is unlikely that this led to a systematic over/underestimation of the balance capacity, which was confirmed by the Bland-Altman plots, thus we believe we can trust our data. The reflective markers on the skin could, theoretically, have altered the calculations of CoM and body positions (i.e. mid-head) of Qinematic™, since the contours could be changed, especially the marker on mid-head. It is, however, difficult to test to what extent this occurred.

Concerning the subjects, the sample was large (> 100 subjects included), and was a convenience sample consisting of both males and females with a range between 15 and 69 years of age, which has to be considered as representative of a working-age population. On the other hand, most of the subjects were physically more active than a normal population visiting a physiotherapy clinic, which could impair the generalizability of these results to a patient population. Some of the subjects had suffered from pain, while the majority were pain-free the last week, and this could also be seen as representative of a general population. However, when sensitivity analyses were performed in which we calculated the psychometric properties for these two groups separately (physical active and non-active subjects; pain and pain-free subjects), no differences were found, indicating that these results are robust for several populations.

In validity and reliability studies there are recommendations for the sample size to be at least 50, but larger samples are preferred [26, 27]. The reason why no more than 30 subjects for each sub-sample were required was due to lack of time, and a study population of 50 or more was not considered realistic. However, our sample sizes were 2–3 times larger compared to the studies discussed earlier [5‐7, 22] in which the sample size was less than ten subjects, except for a recent study, in which 20 subjects were included. We believe that a larger sample size in this present study would not have affected the results, except for narrowing the IQR and lowering the p-values. The use of non-parametric statistics can be discussed. Still, using parametric tests did not alter our results to a large extent. Since we calculated both ICC and Spearman correlations in the validity study, we are able to make comparisons with previous studies, and we retrieved nearly similar results. We have tested Bland-Altman plots for all variables, but there were no additional fixed or proportional biases found in these plots and, due to lack of space, these were not included in the paper.

Qinematic™ has one more part, the motor control part, in which the subject performs two different types of squats. In our next paper, we will present the results of the same psychometric analyses together with the face validity of these tests. Although the results from the presented studies are somewhat disappointing, we believe it is important to reveal the results from this section as well, since dynamic tests like squats are regularly used in all kinds of settings: sports, rehabilitation, physiotherapy, etc., and could be of higher interest compared to the static tests of balance and posture.

We believe that Qinematic™ has the potential of adding important information and as a pedagogic tool in the clinic, and it is exactly the type of software that has been lacking [5]. However, in the current version of Qinematic™, it seems that the individual variation of performing the tests together with the measurement error during the tests of balance, posture and side bending is too large for monitoring an individual over time or comparing different populations on a group level. Perhaps the introduction of cut-offs for good/poor balance, posture and side-bending capacity into dichotomous variables could reduce the large inter- and intra-individual variation and could give directly valuable information to the clinician. However, it is a challenge to define these cut-offs, since no such information is available in the literature, and there is a need for prospective studies on this purpose.