Validation of an open-source ambulatory assessment system in support of replicable activity studies

Wearable devices that track physical activity and vital signs like heart rate for health promotion and monitoring have become prevalent, with fitness trackers or smartwatches enjoying a rise in popularity. In a recent Gallup poll, 19% of US adults report at some point having worn a fitness tracker or smartwatch (34%) or having tracked their health statistics on a phone or tablet (32%) (McCarthy 2019). Commercially available activity trackers are more and more used in clinical trials to assess the physical activity behavior of study participants over time. When the tracker’s original sensor data are not recorded but instead preprocessed descriptors such as counts, steps, sedentary bouts, or activity levels are logged in the study, replication problems arise. Similarly, when such tracking devices are upgraded or have reached the end of their shelf-life, the hardware and algorithms that produced the data are often lost as well.

The past decades have seen a shift in the use of activity trackers for various studies and clinical trials. Objectively measured physical activity data are more accurate compared to subjective physical activity assessments like physical activity questionnaires (Garriguet et al 2015) with frequent overreporting of vigorous activities and underreporting of sedentary behavior (Fiedler et al 2021; Lines et al 2020; Verhoog et al 2019). For physical activity and health promotion strategies and interventions, the accurate recording and appropriate feedback to the user seem essential for intervention success and data interpretation. Among the commercially available trackers, the ActiGraph¹ series of inertial-based trackers are recognized as well validated and reliable devices and have thus far been predominantly used for a majority of published research studies (Wijndaele et al 2015).

Although the internal inertial sensors meanwhile come as standard chipset packages with well-understood data, are well-calibrated, and have common interfaces, commercial inertial-based trackers are commonly sold as black boxes, about little is known about the accuracy of their algorithms (such as step detectors) or inner workings (Brondin et al 2020). Studies have shown that there remain uncertainties about the reliability of the step counts reported by trackers, for instance, during low-intensity activities and when walking with assistive tools (Alinia et al 2017). Few algorithms that perform step detection in commercial inertial-based trackers’ data are published as open-source code so that they can be reproduced, the work of Brondin et al (2020) being an exception.

With the advent of inertial-based trackers, algorithms have been developed to characterize the activity of the wearer over time. Whereas earlier studies focused on device-specific measures, such as counts from legacy devices in which a device-specific acceleration threshold is exceeded (Brønd et al 2017), more recent algorithms are increasingly defined on standard units that remain valid on any recordings of raw inertial sensor signals, even when the trackers are exchanged. In a study by Migueles et al (2019) on young adults wearing the ActiGraph GT3X inertial sensor, for instance, cut points were empirically found for dominant and nondominant wrist placement, based on the Euclidean norm minus one \(g\) (ENMO) (van Hees et al 2013). These values were found to be: sedentary time (less than 50 milli-g), light physical activity (50-–110 milli-g), moderate physical activity (110–440 milli-g), and vigorous physical activity (more than 440 milli-g). Other notable procedures include mean amplitude deviation (Vähä-Ypyä et al 2015), High-pass filtered Euclidean norm (van Hees et al 2013), high-pass filtered Euclidean norm plus (van Hees et al 2013), proportional-integrating mode (PIM) (Jean-Louis et al 2001), zero crossing mode (Acebo et al 1999), and time above threshold (Fekedulegn et al 2020).

Similar efforts haven been under way for directly extracting other measures, such as accumulated number of steps, directly from the raw and normalized acceleration signals so that these become independent from the used sensors or hardware.

The aim of the studies presented is to further develop and evaluate an open-source ambulatory assessment system that produces data that can be complemented and compared against later. This consists of generic software and open algorithms for capturing physical activity under free living conditions, with a special focus on persons with noncommunicable diseases (diabetes, cardiovascular diseases), which can be used for research questions as well as health promotion strategies.

Our proposed system consists of an open-source smartwatch, which records raw accelerometer data simultaneously with locally computed aggregation measures as an immediate feedback to the wearer, a smartphone app that collects and forwards the collected data from the smartwatch at regular intervals, and a back-end server database. For all three components, we contribute with open-source software and algorithms.

Two subsequent studies were performed to evaluate the use of our system: one laboratory study in which participants were asked to move on a treadmill at varying speeds to elicit different activity levels (light, moderate, vigorous) and to compare the raw accelerometer readings. A second study was subsequently performed to measure physical activity behavior for one day during participants’ daily routines. In both studies, participants were asked to wear our customized Bangle.js tracker at the wrist and the ActiGraph GT9X sensor as the reference device, at the hip and at the wrist. The study was approved by the ethics committee of the medical association of (anonymized) and was conducted in accordance with the declaration of Helsinki. Participation was always voluntary and informed consent obtained from all participants. To support the following data analysis and visualization routines, we developed and used a series of custom Python-based scripts that are supplied with this article’s code. The Declarations section at the end of this manuscript provides details on where to obtain the source code.

The following sections will present the outcome of the experiments as described in the previous section, using both visual inspection of the accelerometer signals, as well as through the aggregated performance metric of steps. All code for the analysis was written in Python and is made available online⁴.

In summary, we proposed an approach that relies on strictly open-source components in order to be able to record study data that can be re-analyzed and supplemented afterwards, even after the used devices and software have reached their end-of-life limit or have gone out of production. For mere collection, this means that raw acceleration data needs to be recorded at known units of sensitivity and sampling, but for studies in which devices offer direct feedback to the participants, for instance displaying the steps taken during the day, this also requires open-source algorithms.

The purpose of the studies presented here was to validate our approach of using a low-cost smartwatch and compare the output, both the original sensor values and the aggregated measure such as steps. We took recommendations of the framework to evaluate devices that assess physical activity behavior into account, with a focus on phase I (laboratory development) and phase III (naturalistic validation) (Keadle et al 2019). Experimental results can be summarized into the following points. (1) The three-dimensional accelerometer data, when sampled at similar rates and set to the same range, are highly comparable between devices. The preliminary study where study volunteers walk and run at different speeds on a treadmill show an almost perfect correlation (Pearson correlation coefficient: 0.97) between the wrist-worn devices. (2) The detected steps in natural environments from the open-source algorithm from our custom wrist-worn tracker show strong correlation with the detected steps from a hip-worn ActiGraph using the ActiLife software suite. Deviations in step counts between the two systems can be seen by visual inspection especially in periods when study participants were only walking for brief, intermittent periods.

Our studies focused mostly on raw acceleration signals and evaluated step counts as measured by our algorithms and compared against the ActiLife software suite. Although the Actigraph GT9X is not the gold standard for validating steps, it is probably one of the most frequently used devices in physical activity research and, therefore, seems appropriate for study purposes. This is by far not the only measure that can be put to use for assessing physical activity in daily life. Future research will inadvertently have to focus on further measures that are both fast to comprehend by participants and can be calculated from any given acceleration signal. Robust estimates for specific exercises and activities, energy expenditure, or calorie consumption could be candidates for such measures and are similarly developed as an open-source code base.

Beyond the recording of activity data in a replicable way, systems such as the one presented in this manuscript would form an optimal basis for designing future just-in-time adaptive interventions, where events in the sensor data can trigger interventions on users’ phones or smartwatches, as presented in Ebner-Priemer et al (2013) and Giurgiu et al (2020) for instance for smartphone-based triggers based on sedentary behavior and physical activity. Interventions based on events detected in the sensor data could be integrated directly on the smartwatch, thus guaranteeing that participants would immediately become notified, but would need additional attention to the usability and the reduced interaction modalities on such devices.

The anonymized experiment data described in this article will be placed online so that they are made available publicly for further analysis and comparisons: (URL anonymized for double-blind review).