Technology Assisted Rehabilitation Patient Perception Questionnaire (TARPP-Q): development and implementation of an instrument to evaluate patients’ perception during training

The study was composed of two subsequent phases: questionnaire development through Delphi methodology and questionnaire application. In the second phase, an observational, feasible, multi-centric study relating to patients’ experiences with TAR was conducted in the Neurorehabilitation Unit of Istituti Clinici Scientifici Maugeri Spa SB- IRCCS Montescano, Italy, and at the Habilita Zingonia Centre of Ciserano, Italy.

The study design and the protocol were submitted and approved by the Institutional Review Board and by the Ethics Committee (Comitato Etico Istituti Clinici Scientifici ICS Maugeri Pavia, approval CE number 2206 date 29.5.2018) and were implemented following the World Medical Association code of Ethics (Declaration of Helsinki, 1967).

A multidisciplinary team (neurologists, psychologists, bioengineers, physiatrists, and rehabilitation therapists specialized in technological devices for rehabilitation) was constituted to specifically work on the creation of the Technology Assisted Rehabilitation Patient Perception Questionnaire (TARPP-Q). The working board aimed to create a multidimensional, self-administrable, and close–ended questionnaire to evaluate the different facets of patient experience with TAR devices and exergaming.

The process followed a bottom-up approach started by freely observing inpatients during high-technology training; clinical observations prompted a literature revision on the development of questionnaires administered to assess patients’ experience with TAR. Based both on the existing literature and on clinical experience, the team independently and freely identified a pool of items that were then submitted to a Delphi methodology and progressively reviewed to reach a general agreement [30] (Fig. 1 Questionnaire construction process). The same procedure was applied to determine the questionnaire format, items’ number, questionnaire length, and rating scale degree.

The questionnaire was developed and administered in Italian and subsequently translated into English.

The full version of the questionnaire in English is provided for understandability purposes. Three independent investigators (AG, ST, RCa), whose native language was Italian, translated the original version of the questionnaire into English. An idiomatic translation was preferred over a word-for-word transposition. The investigators compared the 3 translated versions, with consensus. Lastly, an independent translator who had not participated in the first stage and whose native language was English supervised and finalized the last translation, with consensus.

The first version of the TARPP-Q, obtained through Delphi consensus, was administered over 6 months to consecutively admitted inpatients. Patients underwent rehabilitation training with TAR and exergaming at the third (T0) and the last training session (T1).

Inclusion criteria: in-patients, regardless of gender, aged between 18 and 85 years of age, admitted for a TAR program with exergaming due to a neurologic disease.

Exclusion criteria: cognitive deficits (MMSE ≤ 22) [31], insufficient knowledge of written or spoken Italian language, aphasia, severe visual or auditory deficits.

The following data were collected at admission: age, years of education, type of disease, and disease duration. In addition, the FIM scale [32] was administered at T0 and T1. Patients were requested to give their informed consent to the study and the authorization of scientific treatment of their medical records in an anonymous form.

According to their prevailing functional deficit, patients were subjected to a daily upper limb (effective 30-min) technology-assisted treatment or to a daily (30-min) technology-assisted gait training for 5 days/week for 4 weeks. The first and the second session were for adaptation. There was no break during gait session training, while there was a 1-min break between one exercise and another to select the following exercise by the operator during upper limb rehabilitation.

The training was conducted with exergames integrated with the upper-limb and gait devices. The exergames of the two devices used are defined by the vendor (Hocoma, Switzerland) as Augmented Performance Feedback (APF) activities. More specifically, the devices present an inbuilt software that can be used to enhance motivation by providing visual and interactive screen feedback on the ongoing motor performance. The active and live effort from the user is therefore represented on a display monitor (scores, accuracy rates, etc.) through different available games.

Three examples of exercises were below detailed (Figs. 2, 3, 4).

Technology-assisted gait training was performed by Lokomat® (Hocoma, Switzerland) constituted by an exoskeleton, a body weight support, and a treadmill. It facilitates symmetric hip and knee gait patterns thanks to its exoskeleton, driven by programmable actuators.

Rehabilitative exercises with APF are carried out in a semi-immersive environment. During all training sessions, patients were placed in front of a screen where a representation of the patient simulated walking in a virtual landscape, guided by the patient’s movements.

The Functional Independence Measure (FIM®) scale measures the patient's disability level and indicates the degree of assistance required for the subject to carry out activities of daily living; it is largely adopted in rehabilitation as a functional outcome indicator [32]. It consists of 18 items, considering motor (13 items) and cognitive (5 items) functions, respectively. Each FIM item is scored on a 7-point Likert scale, where 1 indicates complete dependence and 7 represents complete independence. The total score ranges from 18 (complete dependence on all items) to 126 (complete independence assessed in all domains). Two sub-scores can be assessed: FIM motor (range 18–91) and FIM cognitive (range 0–35). All the evaluations were performed by certified health care professionals.

Descriptive statistics for individual TARPP-Q items and discrete variables are reported as numbers (percentage frequency).

To investigate the existence of an underlying structure in the interrelationships among the items of the questionnaire, exploratory factor analysis was carried out.

The principal factor method was used to extract factors, followed by orthogonal rotation (Varimax method) to assist in the interpretation of the factors and to ensure that they were uncorrelated.

The determination of the number of factors to extract was guided by theory (variance explained by each factor) and seeing which number of factors yielded theoretical meaningfulness and the most interpretable results. The determination as to what the underlying factor might represent was inferred from the variables significantly loaded on their factors. An absolute factor loading value ≥ 0.35 was considered to indicate that a variable contributed significantly to a factor.

The following were verified: at least three variables (items) had to load significantly on each retained factor, the variables that loaded on a given factor needed to share conceptual meaning, and the variables that loaded on different factors considered different constructs.

Internal consistency of the factors measured in the questionnaire was assessed using Cronbach’s alpha.

The association between TARPP-Q factor scores (values at admission, at discharge, and the difference between values at discharge and admission) and age, education, and FIM total and sub-score, were analyzed by correlation analysis (Spearman r).

As to the questionnaire application, within-group comparisons were carried out by the Wilcoxon signed-rank test.

All statistical tests were two-tailed and statistical significance was set at p < 0.01. All analyses were carried out using the SAS/STAT statistical package, release 9.4 (SAS Institute Inc., Cary, NC, U.S.A.).

A final consensus was obtained on a 31-item TARPP-Q provisional questionnaire, constituted of 10 main questions, of which 5 with multiple answers. The TARPP-Q response to each item was rated using a four-level Likert-type scale (forced choice, no neutral option): strongly disagree (score 1), disagree (score 2), agree (score 3), strongly agree (score 4).

Forty-four patients were included in the study. Demographical and clinical data are reported in Table 1.

All patients answered the questionnaire autonomously. There were no unaccounted answers

The responses to individual TARPP-Q items at T0 and T1 are reported in Table 2. For the sake of completeness, the mean values ± SD of all the items are also reported.

After the analyses, two items (“How would you describe your experience with the device to friends/relatives? Unsatisfied” and “Instructions given by the physiotherapist were useful”) were excluded from the ultimate version of the questionnaire. Specifically, the first item did not load to any of the four factors, then was removed due to its construct inconsistency. The second one was deleted as diverting the attention from the primary interest of the questionnaire—that is, specifically, the patient-device interaction—by introducing another variable, such as the operator’s role. The authors became aware of the inconsistency of this item within the construct coherence of the questionnaire during the patient’s assessment phase only, therefore with a unanimous agreement the item was removed from the questionnaire.

The final version of the questionnaire consisted of 10 questions, 5 with multiple answers, totaling 29 items.

At the factor analysis, four factors emerged, which were named as follows, after a further consensus amongst the members of the multidisciplinary team: Factor 1: Positive attitude, Factor 2: Usability, Factor 3: Hindrance perception, and Factor 4: Distress. The values of Cronbach’s alpha ranged from a moderate 0.72 (“Distress”, Factor 4) to a high 0.92 (“Positive attitude”, Factor 1). In Table 3 items’ analysis of the TARPP-Q is reported. Cronbach’s alpha variations, obtained by removing single items from the factor, are reported in rows for each item. When appropriate, reverse scoring was applied to selected items before their inclusion in the pertaining domain.

In Additional file 1 printable English version is available.

In Additional file 2 printable Original Italian version is available.

Descriptive statistics for the total TARPP-Q score and the FIM (motor, cognitive and total) at T0 and T1 and the 4 factors are reported in Table 4.

FIM scores and all TARPP-Q factors, except for Distress (p = 0.11), significantly improved at the end of the rehabilitation treatment.

The association between age, education and FIM scores, and TARPP-Q factors were also assessed, but no significant relationship was observed with any domain at T0 nor T1. Analogously, no significant association was observed between age, education, and changes (values at T1 – values a T0) in FIM scores vs changes in TARPP-Q factors.

Table 5 reports the correlation analysis between factors.

Overall, the TARPP-Q showed sound psychometric properties. Four factors were identified by an exploratory factor analysis, which resulted in coherence with clinical experience (construct validity) and proved to have a good internal consistency (sound values of total Cronbach’s alpha and of Cronbach’s alpha variations, obtained by removing single items from the factor).

Positive Attitude (Factor 1). The first factor had high internal consistency (Cronbach Coefficient Alpha = 0.92) and included items related to positivity, amusement, comfort, aid, and engagement. Given the nature of the item aggregation, the factor was labeled as “Positive attitude”—referring to patients’ positive set of emotions, beliefs, and behaviors toward the technology-assisted device. Although structural and ergonomic characteristics of devices are indeed drivers of treatment efficacy [35] and safety [14], patients’ Positive attitude could positively impact the device constant usage, preventing discontinuations and drop-offs. Literature findings suggested how personality traits and beliefs may also influence change-promotion behaviors while using devices [36, 37]; moreover, a sense of comfort (items 5A and 7A) is a prerequisite for a safe application of technology-assisted devices over time [14] and primary requirement for a stable motor recovery [38, 39]. In addition, amusement (items 5D and 7D) and enthusiasm (item 10A) also affect rehabilitation outcomes, particularly when related to the use of exergames [9, 40]. Similarly, engagement (items 8A, 9, and 10B) and well-being (item 6B) are known to impact the training, even if mediated by the type of training or device (repetition and variation) [41, 42] and by patient expectations [25]. Lastly, patient perception of aid (item 8C) while performing exercises is only partially covered in literature and deserved further investigation [43].

Usability (Factor 2). Usability can be defined as the capacity of a system to allow users to perform tasks safely, effectively, and efficiently while enjoying the experience [44]. Assessing usability in rehabilitation ensures device maximum functionality, whilst increasing effectiveness, engagement, and ease of learning. As a result, the usability of TAR devices has been widely investigated, with encouraging results as to patients and health care professionals’ device perception and execution of movements that are accurate, natural, and harmless for the patient [25, 44, 45]. Items about Usability emerged as a factor in TARPP-Q, showing a sound internal consistency (Cronbach Coefficient Alpha = 0.84) and collecting items investigating ease and enjoyableness of use. Interestingly, two items related to perceived performance improvement also loaded to Usability (4, 6E). On this note, is it worth noticing how a user-friendly device leads to positive perception (items 1, 2, 3), helping movement execution (items 4 and 6E) in a way to influence treatment motor outcomes and exergames scores [9, 46, 47].

Hindrance perception (Factor 3). The third factor found (Cronbach Coefficient Alpha = 0.73) collects both negative implications on motor execution of exercises (i.e. poor movement control; item 6D) and psychological facets that may interfere with device usage, such as feeling awkward (items 5E and 7E) and being uncomfortable (items 5B and 7B). In the current context, Hindrance perception is to be referred to as subjective sensations (constraint, impediment) deriving directly from “wearing” the exoskeleton while performing exercises, as already expressed in other works [48, 49]. It may be also inferred that “feeling” the device guiding movements and enveloping body parts might be perceived as something out of ordinary, determining a bizarre experience. Indeed, the present distinction deserves further clarification, especially regarding the subjective experience of motor control and body constraint. The emerged meaningful concept, however, witness the multi-level complexity of TAR [38], encouraging the adoption of a holistic perspective to treatment [25, 42]. Negative perceptions towards technological devices are indeed a wake-up call for health care professionals, as they might considerably interfere with motivation and result in treatment discontinuity or rejection [50].

Distress (Factor 4). Distress can be defined as a state of emotional suffering associated with stressors and demands that are difficult to cope with in daily life [51]. As a barrier to technology-assisted devices, Distress emerged as the fourth factor, showing a moderate internal consistency (Cronbach Coefficient Alpha = 0.72). Recent studies suggested conducting an in-depth analysis of the relationship between high-technology or robotic devices and psychological responses [52]. Consistently, assessing negative feelings and psychological domains could provide considerations of clinical interest (items 5F, 7F, 8B, 8D). In this regard, given the exploratory factor analysis, exergaming-items 8B (“Seeing the score reached on the screen: makes me feel under pressure”) and 8D (“Seeing the score reached on the screen: limits me”) (“Distress”) have been included to Factor 4, to further characterize the distinction emerged during the analyses: Factor 4 (Distress), mostly focusing on psychological shades of performance limitations and distress, versus awkwardness and physical sensation of impediment, pertaining Factor 3 (Hindrance perception).

Accordingly, items 8E (“makes me feel inadequate”), 5C, 7C (“clumsy”) and 6A (“discomfort”) evaluate psychological facets of a patient’s perception of TAR. However, further studies are needed to systematically investigate the role of specific psychological traits (such as self-esteem) and emotional statuses during the treatment by using tailored instruments [10, 33, 36, 42].

Finally, factor analysis showed that item 6C “fatigue” (“While exercising with the device, I experienced: fatigue”) loaded to Factor 4 (Distress). Fatigue is a central issue for motor rehabilitation, particularly in neurologic diseases such as Parkinson’s disease [53], multiple sclerosis [54], and stroke [55], and prominently affects treatment motivation and rehabilitation efficacy [56]. However, in the field of rehabilitation, fatigue, both in its definition and etiology, is still under debate. For this reason, the item “Fatigue”, which resulted included in Distress—and thus in a psychological factor—could be regarded as such for its psychological implications only. Future studies on fatigue together with large-scale applications of TARPP-Q may better clarify its role.

As to the questionnaire application, statistical analysis among the TARPP-Q factors, demographic, and clinical variables such as age, education, and patients’ disability level (expressed by FIM score at T0 and T1), identified no significant correlations. This result suggests that, regardless of age [15], socio-cultural context, and level of disability, patient technology acceptance is largely positive in our sample. In this respect, the age of the patient, for example, might be a conditioning factor in the acceptance of TAR [15]. In previous work, similar results were obtained by using the PIADS scale [24].

The four factors’ pre-post training changes (Table 5), obtained during the questionnaire application, further clarify this latter consideration: while FIM scores (Motor and Total subscales), Positive attitude, Usability, and Hindrance perception significantly improved at T1. Distress only remained unchanged after training.

The significant increase in Positive attitude and Usability factors is probably due to progressive experience determined by the training itself, whilst enhancing patient competence [41, 57]. As for the improvement of Positive Attitude, it is well known that patient perception of rehabilitation devices is associated with treatment satisfaction and not exclusively with physical improvement [10, 16]. Overall, results confirm the role of Positive attitude and Usability, while highlighting that patients’ psychological traits are crucial determinants in the interaction with a technology-assisted device [37]. Notably, in our sample, Usability resulted significantly correlated to patient Positive Attitude, proving the pivotal role of patient positive feelings towards the perceived usability of TAR.

Concerning both the factors Hindrance perception and Distress, the frequency distribution of TARPP-Q single item responses at T0 (Table 2) showed low levels of distress and negative perception. Consequently, fear and skepticism towards technology-assisted devices [34] resulted to be low or at least controlled in our sample. This might be due to different reasons, ranging from effective patient-healthcare professional communication to the personal characteristics of the participants. Future studies may furtherly investigate this theme, suggesting best practices to empower patients’ disposition during the interaction with TAR devices.

Interestingly, at present, factor analysis showed that items with a negative meaningful concept aggregate into two separate factors (Factor 3, Hindrance perception; Factor 4, Distress). Given the overall similarity of the contents, further studies are needed to better define the nature of the theoretical differences between these two factors.

As to patients’ perception of movement (6D. Poor control of my movements; 6E. Better control of my movements; 4. Movements (walking, use of arm) improved with the device), the TARPP-Q highlighted a betterment of both motor control and functionality. Indeed, clinical implications of movement control in TAR are noteworthy and still partially to be investigated [44, 45].

Finally, concerning the exergaming section of the questionnaire, factor analysis showed that pertaining items were distributed in separate factors (Factor 1, Factor 3, Factor 4). The non-aggregation may suggest that the role of exergaming in neurorehabilitation is multidimensional and still to be clarified, as a result of the complex combination of both motor and cognitive effects [58‐60].

TARPP-Q development was exclusively guided by collecting theoretical inputs from empirical observation and clinical experience. Nevertheless, the questionnaire aims to collect information from a relatively novel field [3, 18], an assumption that account for the exploratory nature of the study. The questionnaire, however, was administered in expert-lead settings, consisting of an interprofessional board of specialists in neurology, physiatry, physiotherapy, psychology, and bioengineering, whose strict and ongoing confrontation—together with the implementation of the Delphi procedure—may have strengthened its reliability.

It is important to highlight that the present results derive from the self-evaluation of patients who have carried out rehabilitation exclusively with fixed exoskeletons. Future studies may ascertain if similar considerations could be extended to other technologies (wearable, portable devices, etc.) which perform as motor “aids” to patients’ body parts during natural gait or upper-limb movements. Also, further studies are needed to determine the temporal stability of the TARPP-Q and to better define the nature of the two “negative” factors (Factor 3, Factor 4).

Given the multiple facets of exergaming and rehabilitation, future refinements of the TARPP-Q may consider extending the number of items dedicated to the theme (including a more structured reference to Virtual Reality and Augmented Reality), to provide a fine-grained analysis of the phenomenon [60, 61].

As to the pilot study, the main limitation is represented by the small sample, mainly composed of older people. However, it is well known that studies on this topic often consider a small number of patients. Indeed, a strength of the questionnaire relies on its multidisciplinary, as assessing general features of technology-assisted devices and exergames perception to describe both motor and emotional implications of patient-device interaction through a combination of determinants such as device usability, motor control, attitude, fatigue, and distress.

Lastly, the questionnaire was administered to patients who spoke Italian only and still needs full validation in terms of reliability, validity, and sensitivity. Rigorous validation of the TARPP-Q in the English language will be the object of a second study, possibly involving an international consortium so to allow comparison of validation results. Future developments of this questionnaire might also help shed light on the different nuances of meaning that might have been “lost in translation” during the Italian-English transposition.