The intra- and inter-rater reliability of five clinical muscle performance tests in patients with and without neck pain

An intra-rater (between-day) and inter-rater (within-day) design was applied. Each participant attended two assessment sessions. At each occasion both examiners assessed the participant. Intra-rater reliability on two days and was examined by comparing results from the two assessment sessions, with a maximum of three working days between the assessment sessions. Inter-rater reliability between examiner A and B was examined was assessed on both assessment sessions (first and second assessment session). The study followed a three-phase reliability protocol, recommended by the International Academy of Manual/Musculoskeletal Medicine (IAMMM) [63]. The three-phase protocol consisted of a preparation, training and an overall agreement phase. During the preparation phase agreements on study conditions and logistics were achieved, while the training phase focused mainly on replicating test procedures and judgment. The aim of the overall agreement phase was to obtain an overall agreement percentage >80% between the two examiners. After completing the three-phase protocol, both physiotherapists (examiners A and B) agreed upon how to determine a given cut-point (in case a clear cut off point did not already exist) and how to standardise and perform each test.

Between September 2011 and April 2012, two recently certified physiotherapists working at a private physiotherapy clinic (examiners A and B) examined 63 participants. A third physiotherapist (administrator) independently handled the administration of patients in terms of booking appointments and handing out questionnaires. The examiners were blinded to one another’s results and to whether the participant was a subject with or without neck pain. The order of examinations was random; that is, neither physiotherapist was consistently the first or the second examiner.

The Regional Scientific Ethical Committee for Southern Denmark, approved the current study (reference number 30513). All participants gave written informed consent, and the rights of the participants were protected.

The participants consisted of two groups, who were either subjects with neck pain or a healthy reference group. Subjects with neck pain were recruited from five private physiotherapist clinics in Copenhagen, Denmark, and the physiotherapists’ consecutively referred patients, who fulfilled the inclusion and exclusion criteria. Healthy participants were recruited via advertisements in local newspapers or among friends or relatives of the three physiotherapists conducting the data collection. Patients with neck pain were eligible for participation if they met the following inclusion criteria: 1) had experienced non-specific neck pain for more than four weeks; 2) were over 18 years of age; 3) had turned to a general practitioner, chiropractor or physiotherapist regarding their neck pain; and 4) spoke and understood Danish. Patients were excluded if they had radiculopathy (e.g., positive Spurling’s Test, Upper Limb Tension Test [64, 65]). Healthy subjects were eligible to participate if they: 1) were over 18 years of age; and 2) spoke and understood Danish. They were excluded if they: 1) had neck pain within the last year causing absence from work or a significant reduction in daily activity level for more than three days; 2) had back, shoulder or elbow pain; or had 3) a rheumatologic disease (e.g., rheumatoid arthritis). In addition, all participants were excluded if they had been diagnosed with a neurological disorder (e.g., Parkinson’s disease, multiple sclerosis), diabetes or cancer; 2) were pregnant; or 3) had a history of alcohol or drug abuse.

Participants were screened for eligibility before participating in the study. If the participants met the inclusion and exclusion criteria, arrangement for the first assessment was scheduled. The first assessment took place with a maximum of five working days between the screening session and the first assessment session. Referred patients received written information materials in hard copy at the clinics. Healthy participants received written information materials via e-mail. Prior to the first assessment session, study procedures were explained in detail to the participants, and participants gave their informed consent. The administrator collected information from participants regarding their gender, age and self-reported height, weight and education level. Neck pain was recorded using a 100 mm Visual Analogue Scale (VAS) anchored with “no pain” at 0 mm and “worst imaginable pain” at 100 mm. Participants completed the Neck Disability Index (NDI) [66], a questionnaire designed to measure Activities of Daily Living (ADL) in patients with neck pain. It consists of ten items, each with six response categories (range 0–5, total score between 0–50) [66].

After completing the questionnaire, participants performed the five clinical muscle performance tests with one examiner, followed by a short break (approx. 10 min.). After the ten-minute rest period, participants performed the same five clinical muscle performance tests with the second examiner. Each test session lasted approximately 30 minutes and the order of the five tests was random. Efforts were made to ensure that all subjects were examined at the same time of day at the first and second assessment session.

Intra- and inter-rater reliability was assessed as recommended by the COnsensus-based Standards for the selection of health Measurement INstruments (COSMIN) checklist [61, 67]. For assessing intra- and inter-rater reliability, ICC agreement values with 95% CI were calculated [61, 67]. ICC agreement is preferred as it takes systematic and random errors into account [61]. Bland-Altman’s LOA [68] was used for evaluating agreement between the rater’s scores. Furthermore, measurement errors were estimated by calculating the Standard Error of Measurement (SEM) using formula: SEM consistency = SDdifference/√2 (SDdifference = Standard deviation of the mean differences between examiners A and B). The Smallest Detectable Change (SDC) was calculated using the formula: 1.96 * √2 * SEM [61, 67].

Landis [69] criteria were used to interpret ICC agreement values: slight (r = 0.00-0.19); fair (r = 0.20-0.39); moderate (r = 0.40-0.59); substantial (r = 0.60-0.79); and almost perfect (r = 0.80-1.0) reliability [69]. Primary data analyses were performed for the whole group due to the small sample size. Data were analysed using SPSS version 19.0 (IBM®, SPSS, statistics). ICC agreement values (model 2.1.A) and 95% CI were calculated using ‘scale analysis’ with a two-way random effect model and ’absolute agreement’. For JPE, average measurements are reported. For the CCFT, the NFME and NET tests’ single measurements are reported. For head repositioning, no statistically significant differences were found between the three right and three left cervical rotation trials (post hoc analysis two-sample t-test, p = ≥0.70). Therefore, data from left and right cervical rotation were pooled in the final analysis. Adequate sample size is required to achieve an admissible 95% CI for ICC values and a sample size of 50 participants is recommended to assess reliability [70]. Additionally, a post hoc analysis was performed by a two-sample independent T-test to explore possible differences in mean scores between patients with neck pain and healthy subjects. This was done although the study was not designed with power to perform a strict specificity analysis. Statistical significance was accepted at P values less than 0.05.

Summarized statistics are presented for each of the muscle performance tests (examiners A and B) in Table 2. Overall, intra-rater reliability ranged from slight to almost perfect with ICC values between 0.14 and 0.82.

A summary of inter-rater reliability statistics is presented in Table 3. Generally, inter-rater reliability was slight to substantial, with ICC values ranking between 0.19 and 0.86 (Table 3).

Firstly, for head repositioning, the number of trials performed for each movement direction has been reported to affect the estimation of precision and accuracy, with an increasing test stability (i.e., higher ICC values) attained when a larger number of trials are performed (five trials or more) [50, 51]. However, our results indicate that inter- and intra-rater reliability of neck rotation did not differ significantly from neck flexion or neck extension, despite the fact that calculations of ICC values for neck rotation were based on six trials (left and right), while ICC values for neck extension and neck flexion were only based on three trials each. A direct comparison to earlier studies should, however, be made with caution, since the methods of measurement are not directly comparable [50, 51]. Secondly, age has been reported as one factor that can affect an individual’s ability to accurately reposition their head to a neutral position [71]. In the present study the patients are significantly older than the healthy subjects, which could have increased a difference in results. In spite of this the majority of our findings indicate that there are no significant differences between patients with neck pain and healthy subjects. Thirdly, a tendency to overshoot the target position has been found in patients with neck pain [32, 45, 71]. Unfortunately, data collection in the present study does not allow for investigation of a consistently over- or undershooting as part of the observed outcome variability. Fourthly, Treleaven et al. reported significantly larger errors in neck extension and rotation (to the right) in patients with whiplash when compared with controls [48]. However, our findings do not show a similar pattern. Only data from examiner B show significant differences between patients with neck pain and healthy subjects. Likewise, the differences observed for neck extension were only present at the second assessment session, not at the first assessment session. Possible explanations for these inconsistent findings include inadequate sample size and measurement error, since our study was not designed to detect differences between groups. Even though significant differences were found, the mean differences are all smaller than the tests’ measurement errors (Tables 2–3), which indicate that the differences observed may not be evidence of a true difference, but rather can be explained as measurement error. Therefore, our results should be interpreted with caution.

For the CCFT, our findings demonstrated substantial to almost perfect intra-rater reliability and almost perfect inter-rater reliability. These findings are consistent with the existing literature [29, 34, 36, 37]. However, there is a tendency for higher ICC values to be reported with an increased number of trials performed [34, 36, 37]. When performing the CCFT, progressive nodding action increased the pressure from the baseline of 20 mmHg to 22, 24, 26, 28 and 30 mmHg. Despite the fact that the CCFT was found to be fairly reliable, the LOA and SDC were substantial (ranking between 4.11 and 5.11 mmHg). As a result, a change in score has to be at least 5 mmHg to be interpreted as a real change [61, 72]. As previously reported [12, 28, 29, 35], patients with neck pain demonstrated a reduced ability to activate the deep neck flexors, when compared with healthy subjects (Tables 4–5).

The NFME test (supine version) has previously been found reliable [25‐27, 38‐42]. Similarly, we found this test to have substantial inter- and intra-rater reliability. However, broad LOAs were determined for inter- and intra-rater reliability, indicating limited agreement between the examiners. SEM also revealed large measurement errors, with an estimation of 40 sec, estimated as the minimum detectable change. Edmondston et al. reported almost perfect intra-rater reliability with a minimum change of 17.8 sec representing a true change [25]. The mean holding time reported (≈50 sec) was almost twice the holding time reported in the current study (Table 6). However, their patient population was somewhat younger (mean age: 36 ±11) than the current patient population, which could explain the differences in holding time [73]. Previous studies have reported reduced holding time (i.e., reduced isometric neck flexor muscle endurance) in patients with neck pain, when compared with a healthy population (measured with the neck flexor muscle endurance test) [27, 44]. All three muscle performance tests indicated a tendency towards shorter holding time in patients compared with healthy subjects, although the differences were not statistically significant (Tables 4‐5). Due to the fact that patients with neck pain often are unable to perform the supine version of the NFME test, a modified version is often applied in clinical practice. The modified upright sitting version decreases the load on the neck, which for patients enables performance. By and large, our results imply that this modified version is not as reliable as the original supine version (Tables 2‐3). The SDC for the sitting version was above 97 sec (Table 2), which is longer than the actual holding time observed for both healthy subjects and patients with neck pain, implying that changes in scores should be interpreted with caution. Possible confounding factors include the presence or increase of neck pain and the number of trials performed. Olson et al. [40] and Grimmer et al. [26] reported a systematic improvement in performance from a first to a second test [26, 40] even through the tests were performed so close in time that no significant increase in muscle strength was expected. Such a learning curve could have affected the NFME test, increasing the variability of the test results. However, no statistically significant differences were found between the first and the second test indicating a learning curve did in fact not take place (Table 6).

Despite the use of a standardised protocol, the overall level of reliability for the NET was poor, suggesting that this test is too unstable to be used to evaluate neck extensor muscle endurance. Several factors may have contributed to the discrepant findings. Firstly, some of the patients experienced increased pain during the muscle endurance performance tests and neck pain has in patients been shown to affect muscle performance [74, 75]. Secondly, the order of the five muscle performance tests was random. Muscle fatigue has been found to influence muscle performance in patients with neck pain [76, 77]. Theoretically, if the NET was performed last, muscle fatigue might have affected the outcome in both patients with neck pain and healthy subjects. However, post hoc analysis showed no statistically significant differences between the first and the second assessment performed on the same day (Table 6), which indicates that muscle fatigue did not influence the test results. Thirdly, even though great effort was invested into standardising the test protocol, it cannot be ruled out that discrepancy between test procedures could have affected the results.

Test procedures for the CCFT, the NFME tests and the NET entailed each test only being performed once. This was done to replicate a clinical setting, where limited consultation time and the patient’s pain condition often confines the amount of test trials performed. In order to facilitate standardised test procedures that could be implemented in a clinic, we used inexpensive, easily accessible equipment, which allowed us, for example, to establish easily detectable cut off points at which muscle fatigue occurred and thereby reduce measurement error. Nevertheless, significant diversity was observed across the four muscle performance tests.

The order of the examiner was random. This was done in order to avoid introducing measurement bias. However, some of the muscle performance tests aimed at measuring muscle endurance, which could have initiated muscle fatigue. If so, muscle fatigue would have occurred after performing the first set of muscle performance tests. This could theoretically have affected the outcome of the second set of muscle performance tests. Nevertheless, no statistically significant differences were found between the first and the second assessment for any of the muscle endurance tests (Table 6), which indicate that this was in fact not the case.

Despite a sufficient sample size (>50 participants) we found very broad 95% confidence intervals, which points to an inadequate sample size. A post hoc analysis was conducted to compare the results from patients with neck pain and healthy subjects. This was done in order to explore whether lack of variability among healthy subjects partly could explain our present findings. Furthermore, a difference between patients with neck pain and healthy subjects could point to relevant test candidates for future studies of specificity. However, due to the small sample size in the present study caution should be made when interpreting the results.

Inter-rater reliability reflects within-day comparison of the results. This may not mimic clinical practice as muscle endurance tests are often repeated after several days. Assessment of the between-day inter-rater reliability is likely to result in greater differences. Likewise, the use of recently certified physiotherapists may have contributed to the variation. More experienced clinicians might have achieved more reliable results, since the level of clinical skills needed to conduct the muscle performance tests are somewhat high. On the other hand recently certified physiotherapists may tend to follow the written protocol of procedures more strictly as they have no empirical routine to rely on. However, in both cases the findings presented in the present study are only related to test procedures performed in a similar manner. The present study replicated a clinical setting, with a broad range of therapists, including a large group with limited experience. An assessment tool has only limited clinical value if it takes years of practice to be able to reproduce stable results.