Questionnaires

The ÖMPSQ is a widespread screening-questionnaire first developed in Sweden [24] to assess pain (5 items), functional ability (5 items), psychosocial (7 items) and work related variables (4 items) and is recognised in the German guideline for back pain [9]. The main intent was the early identification of psychosocial risk factors about sickness-related absenteeism. Its latest and modified version consists of 25 items [19,25]. Four items concerning sociodemographic information (age, gender, nationality, employment status) were excluded from scoring. Each item score ranged from 0 to 10 points except for three ranging from 1 to 10 points. For one of these 3 items with five categories (number of pain sites) the numbers were multiplied by two. To allow lower scores to indicate a lower risk, items 12, 16, 17, 21–25 have been inverted. Therefore, the linear total score ranged from 4 to 210 to identify individuals with low risk (< 90), medium risk (90–105) and high risk (> 105). Two work related items included an additional box „not working”(items 8, 17). If checked these were treated as a missing value [25]. In reference to Linton and Boersma in 2003 [19], the main outcome of sick leave was extended by adding functional ability and pain. A joining cut-off of 90 showed a sensitivity and specificity of 89%/65% for sick leave, 74%/79% for functional ability and 76%/70% for pain.

To evaluate the predictive validity for the ÖMPSQ we used a German translation developed by T. Kohlmann, C.O. Schmidt and M. Pfingsten in 2007. Development is based on international recommendations for self-reported measurements [26], whereas two independent translations and one re-translation were conducted [23]. It differs slightly from the original ÖMPSQ by including six categories for the number of pain sites and three work related items instead of two with the additional box „not working”(items 8, 16, 17). One sociodemographic item concerning nationality had been removed. The total score ranged from 4 to 212 respectively. Cut-offs regarding the original ÖMPSQ changed in consequence (< 91, 91–106, > 106).

The HKF-R 10 from 2006 [20] was intended for the early identification of psychosocial risk factors about persisting pain. This questionnaire has been also recognised by the German guideline for back pain [9]. It consists of 27 items assessing sociodemographic information (gender and graduation), pain intensity and duration (4 items), efficacy of massage (1 item), the Zung’s depression-scale (5 items) and the Kieler Schmerzinventar (KSI) consisting of catastrophising (5 items) and helpless-/hopelessness (9 items). Item 5 (average pain intensity during the last week) is excluded from scoring and should only be used as follow-up documentation. The non-linear total score was computed by summation of negative and positive sum products using an underlying multiple regression model. Individuals were divided into 5 categories: low risk (score < = -2.5), probably low risk (-2.5 < score < = 8), no prediction possible (8 < score < 28), probably high risk (28 < = score < 37), high risk (score > = 37). Sensitivity for outcome of pain was stated with 75% and specificity with 79%.

Outcomes, Key-Variables and Follow-Up Questionnaire

About our study approach to replicate the ÖMPSQ in a German sample, we strictly followed the methodology of the original study in defining the key items and assessing the outcomes of pain, sick leave and function after six months. Regarding the HKF-R 10 (not validated during this study period), specifications set by EN were used to assess the outcome of pain.

The follow-up questionnaire was designed to assess the outcomes referenced above. It included all individual items of both questionnaires at first data collection to provide additional information if needed. Items to assess the outcomes read as follows.

Sick leave for the ÖMPSQ was assessed by adding self-reported days of sickness related absenteeism during the follow-up period (equivalent to item 6 but with a duration changed from 12 to 6 months). For function, the activity related items 21–25 were used.

Pain was assessed by pain intensity and pain frequency during the last 3 months (items 10+11).

For the HKF-R 10 item 5 considering the average pain intensity during the last week was used to assess outcome of pain. As mentioned previously, this item was excluded from scoring at baseline.

Outcomes for the ÖMPSQ were defined as follows.

Self-reported sick leave was divided into three groups: individuals with no sick leave (0 days), with short-term sick leave (1–30 days) and long-term sick leave (> 30 days). Defining sensitivity and specificity is prevented by using variables with non-dichotomous characteristic attributes. Therefore, ordinal or metric endpoints of outcome variables with more than two categories were artificially dichotomised to classify patients as “test-negative” (recovered) or “test-positive” (not recovered) only. Although loosing subgrouping information we dichotomised self-reported sick leave to divide patients with no sick leave (0 days) from those with short or long-term sick leave (> 0 days).

For function the scores of items 21–15 were summed (possible range 0–50) and a cut-off point at 45 was set to separate recovered from non-recovered patients (> = 45 recovered, < 45 not recovered). Pain for the ÖMPSQ was defined by multiplying items 10+11 (possible range 0–100). The cut-off was set at 17 to separate recovered from non-recovered patients (< 17 recovered, > = 17 not recovered). For the HKF-R 10 we used a cut-off set at 30 to separate recovered from non-recovered patients (< 30 recovered, > = 30 not recovered).

Raw data handling and statistical analysis

For input and processing, such as computing of total scores and sub-scores MS Excel 2013 for Windows was used. Each data set contained items of both questionnaires and was divided into two partial data sets to handle raw data validation and further statistical analyses for each of the questionnaires. Missing values were not allowed for the HKF-R 10. For the ÖMPSQ we followed Linton’s approach to allow total score mean substitution [25]. Regarding the 3 checkboxes “not working” (treated as a missing value) we decided to allow a maximum of 3 substitutions, which is about 14% of 21 available items. In case of double entries on these items, an existing VAS-value overrides a checkbox value to avoid impending mean substitution. Raw data for each partial dataset was treated as valid if it had ended up in completeness (ÖMPSQ = 21 items, HKF-R 10 = 27 items). Invalid partial data sets were excluded.

IBM SPSS Statistics 20.0 and XLSTAT 2013 for Windows were used for calculations. The statistical significance level was set to 5% (α = 0.05) and the confidence interval (CI) to 95%.

Preliminary analyses of variance (ANOVA) were conducted to evaluate relationships between scoring levels at baseline and the characteristic attributes of the outcome variables. The level of mean differences between a low or a high total score at baseline (indicating a lower or a higher risk of CNSBP) and the outcome classification at follow-up (recovered/not recovered) indicates a missing or existing relationship.

To evaluate the discriminatory ability of both questionnaires, ROC curve analyses (Receiver Operation Characteristics) were conducted. The area under the curve (AUC) has been deemed as the most common measure of quality of information [27,28,29]. All additional information was provided by the corresponding tables of the ROC-curve coordinate points and its dependent values. Included measurements were cut-offs, sensitivity and specificity, prevalence’s, positive and negative likelihood-ratios/predictive values (PLR, NLR, LR, PPV, NPV, PV), accuracy of estimation (AAC) and sum of distributions. As recommended elsewhere, the Youden-Index (J = (Sensitivity+Specificity)-1) was used to determine the optimal (statistical) cut-off that proves the maximum potential effectiveness between sensitivity and specificity and affects the derived likelihood-ratios as well as the predictive values inside a sample [30,31]. The PPV and NPV (post-test probabilities) mainly depend on the prevalence of a disease (pre-test probabilities). In case of a sample’s prevalence that did not match the epidemiological one in the German population (10–15%), we followed statistical recommendations to re-calculate the PPV and NPV.

Classifications of test efficiencies concerning likelihood-ratios and its relationship to prevalence and predictive values are based on references that were found elsewhere [32,33,34]. Quality measurement of AUC’s and sum levels of sensitivity and specificity were classified for reference as follows: < 70% (sum < 140%) = low value; 70–80% (sum > = 140%, < 160%) = moderate value, 80–90% (sum 160%–180%) = good or high value, > 90% (sum > 180%) = very good or very high value (based on [32,34]). Tables 1 and 2 illustrate these classifications.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Test efficiencies in regard to likelihood-ratios.

https://doi.org/10.1371/journal.pone.0158850.t001

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Dependencies between predictive values and prevalence.

https://doi.org/10.1371/journal.pone.0158850.t002

Baseline Characteristics

For the ÖMPSQ (n = 241) the mean age was 43 years, 65% of this distribution were female and 49% were full-time working. 63% of the participants reported multiple pain sites. Sickness-related absenteeism due to pain for more than a minimum of one day during the past year was reported by 37% of this sample. Pain duration of more than one week was reported by 94% of all participants and 15% reported pain durations of longer than 24 weeks. Mean pain intensity during the last week was 5.5 and 4.8 during the past 3 months. Mean pain frequency during the past three months was 6.

Distributions for the HKF-R 10 (n = 242) read as follow: 67% were female and 42% reported O-Level graduation. Pain duration of more than 8 days was indicated by 88% of the participants and multiple pain sites were stated by 59%. Pain relief after massage was reported by 52%. The mean pain intensity during the last week was 53.2 (range 0–100), and the best pain intensity was 27.9; mean accepted pain intensity after successful treatment was 11.6.

Tables A-E in S1 Data provide descriptive measures such as frequencies of completed questionnaires for each geographic region, total scores in outcome groups, tests on normal distribution and sociodemographic data with average scores of all individual items.

Follow-Up—Means

Preliminary analyses of variance revealed a strong relationship between scores at baseline and the outcome attributes of pain, function and sick leave after six months (ÖMPSQ Pain, p < 0.001; HKF-R 10 Pain, p < 0.002; ÖMPSQ Function, p < 0.001; ÖMPSQ Sick leave, p < 0.001). Full ANOVA related calculations are outlined in Tables F-H in S1 Data.

Follow-Up—Discriminative Abilities

As indicated in Table 5, ROC-curve AUC-levels for both questionnaires were statistically significant for each of the outcome groups. Results underscore the assumption that the discriminatory ability for each screening instrument is better than chance, with ÖMPSQ qualities ranging from moderate to good and a quality for the HKF-R 10 that was close to moderate. Generated ROC plots for each test variable can be found in Figures A-D in S2 Data.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 5. ROC-Analyses—Areas Under the Curve.

https://doi.org/10.1371/journal.pone.0158850.t005

Follow-Up—Predictive Validity Criteria

Table 6 shows all the predictive parameters for each outcome group. Underlying tables of ROC-curve coordinate points with extensive additional information are shown in Tables A-D in S2 Data.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 6. ROC-Analyses—Cut-off associated parameters of validity for each screening-tool taken from the ROC-curve coordinate point.

https://doi.org/10.1371/journal.pone.0158850.t006

With a total score related cut-off of 84 in prediction of pain, moderate and balanced values were observed for the ÖMPSQ (sensitivity 72%, specificity 75%). Prediction of function showed very high sensitivity (97%) and low specificity (57%) with a total score cut-off of 71. In contrast and with inversed proportions, a total score cut-off of 92 predicted sick leave with low sensitivity (63%) and high specificity (85%). The same applied to prediction of pain for the HKF-R 10 (sensitivity 50%, specificity 84%) with a total score cut-off of 46. Predictive ACC levels for outcomes ranged between low for the HKF-R 10 in pain, moderate for the ÖMPSQ in pain and sick leave and close to high in function. As seen, our results revealed clearly lower cut-offs for the ÖMPSQ in outcomes of pain and function than by Linton and Boersma [19]. For the HKF-R 10 a cut-off clearly higher than the recommended one of 37 was observed [20].

As quality of LR’s rely on levels of sensitivity and specificity as well as on balanced proportions between both; the resulting test efficiencies could be classified as moderate at maximum for all outcomes of the questionnaires (see Tables 1 and 2). LR’s can be used to calculate the prevalence-dependent post-test probabilities of a disease (PPV and NPV). Post-test probabilities affect the clinical utility of a screening-tool by determining if further testing for a single patient due to test results is required or not. The samples’ prevalence appeared to be high and did not reflect the estimated prevalence rate of 10%–15% in the German population. Although PPV’s were all moderate for the ÖMPSQ (71–75%) and good for the HKF-R 10 (88%) within the sample, Table 6 highlights a subsequent fall-down below 33% if these were compared with a prevalence of 10% in the German population.

Table 6 also provides some total score cut-off examples (taken from tables of the ROC-curve coordinate points) to compare changes in quality measurements. If intention is to decrease false-negative results, a lowered cut-off may be chosen to increase sensitivity at cost of some specificity (rising amount of false-positive result). As seen, a cut-off of 20 for the HKF-R 10 and a joining one of 76 for all of the ÖMPSQ outcomes raised the sensitivity levels to at least 80%. Specificity became low in all these cases. Except for the ÖMPSQ’s prediction of function this was also true for the PLR’s and PPV’s. ACC levels were also lowered except for the HKF-R 10 with a value close to moderate. With an intention to find balanced values between sensitivity and specificity, a lowered cut-off of 37 for the HKF-R 10 and a joining one of 84 for the ÖMPSQ (based on prediction of pain) might be chosen. PLR’s, PPV’s and ACC levels changed in a similar way as in the previous cut-off example.

Sensitivity and Specificity

As done by Linton and Boersma, we also provided cut-off examples to fit for high sensitivity or for balanced proportions between sensitivity and specificity. Additionally, and more important, optimised statistical cut-offs determined by the Youden-Index for each outcome group were provided.

Regarding these cut-offs and the intent of both questionnaires to reach high hit rates (sensitivity), agreement to Linton and Boersma exists in a moderate discriminative ability to identify patients at risk or no risk of persistent pain symptoms. Our results did not confirm their findings that tried to underline a high discriminatory power for sick listing.

Although sensitivity for outcome of function was very high, specificity was low and slightly above chance. This combination tends to overestimate patients at risk of CNSBP due to a rising number of false-positive results. An inversed proportion however, tends to underestimate the number of patients at risk by a rising amount of false-negatives. This is seen in prediction of sick leave for the ÖMPSQ and of pain for the HKF-R 10. In this regard, power of discrimination seems to be ambiguous and should be considered with care. To get around this, the provided cut-off example of 84 for all outcome groups should be favoured. This cut-off increases utilisation by unifying three different cut-offs, balances proportions and provides at least an overall moderate discriminative ability for the ÖMPSQ. Similarly, a cut-off of 37 for the HKF-R 10 that equals the original study should be recommended.

Likelihood-Ratios, Test-Efficiencies and Predictive Values

Predictive validity of a screening-tool should include quality measurements of test efficiencies and clinical utility. Test-efficiencies regarding the intent of these questionnaires describe the power of LR’s to change the pre-test probability at risk of CNSBP to a sufficient PPV if the test result was positive. PPV’s will always be low if pre-test probabilities (such as the prevalence) are low—regardless of the power of LR’s. Compared with an established prevalence in the German population, inadequate representation of prevalence’s within our sample led to inadequate PPV’s. This is seen in our findings when trying to re-calculate the PV’s. In these cases, we encourage recommendations to use the prevalence independent LR’s instead. At least, these were moderate for all outcomes of both questionnaires.

Weaknesses

Although strict eligibility criteria were chosen to minimise the amount of individuals’ that were in (recurrent) chronic state of NSBP at baseline, a potential selection bias could not be ruled out. This was indicated by reported pain duration for the ÖMPSQ of more than 24 weeks by 15% of all participants (43% in the original study). In addition, selection of participating patients by physicians and physiotherapists was maybe biased by their subjective view that patients were willing to cooperate and/or had an already existing higher risk of CNSBP. This may explain high prevalences within the sub-samples. Furthermore, by conducting ROC-Curve analyses that require dichotomised values, we lost subgrouping information that were originally available by graded classifications of both questionnaires.

Applying the methodological approach of Linton and Boersma in 2003, our study suffers from the same limitations. Some of the dependent variables have not been assessed by independent variables. This was true for the outcomes of function and pain. As already mentioned by Linton and Boersma, assessment by items that were part of the questionnaire itself (and were included in scoring) may inflate statistical results.

As seen, heterogeneous cut-offs within our sample for ÖMPSQ outcomes impede evaluation for examiners. These are traced back to different sub-sample sizes caused by negative raw data validation at baseline and/or missing replies to the key items at follow-up. The three work-related items were left empty several times. Uncertainties due to an inadequate reflection of the actual situation should be considered here. Furthermore, allowing for missing values (ticking checkboxes) led to an increase of potential dropouts [25]. The mean substitution with a quota of 14% in our population raises a statistical issue to mention. Aggravated by dependencies between the respective items, the advantage of easily applied mean substitution to complete the data sets and raising the sample size may have led to systematic statistical bias [36,37,38]. This is seen by ranges between 2 to almost 3 integers inside the bounds of several CI’s. However, and due to the design of the ÖMPSQ alternate modern statistical methods to handle missing values will be a difficult task in a triage setting. The HKF-R 10 was not affected by these kinds of bias due to prohibited substitution and usage of an outcome criterion that was excluded from scoring.