Development and validation of the quality care questionnaire –palliative care (QCQ-PC): patient-reported assessment of quality of palliative care

QCQ-PC includes four stages of development proposed by the European Organization for Research and Treatment of Cancer Quality of-Life Group [24]: 1) item generation and reduction; 2) scale construction; 3) pilot testing; and 4) field testing. The study was approved by the Institutional Review Board of the Seoul National University Hospital, Korea. Suitable patients signed informed consent forms. The requirements for the recruitment of patients were as follows: (1) 18 years old or older; (2) diagnosed with cancer (cancer was confirmed by oncologists); (3) able to read and comprehend Korean, be able to fill in the questionnaire; and (4) aware of cancer diagnosis.

Phase I was intended to gather a list of relevant quality of palliative care questionnaires and previous palliative care clinical guidelines. We performed an extensive literature review using PUBMED and other databases searching the keywords ‘palliative care’, ‘end of life care’, and ‘quality of care’. We conclusively reviewed and adopted End of Life Quality Standards (2013) published by The National Institute for Health and Care Excellence (NICE) in the United Kingdom, Clinical Practice Guidelines for Quality Palliative Care (3rd Ed. 2013) released by National Consensus Project (NCP) in the United States and A National Framework and Preferred Practices for Palliative and Hospice Care Quality issued by National Quality Forum (NQF) in the United States.

We also considered the quality of care index of the prior study. We investigated and used the Assessment of Chronic Illness Care (ACIC) questionnaire and the Patient Assessment of Chronic Illness Care (PACIC) questionnaire translated by National Evidence-based Healthcare Collaborating Agency (NECA) in a previous study, which was a study of the performance evaluation methods of sub-categories in the management of chronic disease patients in a community by the primary health care center (2015). In addition, the FAMCARE Patient Scale developed by Kristjanson (1993) and verified with respect to validity and reliability was used. Based on the literature, we constructed a list of items for the QCQ-PC from 104 quality care issues divided into 14 scales.

We discussed these 104 items in semi-structured interviews with 31 health care professionals in November 2016 and December 2016. The health care professionals participating in the survey included physicians, nurses, and social workers who worked in university hospitals and were engaged in palliative care. The semi-structured interviews did not include patients because the items built by healthcare professionals were intended to be pilot-tested separately. The survey was conducted twice in total, and the respondents evaluated the validity and feasibility of the developed items by scoring them on a scale from 1 to 5. In the second investigation, the average score of all respondents for each item was compared with the score given by them, and the score changed when the respondent’s evaluation changed. The items that did not meet the following criteria in both the primary and secondary surveys were deleted (in the case of validity, if the respondent average score was three or more, or the respondent score was less than three points in less than 25% of the respondents; in the case of feasibility, the respondent average score is 2.5 or more, or less than three points in less than 30% of the total respondents). Thirteen items (12.5% of the developed items) that did not meet the criteria were deleted; therefore, a total of 91 items were included in the final index.

Among the 91 palliative care quality evaluation indices developed through the process of item development and reduction, we constructed scales of 43 items that only the patients were asked to answer. The 48 items that were excluded were the items that directly asked the medical staff to evaluate the quality of care. For the scoring format, we selected a four-point Likert scale for all of 43 items to evaluate in phase III (Strongly Agree, Agree, Disagree, Strongly Disagree – see Additional file 1: Appendix B).

We conducted pilot tests to find possible administration problems, such as miss-phrasing, and to determine which items should be modified or removed. Before field testing, a total of 15 cancer patients responded to the QCQ-PC questionnaire and debriefed the questionnaire. The mean age of the 15 patients was 60.6 years, with 7 males and 8 females. Regarding educational level, most subjects had more than college or university education. All 15 patients were collected from Seoul National University Hospital. All 15 patients were collected from Seoul National University Hospital. The questionnaire designed for debriefing included which questions were misleading, whether the respondents were offended, or whether they were unable to respond. After the questionnaire, we received comments on the questionnaire. Taking into consideration the responses of these debriefing questionnaires, we clarified and simplified the items that were difficult to understand or answer. We also added one item to reflect the needs of the respondents.

To confirm reliability and validity, we performed a field test with the QCQ-PC questionnaire.

Because the most patients group receiving palliative care are patients diagnosed with cancer, we performed this field testing for the QCQ-PC questionnaire to cancer patients. To obtain a heterogeneous sample by cancer type, patients were recruited to include various primary sites of cancer; these patients were registered at four university hospitals in Korea. We included patients under curative treatment as well as patients under palliative treatment. This study used factor analysis with varimax rotation to analyze construct validity, and performed multi-trait scaling analysis to examine the extent to which QCQ-PC items could be combined into a more controlled multi-item set. For validation, we applied ten rules, for which at least five respondents were required per item [25, 26]. Since we had a total of 44 questions, 220 respondents were necessary for the validation process.

We investigated the correlation between each question and the scale that included that question, and estimated the convergent validity of the QCQ-PC items. In verifying discriminant validity, we assessed the magnitude of the correlation of an item with its own scale as compared to other scales. We considered the convergent validity of items by analyzing correlations that were ≥0.4, and corrected for overlap, as confirmation of validity [27]. We identified scaling errors as cases in which an item correlated significantly less with its own scale as compared to its correlation with other scales. To test the reliability of QCQ-PC, we estimated Cronbach alpha, a degree of internal consistency of responses. An alpha ≥0.70 was generally regarded as acceptably high for the collection of responses into a single score [28].

Furthermore, fit statistics (INFIT/OUTFIT) analysis was performed to evaluate the fitness of the item response, and it was interpreted as a good response when the fit was between 0.7 and 1.3 [29]. The slope was checked to confirm the discrimination of the items. In the case of a fit value less than 1.5, the discrimination of the items was considered unsuitable (see Additional file 1: Appendix A).

We asked the respondents to fill in additional questionnaires on previous validated scales. The following questionnaires were included: The European Organization for Research and Treatment of Cancer Quality of Life Questionnaire Core-15 item palliative care (EORTC QLQ–C15-PAL) measuring patient-reported QoL [30], the McGill Quality of Life Questionnaire (MQOL) constructed by physical symptoms, psychological symptoms, existential well-being, and support [31], the Smart Management Strategy for Health Assessment Tool – Short Form (SAT-SF) for self-management strategies assessments [32], and the Decisional Conflict Scale (DCS) - assesses the level of ‘decisional conflict’ that patients experience while making health care decisions [33].

Data analyses were conducted using WINSTEP version 4.0 for item-fit analysis, and SPSS version 24.0 for other statistical analyses.

We conducted factor analysis on the total sample of respondents (n = 220). We initially had four significant factors with 44 items; twelve items were discarded through the fit statistics process. Upon reanalysis of the remaining 32 items, the items were classified according to four significant factors. Table 2 lists the item-to-factor correlations for the 32 items and four factors – factor 1: appropriate communication with health care professionals (ten items); factor 2: discussing value of life and goals of care (nine items); factor 3: support and counseling for needs of holistic care (seven items); and factor 4: accessibility and sustainability of care (six items). We attained similar of factor analysis results with both multiple and simple imputations.

Table 3 shows the mean and reliability of the QCQ-PC subscale. In all four scales, the degree of reliability was high, with good internal consistency (Cronbach alpha range: 0.889–0.973).

When item-to-self scale correlations were tested, all item-convergent validity exceeded 0.4. Other scales were also compared through item-discriminant validity, and no scaling error was observed.

This study has several limitations. First, since our research was conducted only in Korea, cross-cultural validation studies are necessary for generalizations to other countries. Second, we did not assess test-retest. As QOL and quality of care among cancer patients can frequently change, test-retest may not be feasible and may not limit the psychometric properties of this QCQ-PC. Finally, as this study relied exclusively on patients with cancer, our findings might not be generalizable to non-cancer patients who need palliative care. Furthermore, our study included not only incurable cancer patients, but also patients who had received treatment more than 5 years previously. However, to assess the quality of early palliative care as well, it may be preferable to include all cancer patients rather than only those with incurable cancer for constructing the questionnaire. We suggested that it is necessary to investigate further research on the patients of early palliative care and palliative care with this QCQ-PC questionnaire. Though, some modification of items among the QCQ-PC might be applied to patients with other advanced diseases.