Development and Psychometric Evaluation of Coronary Artery Disease Treatment Adherence Scale

The first phase in the scale development is to generate and obtain items relevant to the question and the targeted population [19]. There are three methods for the initial item generation that included deductive, inductive, and a combination of the two. Deductive methods involve item generation based on an extensive literature review and pre-existing scales [20]. The most widely source of qualitative data is the literature review that produces tests with effectively unlimited item banks [19, 21]. An additional function of the literature review is to gain a more concrete understanding of the construct by identifying conceptual definitions of each dimension and relevant empirical items that describe each dimension [22].

CADTAS was developed deductively based on the existing literature. Accordingly, in order to determine the dimensions of treatment adherence and generate CADTAS items, we searched online databases such as ScienceDirect, Embase, MEDLINE, CINAHL, IranMedex, Magiran, and SID in order to retrieve adherence-related studies. Search terms were treatment adherence, CAD, ischemic heart disease, coronary angioplasty, and coronary artery bypass surgery. Studies were included in the analysis if they were directly related to treatment adherence, were literature review or original study, and had been published in Persian or English after 2000. This search protocol yielded 706 documents. Irrelevant studies were excluded and 352 studies and four questionnaires were kept. Then, the full texts of all retrieved documents were read and 75 documents directly related to CAD treatment adherence were selected. These 75 documents were reviewed and summarized to define CAD treatment adherence, determine its dimensions, and then, generate CADTAS items. We also considered societies’ recommendations for assessment of adherence.

After summarizing the documents, the concept of adherence to the treatment was explained and its dimensions were determined. We wrote the maximum possible items in each dimension.

Therefore, an item pool with 65 items was generated. The items were repeatedly reviewed, revised, and merged and finally, the primary CADTAS was generated with 62 items. Items were scored via a five-point Likert-type scale as follows: 1: “Never” (no treatment adherence); 2: “Rarely” (limited treatment adherence); 3: “Sometimes” (moderate treatment adherence); 4: “Often” (good treatment adherence); and 5: “Always” (strict treatment adherence). Higher scores represented closer treatment adherence. Some items were negatively worded in order to reduce the negative effects of haphazard responses to items.

In this phase, the validity and reliability of CADTAS were assessed. Validity assessment included the assessments of face, content, and constructs validity.

Qualitative and quantitative methods were used for face validity assessment. In qualitative assessment of face validity, ten low-literacy CAD patients were randomly selected and interviewed face-to-face about the difficulty, appropriateness, and clarity of CADTAS items. Then, the items were revised to make them simpler and clearer.

Quantitative assessment of face validity was done via calculating impact score for each item. Ten randomly selected CAD patients rated the importance of each CADTAS item using a five-point rating scale as follows: 1: Not important; 2: Slightly important: 3: Somehow important; 4: Important: and 5: Very important. Impact score for each item was calculated using the following formula, “Impact score = Frequency (%) × Importance”. “Frequency” in the formula was the number of patients rated the item 4 or 5, while “Importance” was the mean score of the item on the 1–5 rating scale. Items were kept in CADTAS if they obtained an impact score of 1.5 or more [23, 24].

Assessment of content validity was also performed via both qualitative and quantitative methods. In the qualitative method, ten experts in instrument development, cardiology, cardiac surgery, cardiac nursing, and treatment adherence evaluated and commented on the wording, essentiality, importance, appropriate placement, and scoring of the items. Then, items were revised using their comments. Quantitative assessment of content validity was performed by calculating content validity ratio (CVR) and content validity index (CVI). CVR was calculated to ensure that the most important and the most correct items had been selected for treatment adherence assessment. Accordingly, 14 experts in cardiology, cardiac surgery, and nursing evaluated the essentiality of the items via a three-point scale with the following three points: “Essential”, “Useful but not essential”, and Not essential”. Then, the CVR of each item was calculated via the CVR formula, i.e., CVR = (Ne – (N/2))/(N/2), where N stands for the total number of experts and Ne stands for the total number of experts rated the intended item as essential. Lawshe noted that for 14 experts, items with CVR values of 0.51 and more are appropriate [25].

CVI was calculated to determine whether the items appropriately measure treatment adherence. The same 14 experts rated the simplicity, relevancy, or specificity, and clarity of each item on a four-point rating scale. Then, the number of experts rated an item 3 or 4 was divided by the total number of experts in order to calculate the CVI of that item. Items with a CVI of more than 0.79 were kept, while items with a CVI of 0.70–0.79 were revised and items with a CVI of less than 0.70 were removed. Finally, the average CVR and CVI were calculated for the whole CADTAS.

After the assessments of face and content validity and before the assessment of construct validity, the reliability of CADTAS was assessed through the internal consistency method. Accordingly, Cronbach’s alpha of the scale was calculated using the data collected from 30 CAD patients. The minimum acceptable Cronbach’s alpha was considered to be 0.70 [26].

Construct validity can be assessed with the use of exploratory factor analysis (EFA), confirmatory factor analysis (CFA), or with convergent, discriminant, predictive/ nomological, criterion, internal, and external validity. The group of extraction and rotation techniques to identify latent constructs is referred to as common factor analysis or exploratory factor analysis [20, 22].

As CADTAS was a new tool and we had no assumption about its dimensions, assessment of its construct validity was performed using exploratory factor analysis. The number of necessary individuals for factor analysis is 5–10 individuals per item [27]. Therefore, we recruited six CAD patients per CADTAS item. Inclusion criteria were a definite diagnosis of CAD via coronary angiography, an age of 18 or higher, and receiving CAD treatments for at least 2 months. In order to select a sample of maximum variation, patients were recruited from three teaching hospitals and four clinics in Zanjan, Iran. Sampling was done conveniently. Recruited patients individually completed CADTAS. The data were entered into the SPSS software (v. 24) for factor analysis. Initially, the Kaiser-Meyer-Olkin (KMO) test was run to ensure the adequacy of the selected sample. KMO test value can range from 0 to 1—the higher the test value, the more accurate the factor analysis. The minimum acceptable value is 0.70. Besides, Bartlett’s test of sphericity was run to determine factor analysis suitability by testing whether the correlations among CADTAS items were strong enough so that the items could be clustered and also by testing whether the correlation matrix was significantly different from zero value.

Factor extraction in factor analysis was done via the principal component analysis method. The results of this method are presented as a matrix of each factor loading on each item. In fact, factor loading represents the correlation of an item with a factor and is used to maintain an item in a factor or remove an item from a factor after factor rotation. There are different methods for determining the number of factors. In this study, the number of factors was determined based on the variance of each item, eigenvalue, and scree plot. Then, varimax rotation was used to simplify the factors and make them interpretable. Extracted factors were labeled based on the items loaded on them. The minimum factor loading for keeping an item in the allocated factor was 0.3.

The internal consistency and the test–retest stability methods were used for the assessment of reliability. Stability, also called test–retest reliability, is the degree to which scores on the same test are consistent over time. In other words, this type of reliability provides evidence that scores obtained on a test at one time (test) are the same or close to the same when the test is re-administered some other time (retest). The more similar the scores on the test over time, the more stable the test scores. A common, minimally accepted sample size for a correlational study is 30 participants. If the variables correlated have low reliabilities and validities, a bigger sample is necessary. The procedure for determining test–retest reliability is quite simple: after administrating the test to an appropriate group and passing some time- about 2 weeks or more- administer it again and calculate the correlation between two series of data [28, 29].

Accordingly, 30 randomly selected patients completed CADTAS. Collected data were used to calculate Cronbach’s alpha. Minimum acceptable alpha value was 0.7 [30]. Two weeks later, the same 30 patients recompleted CADTAS. Then, intraclass correlation coefficient was calculated to assess stability.

All procedures performed in studies involving human participants were in accordance with the Ethics Committee of the Tehran Faculty of Nursing and Midwifery with No: IR.TUMS.FNM.REC.1396.2689 and with the 1964 Helsinki Declaration and its later amendments or comparable ethical standards. Informed consent was obtained from all individual participants included in the study.

After identifying the four dimensions of treatment adherence based on the existing literature, the primary item pool was generated with 65 items. Items were repeatedly revised and reduced to 62.

Qualitative assessment of face validity was done by four female and six male patients with primary and secondary educations. In quantitative assessment of face validity, four items were removed due to an item impact score of less than 1.5.

In qualitative assessment of content validity, changes were made to the items and three items were removed based on the comments of the ten experts. Moreover, in the quantitative assessment of content validity, two items with a CVR of less than 0.51 were deleted. All remaining 53 items had a CVI of more than 0.79. The average CVR and CVI of the whole scale were 0.886 and 0.977. Table 1 shows the number of items deleted during the different steps of psychometric evaluation.

After confirming face and content validity, primary reliability assessment was performed via the internal consistency method. Cronbach’s alpha of the 53-item CADTAS was equal to 0.85.

Construct validity of CADTAS was assessed via factor analysis. We recruited 318 CAD patients (six CAD patients per CADTAS item). Table 2 shows demographics characteristics of CAD patients recruited in evaluating construct validity and primary reliability assessment. KMO test value was 0.809, indicating that the recruited sample of patients was adequate enough for factor analysis. Moreover, the significant results of Bartlett’s test of sphericity showed strong correlations among items (χ² = 4527.762; df = 528; P < 0.001), implying the suitability of factor analysis.

Factor number was determined via scree plot, eigenvalue, and factor rotation. Visual assessment of scree plot showed that 3–5 factors can be considered for CADTAS (Fig. 1). Based on eigenvalues, 17 factors with eigenvalues greater than 1 were identified, which collectively explained 63.5% of the total variance. The 17-factor structure was tested using different factor rotations. Finally, based on scree plot, the characteristics of the items, and factor rotation, we decided on a four-factor structure, which accounted for 45% of the total variance of treatment adherence. This four-factor structure had the highest level of intra-factor homogeneity and consistency. The minimum acceptable factor loading was 0.3. In the four-factor structure, 18 items had a factor loading of less than 0.3. These 18 items were deleted and therefore, the number of items reduced from 53 to 35. The four extracted factors were named based on their items and were compared with the four dimensions of treatment adherence which had been identified in the literature review. The four extracted factors included medication adherence (11 items), dietary adherence (13 items), exercise adherence (four items), and healthy lifestyle adherence (seven items). The five possible answers to CADTAS items were “Always” (scored 5), “Often” (scored 4), “Sometimes” (scored 3), “Rarely” (scored 2), and “Never” (scored 1).

The negative effects of haphazard responses are reduced through the negative wording of some items. The total CADTAS score is 35–175, with higher scores standing for closer treatment adherence among CAD patients.

Table 3 shows characteristics of CAD patients recruited for evaluating final reliability of CADTAS. Final reliability assessment indicated that the Cronbach’s alpha values and the intraclass correlation coefficients of CADTAS and all its four factors were greater than 0.70 and 0.80, respectively (Table 4).

The study was conducted in Iran. This could be a limitation to generalize the results to other regions where disease population/education and culture are different.

The test re-test reliability was done on 30 patients with 2-week interval. Short period between tests can have a memory effect. Also, reliability could have been done on a large sample.