Development of the PRE-HIT instrument: patient readiness to engage in health information technology

To create candidate items, we first sought to understand to the relevant domains. To do this, we conducted four focus groups with 16 patients with the chronic conditions of diabetes, hypertension, heart failure, or coronary artery disease. Separate focus groups with 2–6 participants each were run for self-identified internet users and non-users as it was hypothesized that their issues with technology use would be different. Focus group participants were asked where they got information about their health, internet use, concerns about using the internet for health, information and communication preferences, and past experiences. Focus groups were audio recorded and transcribed by an experienced qualitative transcriptionist. Transcripts were analyzed using grounded theory methodology with investigator consensus on codes and themes, assisted by NVivo 8.0 software. Investigators were RJK a family physician clinical researcher, SMC an MPH experienced in qualitative methods, and JAS a medical student trained by the team in focus group and qualitative methods. All three participated in focus group facilitation and field note recording. RJK and SMC conducted analysis of the focus group data. Both independently coded the transcripts, and then met to agree on codes. Major themes emerged, which then informed the item-writing to capture that domain. Some items were near direct quotes from participants.

Once relevant domains were identified from the focus group themes, we searched the literature for instruments addressing these domains. We identified instruments that addressed internet use [11], internet use for health purposes [12, 13], computer and internet anxiety [14], computer and internet abilities [15‐18], attitudes and beliefs about the internet [14, 19‐21], risk perception [22‐28], internet security and privacy [6], health literacy [7, 29‐32], motivation [33, 34], and media literacy [7]. While items were not culled from these instruments, reviewing them allowed us to examine different approaches to relevant domains and aided in our overall task of item writing. Item writing was also informed by the focus group themes, including some items that were near direct quotes of participants. Items were written only in the English language.

Of 98 candidate items, we selected the 53 best items, eliminating those that were double-barreled, lengthy, or had large words or complex sentence structure, while maintaining coverage of our identified domains. We avoided jargon, value-laden words, negatively worded questions and negative prefixes, all of which can decrease an item’s validity coefficient [35, 36]. We also ensured that every question could be meaningfully answered by both internet users and non-users. Questions are also not specific for any disease and more generally reference “health”. These 53 items made up our candidate item questionnaire. We used a 4-point Likert scale for all items with anchors Strongly Disagree, Disagree, Agree, and Strongly Agree [37]. Items which conceptually might be negatively associated with health information technology use were mixed in with positively associated items to decrease rote responding; these were then scored in reverse order [38].

To ensure that respondents interpreted questions as intended, we administered the 53-item questionnaire to 21 participants in person. We used cognitive interviewing during questionnaire completion, asking them to verbalize their interpretation of the questions as well as the thoughts that led to their answers [35]. The questionnaire was iteratively refined until participants no longer expressed any ambiguity about the meaning of items. During this process a 4-point Likert scale with questions framed as hypothetical scenarios (e.g. If I went on the internet, I would find it frustrating.) was preferable to a 5-point Likert that included a neutral response option. The cognitive interviewing process revealed that participants used the neutral option as a catch-all. The 53-item questionnaire was then administered to 200 additional participants. Information about age, general health status (self-rated), highest level of education, and race/ethnicity was also collected from each participant. Five questionnaires that had more than five skipped questions were omitted for a final development set sample size of 195.

All participants for each phase of the instrument development were patients age 18 years and older with the chronic conditions of diabetes, hypertension, heart disease, or heart failure. Patients were all ambulatory patients attending one of 6 family medicine clinics of the Department of Family and Community Medicine at the University of Missouri. Patients were recruited from the waiting room of the clinic, and were asked to answer a screening questionnaire that ascertained if they had a chronic condition. The researcher made efforts to approach every person in the waiting room and made no assumptions about eligibility or experience using internet/computers. We limited participants to those who primarily speak English. The same recruitment method was used for both the focus groups and questionnaire completion. The University of Missouri Health Sciences Institutional Review Board approved all phases of this study.

Given the multiplicity of analyses employed in the development of a new instrument it is difficult to derive a priori sample size estimates. We approached the sample size issue by estimating the sample required for an exploratory factor analysis (EFA). As a starting point, we posited that the propensity to use health information technology is composed of the five factors Capability, Access, Motivation/Risk Perception, Information Needs, and Privacy/Trust. We further conjectured an initial screening step would reduce the number of candidate items to between 40 and 50. We also made the worst-case assumption that items comprising these five factors would exhibit weak communalities (percent of variance in the observed items explained by the factor model). Under these assumptions, guidelines indicate that a minimum sample size of 130 subjects would be sufficient for the factor analysis [39]. To allow for possibility of retaining fewer items or deriving more factors the sample size was inflated to 200 subjects.

The first quantitative analysis was to examine item response frequencies for the 53 items in the development sample. Items with insufficient variation across response options were considered for deletion. Such items do not discriminate between different levels of the trait they are intended to measure. After item culling we conducted an EFA to determine the factor structure of the instrument. One can use factor analysis in either an exploratory or confirmatory mode, but since this is a new tool we began with exploratory techniques. With EFA we do not specify the number of factors or the items that load on those factors in advance but rather let the data guide us [40].

An essential step in an EFA is to determine the number of factors to retain. Determining the number of factors is both a matter of judgment on the content and quality of the factors, and a statistical issue. By convention one retains factors with an eigen value of greater than 1.0. This criterion is motivated by the fact that EFA operates on the item correlation matrix wherein item variances are fixed at 1.0, and so factors with a variance (eigen value) less than that of a single item are essentially comprised of noise. The initial factoring extracted 13 factors, however several of the factors were comprised of few items, items that cross-loaded with other factors, or that were difficult to interpret. Final solutions were derived after a Promax oblique rotation. Items with factor loadings of less than 0.30 or with substantial loadings on more than one factor were excluded from the instrument. From the initial pattern matrix of 13 factors, we identified 8 strong factors. We selected groups of items in each factor that loaded most heavily on that factor, and with minimal or no dual loading on other factors. The investigators examined the items in each factor and agreed on a name for the overall factor concept. Each item was assessed and was kept if it had a high loading, was a good conceptual representation of the named dimension, and was well worded and clearly measured the variable of interest. The EFA was re-run with the candidate items to achieve the final instrument. We examined the internal consistency of each factor using Cronbach’s alpha [35, 41].

To investigate the possibility that our 8 identified factors clustered into one or more higher order factors, we examined a scree plot of eigen values of the augmented correlation matrix against the number of factors. Factors with eigen values > 1 are candidates for higher order factors. Potential higher order factor solutions were examined using exploratory methods to examine whether the eight factors represent different constructs or whether they reflect different facets of multiple higher order concepts. The preferred way to do this is using confirmatory factor analysis (CFA) techniques, however this analysis had convergence issues, possibly reflecting that the sample size may not be sufficient to support a CFA model with 28 observed variables in 8 factors. Therefore the less demanding EFA method was again used, this time where the raw data is the summated factor scores.

Three months after the initial sample, we mailed retest questionnaires (full 53 item questionnaire) to a randomly selected sample of 53 participants. 2 declined. Of the 51 remaining, 45 returned completed questionnaires. The eight factors were compared in the initial and retest using Pearson’s correlations.

If I went on the internet, I would use it to look up things so that I wouldn’t worry about them anymore.

If I went on the internet, I would use it to look up information about herbals and/or supplements.

If I went on the internet I would use it to look up symptoms.

If I went on the internet I would use it to search for information about my health.

If I went on the internet I would use the internet to find information about medications.

If I went on the computer, I would be able to figure out most computer problems that I might run into.

If I went on the computer, I would have access to the internet.

If I went on the internet, I would find using the internet to be easy.

If I went on the internet, I would find using email to be easy.

If I went on the computer, I would find using it to be frustrating.

If I went on the internet, I would get frustrated with the amount of information I found about health on the internet.

If I went on the internet, I would find searching for information on the internet would be stressful.

If I went on the internet, I would find sorting through information on the internet to be too time consuming.

Looking up health concerns on the internet is more convenient for me than contacting a doctor’s office.

I prefer calling my doctor’s office to emailing them.

I email my doctor.

I trust the internet as a source for health information.

Looking up information online about medications is easier than asking my doctor.

I let my doctor handle the details of my health.

Doctors are my most trusted source of health information.

When I have a health concern, my first step is to contact my doctor’s office.

I go online using my cell phone.

I use my cell phone to text people almost every day.

If I went on the internet, I would be very concerned about giving any personal information.

If I went on the internet, I would be concerned it would lead to invasions of my privacy.

People today want to know too much about their health.

Regarding my health, I agree with the statement “No news is good news.”

I am concerned about what I might find if I look up health issues on the internet.

Examination of a scree plot of eigen values of the augmented correlation matrix against the number of factors revealed the possibility of 2 or 3 higher order factors (Figure 1). Two and three higher order factor solutions were examined using exploratory methods to examine whether the eight factors represent different constructs or whether they reflect different facets of multiple higher order concepts. The three factor solution was problematic with multiple loadings for factors, while the 2 factor solution, Table 4, was both conceptually and analytically unambiguous. Two “meta-factors” were extracted which conceptually represent “Facilitators” and “Barriers” to health information technology use.