Methods
We conducted a Delphi study among Canadian representatives of actual or potential EHR users to confirm the findings of the systematic review and to prioritize the key barriers and facilitating factors for EHR implementation in Canada.
Participants
The study participants are Canadian representatives of actual or potential EHR users, or groups who use or are expected to use EHR. They are subdivided into four groups: physicians, patients, managers and non-physician health [
23‐
25]. In order to better reflect the reality of the Canadian healthcare system, we subdivided non-physician health professionals in two groups: non-physician healthcare professionals and health information professionals. Non-physician healthcare professionals are nurses, physician assistants and other non-physician providers. Health information professionals (HIP) are defined as people providing leadership in all aspects of clinical information management at both the micro and macro levels. At the micro (or individual record level), HIP professionals support data collection, use, access and disclosure, to the retention and destruction of health information regardless of format. HIPs perform qualitative analysis on the documentation within the health record and are responsible for the security of health records. HIPs are advocates of the individual’s right to private, secure and confidential health information. At the macro (or aggregate data level), HIPs deal with the information through the health system, analyze statistics, manage complex information systems including registries and work with public, private and key stakeholders in understanding and using health data to improve the health of Canadians [
26].
A purposive sampling method was used to recruit participants from all Canadian provinces and territories. More than 20 national and provincial e-health and healthcare professional associations, organizations and interest groups, such as the Canada Health Infoway Clinician Peer Support Network, were asked to collaborate through forwarding our study recruitment email to their members (see the full list in Additional file
1). Experts who were known actual or potential users of Canadian EHR systems were purposefully selected and also invited by telephone and email to join the study. Other potential key informants were identified through a snowball sampling technique. Interested participants were directed to the study website to register their contact information and detail their EHR experience.
Our selection criteria aimed to ensure an adequate breadth of expertise and representation of various geographical contexts. Eligible participants were required to have professional experience related to Canadian EHRs. Because there are few fully interoperable EHR projects in Canada, “EHR experience” was broadly defined as being knowledgeable about the electronic organization of patient information, including a full or partial computerized record of a person's medical history, or just certain records, such as laboratory or diagnostic testing results. Participants also needed to belong to one of the five EHR user groups (physicians, non-physician healthcare professionals, health information professionals, healthcare managers and patients), have a valid email address and access to the internet, and speak English or French. As required by the ethics committee, patient participants were also required to hold an official position of “patient representative” within a health association or organization.
Ethical consideration
The Delphi study participants were given specific consent forms presenting research objectives and information about research implications. They were informed that their participation in the study was entirely voluntary and that they implicitly consented to participate by completing the first round of the electronic Delphi study. Ethics approval for the study protocol was received from the Research Ethics Board of the Centre Hospitalier Universitaire de Québec (approved January 23, 2009; ethics number 5-08-12-06).
Design of the Delphi study
Based on factors identified for each users group in the systematic review [
21] we designed a specific questionnaire for each actual or potential user group. The questionnaire proposed a number of questions that corresponded to the most frequent factors found for each user group in the literature review [
21]. In order to limit the length of questionnaires we decided to select the factors that were mentioned by at least three studies, or exceptionally by two studies if one of them was conducted in Canada. Most studies included in the systematic review did not make a distinction between non-physician healthcare professionals and health information professionals. Consequently, it was then not possible to extract the factors specific to each non-physician health professional group. Healthcare professionals and health information professionals received the same questionnaire version (14 factors listed in Additional file
2), but each group was analysed separately because their answers were different. Then, four bilingual (English and French) online questionnaire versions, pertaining to each user group (physicians, non-physician health professionals, managers and patients) were developed (see complete questionnaires presented in Additional files
3,
2,
4, and
5). Two bilingual members of the research team validated the translation. The questionnaires were pre-tested by members of the research team to assess the clarity of the questions, the instructions and the format. Physicians and managers received questionnaire versions with respectively, 18 and 11 factors. Using a five-point Likert scale (where 1 indicated strong disagreement and 5, strong agreement), participants rated each questionnaire statement for its
applicability to the participant’s specific EHR implementation context and its
importance for decision-making regarding EHR implementation in Canada. Participants were also invited to leave written comments after each answer and general comments at the end of the questionnaire. The Delphi study was conducted during a 10 day period in March 2010, using a dedicated website developed for the purpose of this study.
For each Delphi study round, participants were emailed a link to the appropriate questionnaire on the project website, and were allotted 48 hours to complete the questionnaire. Email reminders to complete the questionnaire were sent after 24 hours to participants who had not yet replied. In the first round, participants were asked to simply rate their responses for each item. In the second and third rounds, distributions of participants’ answers to each item in the previous round were presented in percentage form. Participants were invited to take into consideration the other participants’ responses and reassess their answers in light of this new information. Questionnaire formats remained unchanged during the three study rounds and participants were not provided with reminders of their responses in the previous rounds. Individuals who did not complete a previous round were not invited to participate in following rounds.
Analysis
Third round data was used for final analysis. As there is no existing definite criteria determining consensus in a Delphi study [
27], we chose a priori consensus criteria based upon the research team’s previous work [
28]. Consensus on a questionnaire factor was considered “strong” when at least 75% of participants reached an agreement on both the applicability and importance. We chose to consider these two criteria because this would indicate a priority for decision making [
18]. “Moderate” consensus required 60% to 74% of participants to agree on both the applicability and the importance. “Partial” consensus was obtained when at least 60% of the participants reached consensus on only one aspect (applicability or importance) of a factor. Absence of consensus was determined when less than 60% of participants agreed on the applicability and the importance of a factor.
Percentile scores and interquartile range were used to calculate the level and strength of the consensus, respectively. To determine the level of consensus, tenth and twenty-fifth percentile scores were calculated. Tenth percentile scores indicate the lowest number on the Likert scale upon which at least 90% of participants agreed and 25th percentile scores indicate 75% agreement. Interquartile range, a measure of statistical dispersion, indicated the strength of the consensus, where 0 specifies a strong group consensus and 2 indicates dispersed responses. For example, in Table
1 physician consensus, the factor “confidence in EHR developer or vendor” (applicability) shows that according to 10
th percentile scores 90% of respondents responded either 2, 3,4 or 5 on the 5 point Likert scale, while 75% of respondents responded either 4 or 5. The interquartile range of 0 indicates strong group consensus.
Table 1
Physician consensus
Strong | Confidence in EHR developer or vendor | A | ≥ 75 % | 4 | 2 | 4 | 0 |
I | ≥ 75 % | 4 | 4 | 4 | 0 |
Moderate | Cost issues (start-up and maintenance) | A | ≥ 75 % | 4 | 3 | 4 | 0 |
I | 60-74 % | 4 | 3 | 4 | 0 |
Lack of time and workload (clinical tasks) | A | 60-74 % | 4 | 3 | 4 | 0 |
I | 60-74 % | 4 | 4 | 4 | 1 |
Partial | Cost issues (return on investment) | A | 60-74 % | 4 | 1 | 2 | 2 |
I | < 60 % | − | 1 | 3 | 1 |
Patient and health professional interaction | A | < 60 % | − | 3 | 3 | 1 |
I | 60-74 % | 4 | 3 | 3 | 1 |
Lack of time and workload (EHR use) | A | 60-74 % | 4 | 2 | 4 | 0 |
I | < 60 % | − | 4 | 4 | 1 |
Change in tasks | A | 60-74 % | 4 | 3 | 4 | 0 |
I | < 60 % | − | 3 | 4 | 1 |
Choice of the EHR system | A | 60-74 % | 4 | 2 | 3 | 1 |
I | < 60 % | − | 2 | 3 | 1 |
None | Design and technical concerns | A | < 60 % | − | 3 | 3 | 2 |
I | < 60 % | − | 3 | 3 | 2 |
Privacy and security concerns (patient privacy) | A | < 60 % | − | 1 | 2 | 2 |
I | < 60 % | − | 1 | 2 | 2 |
Privacy and security concerns (professional confidentiality) | A | < 60 % | − | 1 | 1 | 3 |
I | < 60 % | − | 1 | 3 | 2 |
Quality standards | A | < 60 % | − | 3 | 3 | 2 |
I | < 60 % | − | 3 | 3 | 2 |
Productivity (loss of) | A | < 60 % | − | 3 | 3 | 2 |
I | < 60 % | − | 3 | 4 | 1 |
Practice size (small) | A | < 60 % | − | 2 | 2 | 2 |
I | < 60 % | − | 2 | 3 | 2 |
Practice size (large) | A | < 60 % | − | 3 | 3 | 2 |
I | < 60 % | − | 3 | 3 | 2 |
Physician salary status and reimbursement | A | < 60 % | − | 2 | 2 | 2 |
I | < 60 % | − | 2 | 3 | 1 |
Human resources (IT support, other) | A | < 60 % | − | 1 | 3 | 2 |
I | < 60 % | − | 3 | 4 | 1 |
Management | A | < 60 % | − | 1 | 2 | 1 |
I | < 60 % | − | 2 | 2 | 1 |
Priority for decision-making concerning EHR implementation in Canada was determined based on the questionnaire items upon which 90% of participants considered both applicable and important (that is, factors scoring 4 or 5 for applicability and importance on the 5-point Likert scale). Tenth percentile scores were used to determine if the questionnaire item reached sufficient consensus. The analyses were performed using SAS software version 9.1.
Discussion
This study aimed to gather this first-hand knowledge held by actual and potential Canadian EHR users. Using the Delphi method, we compared the implementation experiences in the Canadian context with the known factors found in the scientific literature. Our findings show that, overall, many of the factors found in the scientific literature are also applicable to the Canadian context. However the varying levels of agreement between and within user groups could mean that each group has unique professional priorities in the EHR implementation process.
This study is addressing an issue of great relevance in the current Canadian health care context. Notably, key factors to EHR implementation were considered of the highest priority in a recently published report investigating gaps in knowledge, research, and research capacity regarding EHRs in the Canadian primary health care [
29].
However, the results of this study must be interpreted cautiously due to some limitations. In fact, the major flaw in this kind of study is that it relies on the perception of participant on what might potentially influence implementation success without actual evidence from real implementation. We would like to invite EHR implementation researchers to further explore this list of priorities in order to validate them in diverse implementation contexts. Also, despite exceptional recruitment efforts, only two representatives of patient groups volunteered to take part in the study. Consequently, the patient Delphi study was cancelled creating an important gap to our findings. As mentioned above, the small survey sample may reduce generalizability of the results. Also, the 50% participation rate of physicians caused limited prioritization from this group. As well, a bias introduced by a participation dominated by women could occur. Although profession is an important factor contributing to group consensus, other factors such as age, work setting, and specific type of EHR experience are influences that were not considered in this Delphi study. Consequently, the factors presented might have not been exhaustive of the respondents’ experiences and important factors may have been overlooked. Secondly, we do not know if the participants that dropped out of the study and the resulting missing data could represent a bias for the degree of consensus. Finally, results of this study are based upon the informed opinions of the participants, and results should be interpreted appropriately as the well-founding of such opinions cannot be verified.
The three factors common to all user groups reached different levels of consensus (see Table
6). Firstly, physicians and managers disagreed while health professionals and health information professionals partially agreed on the applicability or the importance of “Privacy and security concerns”. Our research question was centered on “which” factors were a priority. The limited consensus reached on certain aspects of the EHR implementation raises the question of “why” this is. Range in practice size or differences in professional settings could be an explanation in some instances, however only a dedicated study can answer this question. Secondly, perceptions toward “Lack of time and workload” appear to be user specific as indicated by strong, moderate, partial or no consensus for health professionals, physicians, health information professionals, and managers, respectively. Thirdly, “Human resources” presents a different consensus pattern in which health professional reached a strong consensus whereas health information professionals reached a moderate consensus. Additionally, physicians disagreed while managers partially agreed on the applicability or the importance of “Human resources”. Consensual factors of applicability and importance identified in our study for each user group should guide future research. We compared the Delphi study results with the factors specifically mentioned by the Canadian studies included in the systematic review [
30‐
35]. Firstly, among the 17 studies included in our systematic review pertaining to physicians, the factor “confidence in the EHR developer” was mentioned only twice (including one Canadian study [
33]). However, Canadian physicians reached a strong consensus about this factor in our Delphi study, indicating that this is an issue of particular relevance that should be explored in further research. Furthermore, while one Canadian study identified fee-for-service payment of Canadian physicians as a barrier to EHR implementation [
33], our physician participants did not reach consensus on this subject. Updated research by the same authors confirms our findings, suggesting that this remuneration approach does not hinder EHR implementation more than other forms of payment [
36]. Our findings concerning healthcare professionals are congruent with those of a recently published Canadian case study [
37]. This study identified facilitating EHR implementation factors such as perceived usefulness, motivation, effective onsite technical support and management, and barriers such as concerns about privacy and security. Participants in our Delphi study reached consensus on all these factors. Moreover, while no Canadian study included in our systematic review mentioned the factors “lack of time and workload” or “resources available”, we found that Canadian healthcare professionals perceive these factors as important barriers to EHR implementation. A recent Canadian report [
38] confirms our findings, indicating that significant labour and skills shortages among health professionals experienced with electronic health information systems are likely to constrain the successful implementation of such systems in Canada.
Conclusion
This study provides some key findings that corroborate and expand current knowledge of the factors influencing EHR implementation in Canada. We conclude that the perspectives and priorities concerning the barriers and facilitating factors to EHR implementation in Canada could vary greatly between and within user groups and present a challenge to successful implementation of EHR programs [
39].
In summary, we would like to invite EHR implementation researchers to further explore the list of priority factors specific to Canadian user groups that were identified in this Delphi study. Although important recruitment efforts have been made, it was not possible to recruit sufficient patient representatives and the patient Delphi study was thus cancelled. This limitation should be taken into account in future research.
Summary table
What was already known on this subject
There is currently a strong focus worldwide on the potential of EHRs to reduce healthcare costs and significantly improve the quality of healthcare provided.
The implementation of EHR programs is complex and involves many user groups (e.g. physicians, healthcare professionals, health information professionals, managers, and patients).
Understanding users’ perspectives of EHR implementation is essential to the effective implementation of EHRs and to the successful integration of EHRs into the healthcare system.
What this study added to our knowledge
Amongst all barriers and facilitators to EHR implementation identified in a previous systematic review, ten were prioritized in the context of the Canadian healthcare system through this Delphi study.
Users’ perspectives of barriers and facilitators are complex and each user group has unique professional priorities and roles in the EHR implementation process.
We recommend that decision-makers integrate the perspectives specific to each user group, and consider the elements of consensus that emerged from this Delphi study, when implementing future EHR projects.
Competing interests
All authors declare that they have no competing interests.
Authors’ contributions
All authors contributed to the design of the study. CAM coordinated the Delphi study and drafted the article, which was critically revised by MPG, FL, CS, NS, SG, and ABA. All authors approved the final manuscript submitted for publication. MPG is the guarantor for the study. All authors had full access to all of the data (including statistical reports and tables) in the study and can take responsibility for the integrity of the data and the accuracy of the data analysis.