Introduction
Methods
Results
Delphi Participant Panel
Background | Computer science / Engineering | Medicine | Nursing | Other health sciences | Other |
---|---|---|---|---|---|
39.3% | 28.6% | 3.6% | 10.7% | 7.1% | |
Professional experience | More than 10 years | 5–10 years | Less than 5 years | ||
85.7% | 3.6% | 10.7% | |||
Working sector | Academia | Public health sector | Privat health sector | ||
92.9% | 17.9% | 7.1% | |||
Country of residence | Europe | Australia and Oceania | North America | ||
75% | 10.7% | 14.3% |
Benefits of Transformer Models in Healthcare
-
A1: Increased efficiency and optimization of healthcare.
-
Transformer models can improve healthcare efficiency by accelerating diagnoses and automating tasks like triage, appointment scheduling, and clinical trial matching. This automation helps reallocate human resources to critical tasks, reducing their burden and workload.
-
-
A2: Quality improvement in documentation tasks.
-
Transformer models can improve clinical documentation by summarizing large amounts of information and tailoring the writing style for different readers, reducing the burden on healthcare professionals and improving documentation quality.
-
-
A3: Improvement of clinical communication.
-
Transformer models can improve clinical communication between health professionals and with patients by reducing errors and tailoring information to the language, cultural level or age of the recipient. They could also facilitate the collection of information from patients at a distance during initial contact or follow-up.
-
-
A4: Enhanced and improved clinical procedures.
-
Transformer models could improve healthcare processes through evidence-based decision making, accurate diagnoses through automated data analysis and prediction (e.g. “help in identifying patterns and predicting outcomes in healthcare data”), and automated generation of treatment plans (e.g. ”develop more effective treatment plans”).
-
-
A5: Provision of personalized care.
-
Automatic data analysis using advanced algorithms enables the implementation of personalized medicine. In this regard, some participants pointed out that treatment and diagnosis can become personalized and preventive by transformer model-based systems.
-
-
A6: Improved access to data and knowledge.
-
Transformer models improve data access and processing for better knowledge creation, efficiently extracting relevant information from large, unstructured healthcare data. They also enable easier human-computer interactions, such as voice user interfaces to access information and knowledge.
-
-
A7: Increased individuals’ empowerment.
-
Transformer models in healthcare will empower individuals, patients, carers as well as health professionals, by supporting them through information provision and enhancing their knowledge as needed.
-
Shortcomings of Transformer Models in Healthcare
-
B1: Quality of the transformer model-based systems.
-
This theme comprises two subthemes: system development aspects and erroneous system results. System development issues arise from data dependency, as the quality of transformer models is affected by biases in the training data, such as race and gender bias. Participants noted the need for high-quality, annotated data for training purposes, which is limited due to high annotation costs. The second subtheme, erroneous system results, involves risks from incorrect information provided by transformer models. Challenges include verifying information, dealing with errors or hallucinations and the lack of explainability and interpretability. These issues could harm patients and reduce health professionals’ trust and acceptance of these models. Participants emphasized the importance of testing transformer models in healthcare and real-world scenarios to ensure reliability.
-
-
B2: Compliance with regulations, data privacy and security.
-
Transformer model-based systems must comply with privacy regulations and protect the privacy of sensitive health data, particularly from potential third-party access and misuse.
-
-
B3: Human factors.
-
This theme relates to the health professionals who are expected to use systems based on transformer models. Issues include the need for human expertise to judge the results and their accuracy, overreliance, carelessness and the underdevelopment of skills.
-
-
B4: Reduced integration into healthcare.
-
The theme concerns the reduced integration of transformer model-based systems into healthcare workflows and challenges related to their uptake and use. Participants identified the increased complexity of care caused by the proliferation of information, including that generated by transformer model-based systems, as a key challenge to adoption and use by healthcare professionals.
-
-
B5: Ethical concerns.
-
Biased training data could exacerbate health inequalities, and the need for technical resources and professional training, which is not uniformly available across health centers, could further contribute to inequalities.
-
-
B6: De-humanization of care.
-
Transformer models could affect the doctor-patient relationship by reducing interaction and increasing de-humanization. The automation of care processes could also make patients feel treated as numbers.
-
Risks Associated with the Use of Transformer Models in Healthcare
Risks for PatientCcare
-
C1: Untrusted, inaccurate or biased information.
-
When used to provide clinical decision support, transformer models may lack accuracy or require verification, leading to the risk of misdiagnosis or incorrect treatment. The increasing availability of such models could lead to the use of unreliable or untested systems by health professionals, patients or carers, potentially causing harm.
-
-
C2: Misuse of transformer model-based systems.
-
A major concern was over-reliance on these systems by both patients and professionals, potentially undermining patients’ self-management and decision-making skills in the care process. To mitigate this, participants emphasized the need for patient education on responsible use and correct interpretation of results from transformer model-based systems.
-
-
C3: Impact on the patient-doctor relationship.
-
The patient-doctor relationship, normally based on trust, empathy, respect and continuity, could be compromised by overreliance on diagnoses or treatment suggestions from digital systems. Some participants noted that the excessive focus on these digital technologies by healthcare professionals could lead to worsen interpersonal relationship with patients. Patients could negatively perceive this overreliance because they could feel that digital solutions are replacing doctors resulting in a de-humanization of the healthcare. One participant commented that this deterioration in relationships could even extend to the institutions, leading to patients underestimating and distrusting the healthcare system.
-
-
C4: Liability in case of errors and misuse.
-
The issue of liability is a major concern in relation to the risk of misdiagnosis and mistreatment. In cases where systems malfunction or fail, determining responsibility remains an unresolved challenge.
-
-
C5: Bias and inequity.
-
Systems based on transformer models, which are often trained on biased data, could exacerbate health inequalities. Factors such as low literacy, accessibility issues and socio-economic status provide barriers to patient use of these solutions.
-
-
C6: Data privacy and security.
-
Participants identified privacy and security risks in patient care (e.g. data breaches or unauthorized access to data) and emphasized that personal health information, especially sensitive data, is protected by law and is essential for a trusting patient-doctor relationship. They agreed that the processing of patient data by transformer model-based systems could lead to violations of patient rights.
-
Risks for the Medical Profession
-
D1: Need for training on new competences, and loss of skills.
-
This category concerns overconfidence, overreliance, undervaluation, the need for specific education and training for health professionals, and the erosion of clinical skills and confidence in quality. Participants stressed the importance of training professionals to understand and correctly use and interpret the results of these systems, not to overrely or undervalue their results, and highlighted concerns about confidence in their quality and effectiveness. Health professionals need to learn when to trust the system versus their own expertise. Finally, there is concern that reliance on these systems could undermine critical thinking skills.
-
-
D2: Impact on the patient-doctor relationship.
-
The negative impact on the patient-doctor relationship is a key issue regarding the risks of using transformer models in medicine. Participants agreed that these systems could reduce patient-doctor communication, potentially leading to a loss of patient trust and weakening the patient-doctor relationship.
-
-
D3: Unintended consequences.
-
The use of transformer models in healthcare can lead to unintended consequences, such as incorrect diagnoses and inappropriate treatment plans, often due to incorrect model outputs or an overestimation of the models’ capabilities.
-
-
D4: Legal, liability and ethical concerns.
-
Participants identified and discussed potential legal and ethical issues in the use of transformer models in healthcare, including privacy, data security and patient autonomy. Concerns were also raised about the liability of healthcare professionals for errors or misuse of these systems.
-
-
D5: Impact on jobs.
-
The introduction of transformer models in healthcare could have an impact on jobs: creating new roles, changing existing roles and possibly leading to job losses in medical professions.
-
Risks for Health IT
-
E1: Need for resources to develop and integrate transformer models in healthcare systems.
-
Participants highlighted the need for multiple resources to develop, deploy, integrate and maintain transformer models in healthcare. They found the integration of these systems into existing health IT infrastructures to be particularly challenging. Concerns included development, integration and operational costs, which could exacerbate inequalities due to financial constraints in healthcare institutions. Lack of reimbursement models and time constraints were also significant factors. The need for specialized human resources and expert development of these systems was emphasized, and the risk of their unavailability was noted. In addition, specific training was considered essential for the effective uptake and use of transformer model-based systems.
-
-
E2: Complex regulatory situation and legal issues.
-
Complex regulations in different countries, such as medical device regulations, General Data Protection Regulation (GDPR) and Health Insurance Portability and Accountability Act (HIPAA), already pose risks to the health IT sector and even more regulation is needed. The adoption of transformer models in health IT raises issues around intellectual property, patents and licensing, potentially hindering collaboration, knowledge sharing and industry adoption, and increasing the risk of litigation. Despite their potential to advance medical research, diagnosis and treatment, challenges remain in the ownership and licensing of these models. In addition, determining liability and responsibility for misdiagnosis and mistreatment due to incorrect system outputs remains a pressing issue.
-
-
E3: Quality of solutions.
-
Participants identified quality issues related to transformer models, including the quality of information, data, models, validation and evaluation. They emphasized the importance of the quality of system results, noting that inaccurate, inappropriate or confusing information could lead to unintended consequences. The quality of systems was linked to training data, with concerns about the use of models outside their training context. Despite recognizing the need for high quality systems to prevent patient harm, participants found it challenging to evaluate and validate transformer models due to the lack of standardized evaluation frameworks. They also noted that competitive pressures to develop and market new tools could compromise system quality.
-
-
E4: Data privacy and security.
-
Transformer models handle large amounts of sensitive data, which contributes to associated security and cybersecurity risks.
-
-
E5: Ethical aspects.
-
Participants reported ethical concerns related to the use, development, and training of transformer models as important factors to consider.
-
Risks for Data Protection
-
F1: Unauthorized exposure of data.
-
The use of transformer models in healthcare could lead to confidentiality issues, including unauthorized data disclosure, breaches of privacy regulations, data leakage, and insecure data storage and transmission.
-
-
F2: De-identification and anonymization.
-
Participants raised concerns about de-identification and anonymization in transformer models, noting the risk of exposing sensitive data and the use of weak anonymization techniques that reduce their trustworthiness.
-
-
F3: Data governance.
-
There are risks of lack in transparency and a need for clear descriptions of how transformer model-based systems handle patient data. Concerns have also been raised about inadvertent disclosure of medical data to third parties during development, which poses privacy and security risks.
-
Reliability of Health Systems Based upon Transformer Models
-
G1: Supervised and transparent use.
-
Participants emphasized that the reliability of transformer model-based systems can increase when a human is involved. The ability to interpret and repeat results is key to reliability. The systems should explain how the model arrived at its results. Their use should be made transparent to patients.
-
-
G2: Data integrity and generalizability.
-
Data quality, particularly in terms of diversity and representativeness of the target population and health context, was considered critical for reliability. Participants also identified generalisability as a key factor in the real-world applicability of transformer models.
-
-
G3: System quality.
-
This theme covers aspects such as output, outcome, model quality, regulatory compliance, accuracy, efficiency, effectiveness, robustness, resilience, bias minimization and fairness. Key issues include compliance with security and privacy regulations, accuracy through validation and testing, and the importance of effectiveness and efficiency for reliability. Robustness and resilience of models are seen as critical, and minimizing bias and ensuring fairness are also essential for system reliability.
-
Discussion
Principal Results
Relation to Other Work
Research Agenda
-
Responsible design: Considering ethical and other risks during development to create solutions that mitigate these issues.
-
Utilizing real-world data: Evaluating model quality and performance using authentic, diverse healthcare data for a realistic assessment of capabilities.
-
Testing and Integration: Rigorous testing and seamless integration into health IT systems and workflows to ensure practicality and effectiveness in clinical settings.
-
Education and training: Providing education and training for patients and health professionals to improve interaction with transformer-based systems [33].
-
Continuous risk assessment: Ongoing evaluation of potential risks and shortcomings during the design and development process.
-
Postmonitoring procedures: Implementing robust postmarketing surveillance to ensure patient safety, quality, transparency, and ethics, addressing challenges and risks over time [34].
Limitations
Conclusions
-
Human-in-the-loop systems to ensure oversight and accountability.
-
Transparency in explaining the results of these models.
-
Ensuring high quality data.
-
Maintaining robust system quality, including reliability and accuracy.
-
Compliance with regulatory standards.