Design, implementation, and evaluation of the computer-aided clinical decision support system based on learning-to-rank: collaboration between physicians and machine learning in the differential diagnosis process
We are researching, developing, and publishing the clinical decision support system based on learning-to-rank. The main objectives are (1) To support for differential diagnoses performed by internists and general practitioners and (2) To prevent diagnostic errors made by physicians. The main features are that “A physician inputs a patient's symptoms, findings, and test results to the system, and the system outputs a ranking list of possible diseases”.
Method
The software libraries for machine learning and artificial intelligence are TensorFlow and TensorFlow Ranking. The prediction algorithm is Learning-to-Rank with the listwise approach. The ranking metric is normalized discounted cumulative gain (NDCG). The loss functions are Approximate NDCG (A-NDCG). We evaluated the machine learning performance on k-fold cross-validation. We evaluated the differential diagnosis performance with validated cases.
Results
The machine learning performance of our system was much higher than that of the conventional system. The differential diagnosis performance of our system was much higher than that of the conventional system. We have shown that the clinical decision support system prevents physicians' diagnostic errors due to confirmation bias.
Conclusions
We have demonstrated that the clinical decision support system is useful for supporting differential diagnoses and preventing diagnostic errors. We propose that differential diagnosis by physicians and learning-to-rank by machine has a high affinity. We found that information retrieval and clinical decision support systems have much in common (Target data, learning-to-rank, etc.). We propose that Clinical Decision Support Systems have the potential to support: (1) recall of rare diseases, (2) differential diagnoses for difficult-to-diagnoses cases, and (3) prevention of diagnostic errors. Our system can potentially evolve into an explainable clinical decision support system.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Abkürzungen
CDSS
Clinical decision support system
DDSS
Diagnosis decision support system
RD
Rare diseases
IR
Information retrieval
LTR
Learning to rank
ML
Machine learning
DDx
Differential diagnosis
NDCG
Normalized discounted cumulative gain
A-NDCG
Approximate NDCG, as a loss function
MSE
Mean squared error, as a loss function
ndcg
NDCG, as an evaluation function
mse
Mean squared error, as an evaluation function
Introduction
We are researching, developing, and publishing the Clinical Decision Support System (CDSS) based on Learning-to-Rank (LTR) [1, 2].
This paper discusses our system's design, implementation, and evaluation.
Anzeige
Diagnostic errors and clinical decision support system
Medical errors are among the most critical safety issues in today's healthcare. Medical errors cause the most significant damage (human and economic) to the public.
The well-known report "To Err Is Human." reports that 44,000–98,000 patients die annually in the United States due to medical errors. Deaths due to medical errors are more incredible than deaths due to the three leading causes of death (automobile accidents, breast cancer, and AIDS) [3].
The CDSS will be a competent partner with physicians to prevent diagnostic errors.
In clinical practice, internists and general practitioners also want the practical application of CDSS [5].
Rare diseases, difficult-to-diagnose cases, and clinical diagnosis support systems
Rare diseases (RD) are a generic term for diseases with small patient populations. Rare diseases are the antonym of Common diseases. The definition of rare diseases and the criteria for prevalence are different for each country.
Table 1 shows the Definitions of rare diseases for each country.
Table 1
Definitions of rare diseases for each country
Country
Prevalence
Source
The EU
< 1 person in 2000
EU research on rare diseases
Japan
Not defined
Act on Medical Care for Patients with Intractable Diseases
The UK
< 1 person in 2000
The UK Rare Diseases Framework
The US
< 50,000 persons in the US
Rare Diseases Act of 2002
Difficult-to-diagnose cases have no formal definition. For example, many case reports describe difficult-to-diagnose cases. Rare diseases are often difficult-to-diagnose cases.
Various leading researchers have reported the application of the CDSS for the diagnosis of RD [6, 7].
Main objectives of the clinical decision support system
In our study, the main objectives of the Clinical Decision Support System (CDSS) are as follows:
To support differential diagnoses performed by internists and general practitioners.
To prevent diagnostic errors made by physicians
Main features of the clinical decision support system
In our study, the main features of the Clinical Decision Support System (CDSS) are as follows:
A physician inputs a patient’s symptoms, findings, and test results to the system, and the system outputs a ranking list of possible diseases.
Anzeige
The input information is as follows:
Subjective symptoms
Objective findings
Physical findings
Laboratory test results
Imaging test results
Other Information
(From now on, referred to as "inputted symptoms").
The output information is as follows:
A ranking list of possible diseases
(From now on, referred to as "predicted diseases").
Anzeige
Clinical Decision Support system (CDSS) for Differential Diagnosis (DDx) is also known as Diagnostic Decision Support System (DDSS) [8].
Example of the clinical decision support system
Figure 1 shows the Example of the prediction screen of our system.
From the perspective of LTR, the DDx process by experienced physicians IS NOT a pointwise or pairwise approach.
Pointwise approach:
Score one differential disease at a time.
Pairwise approach:
Compare two differential diseases at a time.
This process IS a listwise approach.
Listwise approach:
(1)
Recall multiple differential diseases
(2)
Refine the recalled differential diseases
(3)
Rank the refined differential diseases
Once again, we propose the DDx process is highly affinitive to LTR, especially the listwise approach.
Case data for clinical decision support system
The case data (= training data) for CDSS is prepared using a literature base [11].
Real World Data (RWD) has not been validated its reliability.
We do not use them as case data for CDSS.
The medical literature includes the following types:
Medical textbooks
Medical treatises
Medical articles
Case reports
(From now on, referred to as "literature").
Good literature, such as case reports, contains information on confirmed disease(s) and (multiple) differential diseases.
Excellent literature, such as Clinical Problem Solving (CPS), contains information on confirmed disease(s) and (multiple and changing) differential diseases by following the DDx process by experienced physicians [12].
The information discussed in case reports is as follows:
Symptoms
Confirmed disease(s)
Differential diseases (related or to be excluded)
The procedure for making the case data for CDSS is as follows:
(1)
Select the literature
(2)
Retrieve the information on cases by text-mining from the literature
(3)
Convert the retrieved data by text-mining to the symptoms and diseases
(4)
Store the symptoms and diseases in the database
Technologies have already been developed to automatically text-mining information on the only confirmed disease from the abstracts of case reports [11].
No technology has yet been developed to automatically text-mining information on confirmed disease(s) and (multiple) differential diseases from the body of literature.
No technology has yet been developed to convert retrieved information by text-mining to metadata automatically.
To improve the predictive performance of the CDSS, we propose it is necessary to define strict criteria for symptoms, diseases, and cases.
The criteria we defined for target cases are as follows:
Rare diseases and difficult-to-diagnose cases that internists and general practitioners may close encounter in actual cases.
The case data in our system are text-mining data from the literature by us.
Information retrieval and clinical decision support system
Information Retrieval (IR) is a technique for retrieving information from information resources that match objectives [10].
Google Scholar is a primary IR service that targets scholarly literature on the Internet.
IR systems such as Google Scholar and CDSS have much in common (target data, framework, etc.).
Table 3 shows the Information Retrieval and Clinical Decision Support System.
Table 3
Information retrieval and clinical decision support system
Items
Information retrieval (Ex: Google scholar)
Clinical decision support system
Objectives
Get medical literature for target diseases
Get possible diseases
Target data
Medical literature
←
Method of retrieving target data
Web crawlers, etc
Selection by physicians
Framework
Learning-to-rank
←
Input data
Symptoms, Diseases
Symptoms
Output data
Ranking list of useful medical literatures
Ranking list of possible diseases
Evaluation method
Subjective evaluation
Objective evaluation
Physicians
Case reports
Evaluation Functions
Retrieval algorithms for IR often use LTR, especially the listwise approach. We propose that CDSS should use several IR technologies (LTR, etc.).
Conventional clinical decision support systems
Various leading researchers have reported on CDSS based on ML [13‐17].
The output of these systems is "predicted diseases." It is "a ranking list of possible diseases." Therefore, these systems are also a type of CDSS based on LTR. However, we assume that the prediction algorithm of these systems uses the pointwise approach. In addition, we assume that the case data of these systems use only confirmed disease information.
We assume that these systems have the following problems:
The predictive algorithms are LTR with a pointwise approach.
These algorithms are less affinitive to the DDx process by experienced physicians.
The case data does not include information on differential diseases.
These algorithms do not use the relationship between confirmed disease(s) and differential diseases.
To address the issues of conventional CDSS, the design principles of our system are as follows:
The prediction algorithms should be higher affinitive to the DDx process by experienced physicians.
The case data should include not only information on confirmed disease(s) but also information on differential diseases.
These algorithms should utilize the relationship between confirmed disease(s) and differential diseases.
To focus on commonalities between IR and CDSS, utilize various IR technologies for CDSS.
Library for learning-to-rank
We used TensorFlow and TensorFlow Ranking as our system's Machine Learning (ML) libraries to satisfy the design principles [18, 19].
TensorFlow Ranking is a library for Learning-to-Rank (LTR). The main targets for TensorFlow Ranking are Information Retrieval (IR) systems and Recommendation systems.
For the ranking metrics of LTR, we selected Normalized Discounted Cumulative Gain (NDCG). NDCG is the ranking metric of LTR (listwise approach) [10].
As discussed before, we propose that the calculation algorithm of NDCG is more affinitive to the DDx process by experienced physicians.
For the loss function of LTR, we selected Approximate NDCG loss.
Approximate NDCG loss is an approximation for NDCG. It is a differentiable approximation based on the logistic function [20].
Case date for learning-to-rank with the listwise approach
The case data for conventional CDSS based on LTR (pointwise approach) has the following information:
Symptoms
Confirmed disease
Table 4 shows the Example of case data (pointwise approach).
Table 4
Example of case data (pointwise approach)
Code
Observed symptoms
Code
Diseases
a
Fever
Fever
548
Acute HIV-1 infection
b
Head
Headache
c
Sore
Sore throat
d
Myalg
Muscles ache
e
Fatig
Fatigue
→
f
Weigh
Weight loss
g
Arthralg
Arthralgia
h
Diarrh
Diarrhea
i
Lymphn
Lymphadenopathy
j
Mening
Meningitis
…
Based on: case data of our system
These have only information on a confirmed disease.
As discussed before, technologies have already been developed to automatically text-mining this information from the abstracts of case reports.
The case data for our CDSS based on LTR (listwise approach) has the following information:
Symptoms
Confirmed disease(s) and these scores
Differential diseases (related or to be excluded) and these scores
Table 5 shows the Example of case data (listwise approach).
Table 5
Example of case data (listwise approach)
Code
Observed symptoms
Scores
Code
Diseases
a
Fever
Fever
17.078
548
Acute HIV-1 infection
b
Head
Headache
12.086
296
Acute hepatitis
c
Sore
Sore throat
11.250
102
Toxoplasmosis
d
Myalg
Muscles ache
11.000
491
Severe fever with thrombocytopenia syndrome (SFTS)
e
Fatig
Fatigue
→
10.836
391
Osteomyelitis
f
Weigh
Weight loss
10.836
589
Polyneuropathy
g
Arthralg
Arthralgia
10.836
641
Coccidioidomycosis
h
Diarrh
Diarrhea
10.664
627
Cat-scratch disease
i
Lymphn
Lymphadenopathy
10.500
541
Infectious endocarditis
j
Mening
Meningitis
10.414
989
Dengue (hemorrhagic) fever
…
…
Citation: case data of our system
This information has not only confirmed disease(s) but also differential diseases. In addition, these diseases are assigned a score according to possibility. This information is described not only in the abstracts of literature but also in the bodies.
Thus, the Information Retrieval (IR) system should parse the abstracts and the bodies (See the section Implementation in Additional file 1).
In this paper, the other conventional systems we cited were not used for comparison [13‐16].
The reasons are:
The main objective is to propose the prediction algorithm (Learning-to-Rank; listwise approach) for CDSS. In the interest of fairness, the comparison conditions (training data, etc.), except for the algorithm, must be the same. However, these systems' algorithms and training data are not publicly available.
Each CDSS has different objectives and target diseases.
The compared system also uses Learning-to-Rank (LTR). However, LTR for the compared system is the pointwise approach. The loss function of the compared system is Mean Squared Error (MSE).
Evaluation criteria for differential diagnostic performance
As evaluation criteria for DDx performance, we focused on confirmed diseases (or related diseases) that should be ranked in the top 10th predicted diseases.
The reasons are:
The DDx process by physicians is a kind of incomplete information game [21]. The acquired information, thoughts, and knowledge may contain mistakes or omissions in this process [22]. In today's CDSS, the main objective is a Decision Support System, not a Diagnosis System.
Physicians decide the final confirmed disease(s) by themselves, using the predicted diseases of CDSS as a reference.
Case selection criteria for evaluation of differential diagnostic performance
In previous articles, cases for evaluation of DDX performance are often actual cases [23].
However, they should be validated cases with case reports, etc.
The reasons are:
Our main target diseases are rare diseases and difficult-to-diagnose cases that internists and general practitioners may close encounter in clinical practice. However, the probability of encountering these diseases is low.
For correct evaluation, it is important to evaluate with validated cases.
"The New England Journal of Medicine (NEJM)" publishes many excellent case reports that fit these purposes.
Therefore, we used case reports from NEJM to evaluate the DDx performance of the CDSS.
Evaluation: machine learning performance
Evaluation method
The Machine Learning (ML) performance of Clinical Decision Support System (CDSS) valuated was as follows:
Learning curves
Value of evaluation function
The data used to evaluate the ML performance were the case data we collected. The number of case data was around 26,000.
We evaluated the ML performance on k-fold cross-validation (k = 5).
In the interest of fairness, the comparison conditions (training data, validated data, hyperparameters. etc.), except for the loss function, were the same.
Loss functions: A-NDCG: Approximate NDCG loss, MSE: Mean Squared Error
In both systems, the predicted ranking of confirmed disease was 1st.
In the predicted diseases of our system, the excluded diseases for AIP (ex: lead poisoning) were listed at the top of the list [28, 29].
In this case, the predicted diseases of our system provided useful information for the DDx process by physicians.
Regarding "Inputted symptoms and the target disease's ranking," in both systems, at the point where the characteristic symptoms (hyponatremia and abnormal liver function) were inputted, the final confirmed disease was listed at the top of the list.
For the DDx of diseases with characteristic symptoms, we suppose that the DDx performances of both systems are not significantly different.
Difficult-to-diagnose case with few characteristic symptoms
We evaluated the Differential Diagnosis (DDx) performance of the difficult-to-diagnose case with few characteristic symptoms.
The DDx of these diseases is difficult to conventional Clinical Decision Support System (CDSS).
Loss functions: A-NDCG: Approximate NDCG loss; MSE: Mean Squared Error
In our system, the predicted rankings of related diseases were as follows:
1st: Acute HIV-1 infection.
3rd: Acute viral meningitis.
However, in the compared system, the predicted rankings of related diseases were less than the 20th.
Regarding "Inputted symptoms and the target disease's ranking," in our system, at the point where few symptoms were inputted, related diseases were listed at the top of the list.
In this case, many of these symptoms are common in other diseases.
For the DDx of difficult-to-diagnose cases with few characteristic symptoms, we suppose that DDx performance of our system is superior.
Case with diagnostic errors
Cognitive biases, such as confirmation bias, are among the most frequent causes of diagnostic errors [31].
Clinical Decision Support System (CDSS) is useful for preventing diagnostic errors.
We evaluated the Differential Diagnostic (DDx) performance of a case with diagnostic errors. The system used for the evaluation of this case was only our system.
The final confirmed disease of the case was subacute bacterial endocarditis caused by bartonella [27].
The title of the case report is "Copycat." In this case, this patient had a history of HCV infection. Initially, due to confirmation bias, the case report's authors did not focus on the characteristic symptoms of endocarditis (heart murmur, purpura, etc.) but this HCV infection. As a result, they reported the misdiagnosed case as mixed cryoglobulinemia by HCV.
In our system, the related diseases, including the confirmed disease, in this case, are as follows:
Subacute bacterial endocarditis (SBE)
Acute bacterial endocarditis
Infectious endocarditis
Therefore, these diseases were also defined as related diseases to confirmed diseases.
In addition, the misdiagnosed disease is as follows:
Mixed cryoglobulinemia
Table 9 shows the Predicted diseases: case of the subacute bacterial endocarditis caused by bartonella: In progress.
Table 9
Predicted diseases: case of the subacute bacterial endocarditis caused by bartonella: In progress
Predicted diseases
Classification
1
Zieve syndrome
2
Disseminated intravascular coagulation
3
Chronic hepatitis
4
Wilson's disease
5
Acute hepatitis
6
Hepatic amyloidosis
7
Infectious endocarditis
Related disease
8
(Compensated/uncompensated) liver cirrhosis
9
Subacute bacterial endocarditis
Related disease
10
Gastric cancer
…
Case: Subacute bacterial endocarditis caused by bartonella [27]
Loss functions: A-NDCG: Approximate NDCG loss; In progress: Number of inputted symptoms = 9
Table 10 shows the Predicted diseases: case of the subacute bacterial endocarditis caused by bartonella: Final.
Table 10
Predicted diseases: case of the subacute bacterial endocarditis caused by bartonella: Final
Predicted diseases
Classification
1
Mixed cryoglobulinemia
Misdiagnosed disease
2
Chronic hepatitis
3
Subacute bacterial endocarditis
Related disease
4
Hepatic amyloidosis
5
Rapidly progressive glomerulonephritis syndrome
6
Acute bacterial endocarditis
Related disease
7
Infectious endocarditis
Related disease
8
Polyarteritis nodosa
9
Autoimmune hemolytic anemia
10
Disseminated intravascular coagulation
…
Cited case: Subacute bacterial endocarditis caused by bartonella [27]
Loss functions: A-NDCG: Approximate NDCG loss; Final: Number of inputted symptoms = 18
In the final predicted diseases (Table 10), the misdiagnosed disease was ranked 1st. The cause was the information by confirmation bias. Nevertheless, the related diseases were ranked in the top 10.
In the progress predicted diseases (Table 9), the related diseases were ranked in the top 10.
Despite the biased information, the system listed the related disease at the top. In the DDx process by physicians, if they had this information, we assume that their differential disease list would include not only HIV infection but also SBE.
We propose that the CDSS, including our system, will prevent diagnostic errors by physicians.
This paper discusses the design, implementation, and evaluation of our Clinical Decision Support System (CDSS) based on Learning-to-Rank (LTR) with the listwise approach.
Evaluation results
We evaluated Machine Learning (ML) performance and Differential Diagnosis (DDx) performance.
The ML and DDx performance of our system (listwise approach: A-NDCG) was higher than that of the compared system (pointwise approach: MSE).
In terms of both ML and DDx performance, we have demonstrated that the CDSS is useful for physicians to support DDx and prevent diagnostic errors.
Differential diagnosis process by physicians and learning to rank by machines
The prediction algorithm of our system is Learning-to-Rank (LTR) with the listwise approach. The Differential Diagnosis (DDx) process by physicians is an iterative process with Recalling, Refining, and Ranking differential diseases.
Case data and information retrieval
Our system's case data (= training data) and predicted results are almost the same data structure.
Table 11 shows the Case data and predicted results of our system.
Table 11
Case data and predicted results of our system
Case data (= training data)
Predicted results
X: explanatory variables
Observed symptoms
Inputted symptoms
y: explained variables
Confirmed disease(s) and those score(s) & Differential diseases (related or to be excluded) and those scores
Predicted diseases and those scores
When experienced physicians validate the predicted diseases, for feedback on validation results to the predictive model, we propose that the results of our system (listwise approach: A-NDCG) are more pertinent than the results of the compared system (pointwise approach: MSE).
As discussed before, no technology has yet been developed to automatically optimize case data for a listwise approach.
Therefore, we had to do these tasks manually (and by only one physician).
As a result, due to his knowledge and thought, our system may have both bias and outstanding performance.
For the practical application of Clinical Decision Support System (CDSS), we propose that developing the following Information Technologies (IT) is necessary:
Technology for predicting diseases, such as Learning-to-Rank (LTR)
Technology for text-mining information on diseases from literatures
Technology for converting text-mining data to the symptoms and diseases
For this purpose, using Information Retrieval (IR) technologies is effective.
Potentials for clinical decision support system
According to our experience and knowledge, we presume that Clinical Decision Support System (CDSS), including our system, has the following potential:
Recall rare diseases
Support differential diagnoses for difficult-to-diagnose cases
Prevent diagnostic errors
Evolution into explainable clinical decision support system
We suppose our system can evolve into an Explainable Clinical Decision Support System (X-CDSS) [32].
The reasons for this are as follows:
The affinity between Differential Diagnosis (DDx) processes by experienced physicians and LTR with the listwise approach
The similarity between case data (= training data) and predicted results
The simple neural network
The number of internal hiding layers is one.
The number of learnable times (epochs) is relatively small.
We will continue to develop the Ultimate Clinical Decision Support System (U-CDSS).
All authors declare that they have no competing interests.
Open AccessThis article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Design, implementation, and evaluation of the computer-aided clinical decision support system based on learning-to-rank: collaboration between physicians and machine learning in the differential diagnosis process
