Detection of primary Sjögren’s syndrome in primary care: developing a classification model with the use of routine healthcare data and machine learning

Primary Sjögren’s Syndrome (pSS) is a chronic autoimmune disease, affecting mainly salivary and lachrymal glands, primarily in post-menopausal women [1]. Due to the low prevalence, poorly understood pathophysiology and a wide variety of symptoms, pSS is under-recognized and thus heavily underdiagnosed or misclassified in clinical practice today. This has a negative impact on the time to diagnosis (TTD) [2], which typically takes place in specialized secondary care. The prevalence of pSS varies greatly across studies, with a point estimate of 0.61‰, but ranging from 0.11–37.9‰ [3]. Currently, no separate code for pSS is present in the International Classification of Primary Care (ICPC) coding system [4]. With the growing usage of real-world data derived from administrative and clinical data, new possibilities for the earlier recognition and diagnosis of complex diseases with low prevalence like pSS arise. Routinely collected healthcare data can offer additional sources of specific data collection for clinical research [5] regarding pSS and machine learning models have been shown to be useful methods to extract previously unknown and potentially useful information regarding complex diseases like pSS [6]. In this study, we aimed to apply machine learning (ML) techniques to data from routinely recorded electronic health records (EHRs) from general practitioners (GPs) and to develop an algorithm that identifies possible pSS patients at an early stage. The algorithm might be able to facilitate the diagnostic process by GPs as a clinical decision support system (CDSS) [7].

Detecting pSS in primary care can be improved, as patients with pSS are dealing with a significant delay in their diagnosis. We attempted to build a classification model which aims to predict a Diagnosis Related Group (DRG) for pSS, based on primary care data like drug prescriptions, number of visits to the GP and registered symptoms or diagnoses. Besides primary care data, patient characteristics like age and gender were also included in the model. Machine learning was used to build this model as ML can be used to deal with large amounts of data needed to train accurate classification models in complex diseases like pSS. Data sharing initiatives for pSS research have been started [8], but so far no study has been conducted using ML techniques to classify patients with pSS in primary care. This study will be the first to look at the potential of ML to classify patients with pSS in primary care, based on GP EHR data.

The aim of the study was to develop a classification model to identify possible pSS patients at an early stage. The model might be able to expedite specialist referral by GPs, by forming the basis for the development of a CDSS for GPs, as was recently been done for several other low prevalent diseases [7, 9]. Since ML models differ in terms of transparency and interpretability, both logistic regression and random forest were used and compared. A logistic regression model is more transparent and interpretable, but can only capture linear relationships between features and outcomes. A random forest model is less transparent and interpretable, but is better at handling non-linear relationships. This can be an advantage in complex and multi-dimensional datasets.

A total of 930,590 patients from the Nivel PCD were linked to 12,991,265 secondary care patients in the DIS database. All 930,590 patients present in the Nivel PCD could be linked to the DIS database, so combined primary and secondary care trajectories could be analyzed for these patients. We removed 28,675 pSS patients, as they were diagnosed before 2017. Ultimately, 1411 pSS patients were identified in the DIS database in 2017.

In Table 1, patient characteristics and most common prescriptions, diseases and symptoms are shown and stratified by gender and age group. For women, these descriptive statistics are also stratified by age below or above 45 years old, to check for pre- and postmenopausal differences. The chosen age of 45 was conservatively based on the age range at which the menopause usually starts [21]. On average, pSS patients were between 60.5 and 64.0 depending on gender. Most of patients were women (71.9%).

The aim of this study was to develop an algorithm to detect pSS patients in primary care. This could be used as input to develop a CDSS for GPs to improve the early detection of pSS. This is the first study that looks at the potential of using machine learning to classify patients with pSS in primary care using EHR data. By linking primary and secondary care data for pSS and non-pSS patients, we were able to confirm basic but important differences between these groups reported in the literature [2, 3]. Overall, our two models performed quite well in detecting patients with pSS. However, due to the low prevalence of the disease, many false positives occurred. Both models were near perfect in classifying non-pSS patients, with 99.9% of classified non-pSS not having a pSS diagnosis. These models could be used as input to develop a CDSS in the future, but still need improvement before being useful in general practices.

Although the negative predictive value, sensitivity and specificity varied between moderately good to near perfect for both models, the positive predictive value was poor. The low positive predictive value could partly be attributed to the low prevalence of pSS, making the chance of having pSS within a population very low in the first place. The prevalence in our dataset (0.15‰) is comparable to the prevalence reported in international literature, with a range of 0.11–37.9‰ and a point estimate of 0.61‰ [3]. A study performed by Lutgendorf et al. (2016), showed that even with a sensitivity and specificity of 99.9%, the PPV was only 45% in a population with a prevalence of 0.8‰. However, with a prevalence of 5%, the PPV with the same specificity and sensitivity was almost 100%. This phenomenon is known as the false positive paradox [22, 23]. The target population when testing for a rare disease is crucial for the model performance and usefulness and is also of paramount importance for both our models. This means that positive results in a general practice setting should be interpreted with caution due to the low prevalence. Negative results, however, can be reassuring, as the negative predictive value has a very low error rate.

Key features of the models were S01X (other ophthalmologicals), N02A (opioids), F13 (abnormal sensations in eye), age, gender and number of visits to the GP. These features are in line with reported symptoms in the literature [1‐3, 24]. However, L99 (other musculoskeletal diseases), F99 (other diseases eyes) and non-Hodgkin’s disease (B72.02) were not as important as expected when compared to previous studies [4, 24]. None of the three diseases above were in the feature importance top-15 for either of the models. Of these features, F99 was the most important feature, ranked 23th in the RF and 42th in the LR. The RF had a lower sensitivity and a better specificity compared to the LR model, but even though feature importance can be calculated, the exact build-up of the final RF model is difficult to interpret. This is due to the characteristic ‘black box’ for which RF is known [25], whereas the LR model is more transparent and understandable. In practice, interpretation of the LR model might be of high value for clinicians, even though this could mean a slight underperformance compared to the RF. Also, even though model performance of the RF was stable during the cross validation procedure, it was more prone to overfitting (as can be seen in Appendix III) and the LR model might be a more safe model when used on an external dataset.

This study has several limitations. First, the data was derived from primary care EHRs and medical claims data from secondary care, for which no data is known before 2013 in case of the latter. It could be that some of the patients were diagnosed with pSS before 2013 in secondary care but did not have any medical claims after 2013 and we have missed these patients (and are therefore considered to be non-pSS patients). Second, the diagnosis of pSS in the medical claims database was the outcome feature in this study, for which the reliability is unknown. However, no potential bias (i.e., over- or underestimation of the actual amount of pSS patients) is expected, as there is no known incentive for either over- or underdiagnosing patients with pSS in secondary care. As stated in the method section, no discrimination between pSS and sicca syndrome is made within the ophthalmologic DRG for pSS. Therefore, part of the outcome feature might be sicca syndrome and not pSS, as we did not want to exclude any potential pSS patients. Third, the quality of the determinants on which the classification models is trained, is of high importance as well [26]. These determinants are derived from EHRs, which carries a risk for error. Because even though EHR data will probably be more reliable than claims data, as its purpose is to facilitate the medical process, its purpose is not to provide information for clinical research. Finally, different doctors may use different classification criteria to diagnose pSS, as is reported by Argyropoulou et al. (2018) [27]. The strengths of this study are the inclusion of real-world data from many patients with an until now relatively poor understood illness, the long period over which the data has been analyzed and a unique combination of primary and secondary care data for patients with pSS.

To our knowledge, this is the first study which looks at the potential of using machine learning to classify patients with pSS in primary care using EHR data from the GP. Previous machine learning studies in pSS have focused on 1) determinants of diagnoses based on hospital EHR data [28], 2) pathogenesis based treatments [29] or 3) identifying pSS subtypes [6]. This study resembles the first exploration of the potential of machine learning methods for classifying patients with pSS in primary care. Since the purpose of data does not always fit with the use of this data in classification and prediction models, external validation of these models is extremely important to confirm their performance in the real world. Throughout the literature, studies on prediction and classification models are notorious for their lack of (external) validation [30]. A systematic review comparing ML with health professional assessment in diagnosis of various diseases from medical images showed that only 24% of the studies evaluated the performance of their algorithm on external data [26]. Moreover, only 17% compared this out-of-sample performance of their algorithm with the assessment of actual health professionals. Adding to this is the point that low prevalence diseases like pSS are even more complex to assess, and this makes the results of a recent scoping review on the use of ML in rare diseases highly relevant. This study found that only 11.8% of current studies validated their classification models on an external data set and even fewer on a human expert (2.4%) [31]. Our models should therefore be validated in the future with the use of external datasets. Including frontline clinicians in this process will be critical to further evaluate the algorithm and to further advance success in the field of early detection of pSS in primary care.

Our study shows that machine learning techniques can classify patients as having pSS with the use of routine healthcare data. These results can be used to further develop a CDSS for detecting pSS in primary care. Combining big data with machine learning techniques is a promising approach, but careful considerations for the selection and interpretation of real-world data are needed. Future studies should validate these models based on external datasets and in real-world settings, in order to bridge the gap between research and clinical practice.