Anzeige
11.09.2024 | Original Article
Evaluating the accuracy and adequacy of ChatGPT in responding to queries of diabetes patients in primary healthcare
Erschienen in: International Journal of Diabetes in Developing Countries
Einloggen, um Zugang zu erhaltenAbstract
Objective
This study evaluates the accuracy and adequacy of Chat Generative Pre-trained Transformer (ChatGPT) in responding to common queries formulated by primary care physicians based on their interactions with diabetic patients in primary healthcare settings.
Methods
Thirty-two frequently asked questions were identified by experienced primary care physicians and presented systematically to ChatGPT. Responses underwent evaluation by two endocrinology and metabolism physicians which utilized a 3-point Likert scale for accuracy (1, inaccurate; 2, partially accurate; 3, accurate) and a 6-point Likert scale for adequacy (1, completely inadequate to 6, completely adequate). Questions were categorized into groups including general information, diagnostic processes, treatment procedures, and complications.
Results
The median accuracy score was 3.0 (IQR, 3.0–3.0), and the adequacy score was 4.5 (IQR, 4.0–5.8). None of the questions received an inaccurate rating, and the lowest accuracy score assigned by both evaluators was 3. Significant agreement was observed between the evaluators, demonstrated by a weighted κ of 0.61 (p < .0001) for accuracy and substantial agreement with a weighted κ of 0.62 (p < 0.0001) for adequacy. The Kruskal–Wallis tests revealed no statistically significant differences among the groups for both accuracy (p = .71) and adequacy (p = .57).
Conclusions
ChatGPT demonstrated commendable accuracy and adequacy in addressing diabetes-related queries in primary healthcare.