Here’s a study evaluating the accuracy of ChatGPT, an artificial intelligence (AI) tool, in providing information about eosinophilic esophagitis (EoE).
Here’s a summary of the key findings:
Mixed Accuracy in Responses: The study found that ChatGPT’s responses to common questions about EoE were a mix of accurate and inaccurate. Notably, the AI provided incorrect information regarding the association between EoE and cancer.
Complexity and Readability Issues: The responses from ChatGPT were found to have low readability and high complexity, potentially creating a barrier for users with different levels of health literacy.
Implications for Clinical Use: The findings suggest caution in using ChatGPT for clinical practice or patient education related to EoE. The technology, as it stands, is not adequately trained to answer clinical questions about this condition, and its use could lead to misinformation.
Study Methodology: The study involved evaluating ChatGPT’s responses to 40 common EoE-related questions, categorized into general topics, complications, and therapeutics. The questions were posed both individually and sequentially to assess the tool’s conversational context capabilities. The responses were scored for scientific correctness and educational value based on known EoE literature and guidelines.
Results and Recommendations: A significant portion of the responses received low scores for scientific correctness and educational value. The study highlighted a specific incorrect suggestion by ChatGPT about the risk of esophageal adenocarcinoma and a correlation between EoE and Barrett’s esophagus. The authors recommend clinical oversight when using ChatGPT for EoE-related information and caution its use by physicians in patient care.
The study underscores the need for careful evaluation and potential limitations of AI tools like ChatGPT in medical education and patient care, especially for conditions with less robust literature or those that are less common.