← Back

Health Informatics

Information technology in healthcare, including electronic health records, clinical decision support, telemedicine, health data analytics, and the use of AI in medicine

11 papers

Papers

Beyond the Leaderboard: Rethinking Medical Benchmarks for Large Language Models

This study introduces MedCheck, a framework with 46 criteria to assess the quality of medical benchmarks for large language models (LLMs). Analysis of 53 existing benchmarks revealed systemic issues including a disconnect from clinical practice, poor data quality control, and a lack of safety and fairness evaluations. The paper proposes MedCheck as a tool to guide the creation of more robust and clinically relevant benchmarks.

★ ★ ★ ★ ☆
Health Informatics Aug 14, 02:32 PM

Multimodal Al predicts clinical outcomes of drug combinations from preclinical data

MADRIGAL uses multiple data sources to predict how well and how safely different drugs will work together. It predicts the effects before the drug combinations are tested on people and can be used to screen for adverse drug interactions and personalize cancer treatments. However, the predictions rely on existing data, which may be biased, and clinical trials are still necessary to confirm the results.

★ ★ ★ ★ ☆
Health Informatics Aug 12, 02:26 AM

Towards physician-centered oversight of conversational diagnostic AI

In a simulated clinical setting with human oversight, a diagnostic AI (g-AMIE) demonstrated high performance in taking patient histories, generating SOAP notes, and proposing diagnoses and management plans. While the study setting was artificial and didn't fully replicate real-world clinical practice, g-AMIE performed favorably compared to human clinicians (nurse practitioners, physician assistants, and early-career physicians) under a similar constrained workflow. Notably, the AI consistently observed safety guardrails by deferring individualized medical advice to overseeing physicians.

★ ★ ★ ★ ☆
Health Informatics Jul 31, 02:29 AM

AI-based Clinical Decision Support for Primary Care: A Real-World Study

This real-world study found that an AI-powered clinical decision support tool, AI Consult, led to a significant reduction in diagnostic and treatment errors made by clinicians in primary care clinics in Nairobi, Kenya. The tool improved clinician performance across various areas, including history taking, investigations, diagnosis, and treatment, with particularly strong effects observed after the introduction of active deployment strategies. While there was no statistically significant difference found in patient-reported outcomes, the study suggests the potential of LLM-based tools to improve the quality of care in real-world settings.

★ ★ ★ ★ ☆
Health Informatics Jul 24, 10:50 AM

ChatGPT: Bullshit spewer or the end of traditional assessments in higher education?

This paper explores the implications of ChatGPT for higher education, focusing on its potential to disrupt traditional assessments. It discusses the challenges and opportunities presented by the technology, along with recommendations for students, teachers, and institutions. The authors advocate for a balanced approach, emphasizing the need for educators to adapt and innovate rather than simply banning the use of AI tools.

★ ★ ★ ☆ ☆
Health Informatics Jul 14, 11:25 AM

How Does ChatGPT Perform on the United States Medical Licensing Examination (USMLE)? The Implications of Large Language Models for Medical Education and Knowledge Assessment

ChatGPT demonstrated performance equivalent to a passing score for a third-year medical student on USMLE Step 1 and Step 2 practice questions, exceeding the accuracy of other large language models like GPT-3 and InstructGPT. While demonstrating logical reasoning in all responses and using internal information effectively, the model's reliance on external information was stronger for correct answers, highlighting a potential link between knowledge access and performance.

★ ★ ★ ★ ☆
Health Informatics Jul 14, 11:25 AM