Research Summary: Large Language Models Versus Physicians in Traditional Chinese Medicine: A Real-World Clinical Case Evaluation

A recent study assessed the performance of Large Language Models (LLMs) against practicing Traditional Chinese Medicine (TCM) physicians in a real-world clinical case evaluation. Using a clinical case library from 62 hospitals, 16 LLMs and 60 TCM physicians were evaluated on 60 representative cases by five senior TCM experts. The findings indicate that leading general-purpose LLMs achieved higher expert scores than physicians in several diagnostic and therapeutic areas, including medical advice and treatment principles. The abstract suggests limitations may exist for LLMs in some areas, though the provided text is truncated.

Why it matters

This research highlights the potential for advanced AI models to augment or even surpass human performance in complex, nuanced domains like Traditional Chinese Medicine. Such capabilities could significantly impact healthcare delivery, training, and the development of diagnostic and therapeutic guidelines across various medical disciplines. Organizations should monitor the capabilities and limitations of AI in specific clinical applications to inform future strategy and investment.

Key insights

• 16 Large Language Models (LLMs) and 60 Traditional Chinese Medicine (TCM) physicians were evaluated on 60 clinical cases.
• The clinical cases were selected from a library of 349 de-identified outpatient cases sourced from 62 hospitals.
• Evaluation was conducted by five senior TCM experts across nine diagnostic and therapeutic dimensions.
• Cutting-edge general-purpose LLMs scored higher than physician comparators in medical advice, treatment principles, and selected diagnostic tasks.
• The source text is truncated, suggesting potential caveats or further findings that are not fully available.

Source

arXiv — Computers and Society — https://arxiv.org/abs/2609.17544