Research Summary: Auditable Emergency Triage for Maternal and Newborn Care in India

Noora Health's WhatsApp-based service for maternal and newborn care in India handles over 50,000 medical queries monthly, with emergency triage being a critical, time-sensitive function. An AI system using a Large Language Model (LLM) was developed to classify emergency messages and provide rationales. However, this system proved opaque, hindering error analysis and making prompt changes and re-evaluation operationally challenging and costly due to its scale, contrasting with clinicians' traditional decision-tree approach.

Why it matters

This research highlights the significant challenges and strategic implications of integrating advanced AI, specifically LLMs, into critical operational processes like emergency triage in healthcare settings. It underscores the tension between technological advancement, operational efficiency, and the imperative for transparency and auditability, particularly in high-stakes domains such as maternal and newborn care. Addressing these issues is crucial for maintaining trust, ensuring patient safety, and scaling digital health services responsibly.

Key insights

• Noora Health manages a high volume of medical queries (50,000+ per month) for maternal and newborn care via WhatsApp.
• Emergency triage, determining queries requiring immediate in-person attention, is a critical and time-sensitive task.
• An LLM-based system was implemented for emergency classification and interpretability through rationales.
• The initial LLM system was opaque, making mistake analysis difficult at scale.
• Modifying LLM prompts necessitated full re-evaluations, which were costly and operationally complex.
• Clinicians traditionally utilize a decision-tree approach for emergency assessments.

Source

arXiv — Computers and Society — https://arxiv.org/abs/2609.09356