Noora Health deployed a hybrid system to triage emergency health questions in India
Noora Health described a hybrid system (LLM + deterministic rules) for triaging emergency health questions on WhatsApp in India. Recall increased from 56.5 % to 81 %, F1 from 0.606 to 0.702; it has processed 152 421 questions since deployment.
Noora Health runs a WhatsApp service in India through which nurses answer over 50 000 health questions each month from caregivers of mothers and newborns. The most demanding task is triage – deciding which questions require immediate in-person assistance. According to the organization, the original system, based on a large language model (LLM) that classified a question as an emergency and provided a rationale, was opaque: analyzing errors required reading chains of reasoning for each message, which was impractical at that volume, and every prompt change required a new evaluation of the entire system. Moreover, the clinical decision tree followed by nurses had never been documented or provided to the model, which worked only with a flat list of warning signs.
The new system divides triage into two steps: the LLM first extracts standardized symptoms and patient context using a vocabulary compiled by clinicians, then a deterministic rule system evaluates whether the combination matches an emergency scenario. According to the organization, this change increased recall from 56.5 % to 81 % and the F1 score from 0.606 to 0.702, with structured rules accounting for most of the improvement in accuracy. Splitting the process into two steps also enables auditability – clinical staff can review each stage retrospectively and determine whether there was an incorrect translation, incorrectly extracted symptoms, incorrectly inferred patient context, or a missing rule, and can add new rules independently without risking regressions that would require costly reevaluation.
Since deployment, according to the organization, the system has processed 152 421 patient questions, of which it flagged 28 535 (18.7 %) as emergencies. The over-escalation rate reached 17.8 % without an increase in the number of missed emergency cases. Nurses have added 48 new rules since deployment, which the organization cites as evidence of the faster feedback it wanted the new system to enable.
Why it matters
The case shows a concrete alternative to a purely LLM-based solution for safety-critical classification: according to the organization, splitting the task into structured data extraction by a model and decision-making using deterministic rules increased accuracy and allowed non-engineers (clinical staff) to inspect and fix the system without the need for repeated costly evaluation. This is relevant to anyone building AI classification systems in regulated or high-risk areas where decision traceability is required.
Relevant practical impact
What this means
For a business
The case demonstrates a pattern for companies building AI classification systems in high-risk areas: according to the organization, combining an LLM for structured data extraction with deterministic rules for decision-making improved accuracy while allowing domain experts (not engineers) to audit and modify rules without the need for repeated costly evaluation of the entire…
DevelopmentCheck the original
Event sources
only one source so far · 1 publisher, 1 independent. We count feeds from the same owner only once.