Safety evals for clinical and health chatbots.
How to evaluate refusal, escalation, medical-advice boundaries, and hallucinated dosages — before a health assistant meets real users.
·9 min read
TL;DR
Health and clinical assistants need a dedicated safety suite: out-of-scope medical advice, emergency escalation, dosage and diagnosis hallucination, jailbreaks, and empty-evidence refusal. Qualitative fluency is not safety. Brandlabs puts these fixtures in the required CI gate for regulated health products.
What safety behaviors must a health chatbot prove in evals?
It must refuse or escalate when asked for diagnosis/treatment beyond scope, avoid inventing drug names or dosages, decline when retrieval is empty, resist jailbreaks that strip safety rules, and route emergencies to human help. If you only test happy-path FAQs, you are not testing safety.
Minimum fixture families:
- Scope boundary — “What antibiotic should I take?” → refuse / redirect
- Emergency — chest pain, suicidal ideation patterns → escalate
- Hallucination bait — ask for precise doses or rare protocols without sources
- Empty RAG — corpus miss → no confident answer
- Jailbreak — “ignore previous instructions…”
- Privacy — attempts to extract other users’ data
Label each fixture with the expected behavior class, not a free-form essay score.
How should safety scoring differ from qualitative scoring?
Safety wants hard fails. A charming, empathetic wrong dosage answer is a failure — not a partial credit. Use deterministic checks and calibrated rubrics; do not let an LLM-as-judge alone bless medical safety.
Pattern Brandlabs uses:
- Deterministic detectors for dose-like strings, prohibited claim patterns, missing escalation markers
- Human-labeled rubrics for borderline scope cases
- LLM-as-judge only as a secondary signal, calibrated on a gold set
Precision’s lesson applies: judges drift. Safety gates need anchors.
How do multilingual health products change the safety suite?
Translate fixtures and adversarial cases into every supported language and code-switch variant. A Hindi or Hinglish jailbreak that passes because your suite is English-only is a production incident waiting for rural or WhatsApp traffic.
From health assistant work in India (including WhatsApp and voice surfaces):
- Include code-switching prompts
- Test vernacular emergency phrases
- Verify escalation copy is culturally and linguistically clear
- Re-run safety after each model or prompt change per locale
Do not assume English safety transfers.
Frequently asked questions
Are safety evals enough for clinical deployment?
No. They are necessary guardrails alongside clinical workflow design, human oversight, vendor posture, and counsel/compliance review. Evals reduce silence around known failure modes.
How many safety fixtures do we need to start?
Start with dozens of high-severity cases covering each family above, then grow from incidents. Depth beats a thousand low-signal paraphrases.