Building a health chatbot: 12 do's and don'ts

What should a health chatbot always do?

1. Do disclose it is an AI, in the first message and whenever a user asks. Most US state chatbot laws now require it.
2. Do hand off crises to a real person. Detect distress and self-harm signals and share a working helpline for the user’s country, such as 988 in the US or Tele-MANAS 14416 in India, in the first reply.
3. Do check for signs of a minor and change behaviour: simpler language, stricter topics, and parental consent where the law requires it.
4. Do cite a source for medical facts, or say you do not know.
5. Do support the languages your users actually write in, including mixed ones such as Hinglish, and test each of them.
6. Do keep an evaluation suite and run it before every release.

What should a health chatbot never do?

1. Don’t claim or imply it is a doctor, nurse or therapist, even in role-play. Tennessee bans it for mental health from July 2026.
2. Don’t invent doses or diagnoses. Route dosage questions to a pharmacist, a clinician or an official source.
3. Don’t store names and phone numbers in logs or prompts. Redact them before storage.
4. Don’t send health data to third-party tools by default, including analytics and model providers, without a legal basis and a contract.
5. Don’t rely on the system prompt alone for safety. Prompts drift; tests catch it.
6. Don’t call your product “compliant” because a vendor is. Compliance depends on how you build and run it.

How do you know your chatbot follows these rules?

Write one test scenario per rule, in every language you support, with a clear pass condition. Run each scenario many times, because model answers vary. Track the pass rate per rule over releases, and block a release if a safety rule drops.