HealthBench explained: what OpenAI's health benchmark measures
A plain-English breakdown of HealthBench: 5,000 conversations, 262 physicians, 48,562 rubric criteria, and what it does and does not tell you about a health chatbot.
- Global
Key takeaways
- HealthBench has 5,000 multi-turn conversations with users and health professionals.
- 262 physicians from 60 countries, across 26 specialties, wrote the rubrics.
- Each criterion carries points from −10 to +10, so harmful answers lose points rather than just failing to earn them.
- A model grader (GPT-4.1) checks each criterion. HealthBench Hard is a subset where no model scored above 32% at release.
- Use it as a baseline, then test your own scenarios and languages.
What is HealthBench?
HealthBench is a benchmark that OpenAI released on 12 May 2025 to measure how well large language models handle health conversations. Instead of exam questions, it uses 5,000 realistic multi-turn conversations between a model and either a member of the public or a health professional.
How is it scored?
Each conversation has its own rubric written by physicians: 48,562 unique criteria in total, from 262 physicians practising in 60 countries across 26 specialties. Every criterion has a point value from −10 to +10. Good behaviour, such as advising emergency care when it is needed, earns points. Harmful behaviour, such as a confident wrong dosage, loses them. A model grader checks which criteria a response meets, and the score is the points earned out of the maximum possible.
What do the variants measure?
- HealthBench Consensus uses 34 criteria that several physicians agreed are important, for a stricter, lower-noise signal.
- HealthBench Hard is a subset of difficult conversations. At release, no model scored above 32%.
What does it not tell you?
HealthBench measures general ability on its own conversations. It does not know your users’ languages, your product’s scope, your crisis hand-off or your regulator. A model can score well and still fail on Hinglish questions about contraception, or miss distress expressed indirectly. Treat it as a baseline and build your own test set on top.
How should a product team use it?
Run it when you choose or change a model, track the score over releases, and read the failing conversations, not just the average. Then run your own scenario suite, in your users’ languages, before every launch.
Frequently asked questions
Can we run HealthBench on our own chatbot?
Yes. The conversations, rubrics and grading code are public in OpenAI's simple-evals repository. You can run your system's responses through the same grader, though costs scale with the model grader's usage.
Does a high HealthBench score mean a chatbot is safe?
No. It means the model performs well on HealthBench's conversations. Your users may write in other languages, ask about different conditions or be in crisis in ways the benchmark does not cover.
Sources
- Introducing HealthBenchOpenAI · 2025-05-12
- HealthBench: evaluating large language models towards improved human healtharXiv · 2025-05
Found something out of date? Write to contact@brandlabs.app and we will fix it.