# HealthBench explained: what OpenAI's health benchmark measures

> A plain-English breakdown of HealthBench: 5,000 conversations, 262 physicians, 48,562 rubric criteria, and what it does and does not tell you about a health chatbot.

Research · Regions: Global · By Harsh Vitra (BrandLabs) · Published 2026-11-14 · Reviewed 2026-11-14 · Source: https://www.brandlabs.app/learn/healthbench-explained

**TL;DR:** HealthBench, released by OpenAI in May 2025, scores how well a model handles realistic health conversations. It has 5,000 multi-turn chats, each with its own rubric written by physicians, 48,562 criteria in total. It is a strong general benchmark, but it cannot replace tests built from your own users, languages and risks.

## Key takeaways

- HealthBench has 5,000 multi-turn conversations with users and health professionals.
- 262 physicians from 60 countries, across 26 specialties, wrote the rubrics.
- Each criterion carries points from −10 to +10, so harmful answers lose points rather than just failing to earn them.
- A model grader (GPT-4.1) checks each criterion. HealthBench Hard is a subset where no model scored above 32% at release.
- Use it as a baseline, then test your own scenarios and languages.

## What is HealthBench?

HealthBench is a benchmark that OpenAI released on 12 May 2025 to measure how well large language models handle health conversations. Instead of exam questions, it uses **5,000 realistic multi-turn conversations** between a model and either a member of the public or a health professional.

## How is it scored?

Each conversation has its own rubric written by physicians: **48,562 unique criteria** in total, from **262 physicians** practising in **60 countries** across **26 specialties**. Every criterion has a point value from −10 to +10. Good behaviour, such as advising emergency care when it is needed, earns points. Harmful behaviour, such as a confident wrong dosage, loses them. A model grader checks which criteria a response meets, and the score is the points earned out of the maximum possible.

## What do the variants measure?

- **HealthBench Consensus** uses 34 criteria that several physicians agreed are important, for a stricter, lower-noise signal.
- **HealthBench Hard** is a subset of difficult conversations. At release, no model scored above 32%.

## What does it not tell you?

HealthBench measures general ability on its own conversations. It does not know your users' languages, your product's scope, your crisis hand-off or your regulator. A model can score well and still fail on Hinglish questions about contraception, or miss distress expressed indirectly. Treat it as a baseline and build your own test set on top.

## How should a product team use it?

Run it when you choose or change a model, track the score over releases, and read the failing conversations, not just the average. Then run your own scenario suite, in your users' languages, before every launch.

## Frequently asked questions

### Can we run HealthBench on our own chatbot?

Yes. The conversations, rubrics and grading code are public in OpenAI's simple-evals repository. You can run your system's responses through the same grader, though costs scale with the model grader's usage.

### Does a high HealthBench score mean a chatbot is safe?

No. It means the model performs well on HealthBench's conversations. Your users may write in other languages, ask about different conditions or be in crisis in ways the benchmark does not cover.

## Where BrandLabs fits

We use public benchmarks as a baseline, then score chatbots on scenarios built from your users' real questions, in their languages, with crisis and minor-safety cases added. https://www.brandlabs.app/services#safety-check

## Sources

- Introducing HealthBench, OpenAI, 2025-05-12: https://openai.com/index/healthbench/
- HealthBench: evaluating large language models towards improved human health, arXiv, 2025-05: https://arxiv.org/abs/2505.08775