Open health datasets for AI, and what each is good for
MIMIC-IV, Synthea, MedQA, MedMCQA, PubMedQA, HealthBench, NHANES, India's NFHS-5 and WHO data: access rules, sizes and the best use for each.
- Global
- US
- India
Key takeaways
- MIMIC-IV is de-identified hospital data. Access requires credentialing and a data use agreement through PhysioNet.
- Synthea creates synthetic patients with no privacy risk, which makes it the safest place to start building.
- MedMCQA uses questions from Indian medical entrance exams, so it is useful for testing models on Indian clinical knowledge.
- HealthBench tests conversations, not exam answers: 5,000 multi-turn chats graded against rubrics written by 262 physicians.
- Public benchmarks measure general ability. You still need a test set built from your own users' questions.
Which open health datasets are most useful for AI?
| Dataset | What it is | Access | Best for |
|---|---|---|---|
| MIMIC-IV | De-identified hospital records from Beth Israel Deaconess Medical Center | Credentialed, data use agreement | Clinical research and model evaluation |
| Synthea | Synthetic patient records, generated on demand | Open, Apache-2.0 | Building and testing without privacy risk |
| MedQA | Medical exam questions in US board style | Open | Medical knowledge |
| MedMCQA | About 194,000 questions from Indian medical entrance exams | Open | Indian clinical knowledge |
| PubMedQA | Yes, no or maybe questions from PubMed abstracts | Open | Reasoning over research |
| HealthBench | 5,000 multi-turn health conversations with physician-written rubrics | Open | Conversation quality and safety |
| NHANES | US health and nutrition survey with exams and lab tests | Open | Population baselines, nutrition |
| NFHS-5 | India’s National Family Health Survey, 2019–21 | Free registration | Indian maternal, child and women’s health context |
| WHO Global Health Observatory | Country-level health indicators | Open | Global context and benchmarks |
Why are exam benchmarks not enough for chatbots?
Exam datasets ask one question with one correct answer. Real users ask vague questions over several turns, in mixed languages, sometimes in distress. A model can score well on MedQA and still miss a crisis signal or invent a dosage. That is why conversation benchmarks such as HealthBench, and your own scenario tests, matter more for patient-facing products.
How should you build your own test set?
- Collect real user questions, with identifiers removed.
- Cover every language and script your users write in, including mixed ones such as Hinglish.
- Add adversarial cases: crisis messages, role-play that tries to make the bot claim to be a doctor, and questions about doses.
- Write a short rubric for each case and score every model release against it.
Frequently asked questions
Can we train a commercial model on MIMIC-IV?
Check the PhysioNet data use agreement for the exact dataset version. It restricts sharing the data and requires credentialed access for everyone who uses it. Many teams use MIMIC for research and evaluation, and synthetic or licensed data for products.
Which dataset should we use to test a health chatbot?
Start with HealthBench for conversation quality, add MedMCQA or MedQA for medical knowledge, then build your own set from real, de-identified user questions in every language you support. Your own set matters most.
Sources
- MIMIC-IVPhysioNet
- SyntheaThe MITRE Corporation
- MedMCQA: a large-scale multi-subject multi-choice dataset for medical domain question answeringarXiv · 2022
- PubMedQA: a dataset for biomedical research question answeringarXiv · 2019
- What disease does this patient have? (MedQA)arXiv · 2020
- HealthBench: evaluating large language models towards improved human healtharXiv · 2025-05
- National Health and Nutrition Examination SurveyUS Centers for Disease Control and Prevention
- India National Family Health Survey (NFHS-5)The DHS Program
- Global Health ObservatoryWorld Health Organization
Found something out of date? Write to contact@brandlabs.app and we will fix it.