LLM evaluation tools compared for health AI teams
Promptfoo, DeepEval, Inspect, Langfuse, Arize Phoenix and Braintrust: licences, strengths and which fits a regulated health product.
- Global
Key takeaways
- Promptfoo and Inspect are MIT-licensed; DeepEval is Apache-2.0.
- Inspect was built by the UK AI Security Institute for research-grade evaluations.
- Langfuse is mostly MIT and self-hostable; some enterprise admin features need a licence key.
- Arize Phoenix is source-available under the Elastic License 2.0, not an OSI-approved open-source licence.
- The tool matters less than the test set: health scenarios, languages and rubrics written with clinicians.
Which LLM evaluation tools do teams use?
| Tool | Licence | Best at | Self-host |
|---|---|---|---|
| Promptfoo | MIT | CLI and CI tests, red-teaming | Yes |
| DeepEval | Apache-2.0 | Python test framework with ready-made metrics | Yes |
| Inspect | MIT | Research-grade evaluations, agents | Yes |
| Langfuse | Mostly MIT | Tracing, prompt management, production scoring | Yes |
| Arize Phoenix | Elastic License 2.0 | Tracing and evaluation in notebooks and production | Yes |
| Braintrust | Closed platform | Hosted experiments and review workflows | No |
What does a health AI eval setup need?
Three layers. Functional tests check that features work. Quality tests score answers against rubrics. Safety tests probe crisis handling, professional claims, medical accuracy, minors and data leakage, in every language you support. Each layer runs in CI and blocks a release when a safety score drops.
Code-first framework or platform?
Code-first frameworks keep tests next to your code and data on your machines, which suits regulated teams. Platforms add dashboards, human review queues and production traces. Many teams use both: a framework in CI and an observability tool in production.
What matters more than the tool?
The test set. Write scenarios from real user questions, add adversarial cases, get clinicians to define what a safe answer looks like, run each scenario several times, and keep failing cases as regression tests forever.
Frequently asked questions
Which eval tool should a small health startup start with?
A code-first framework you can run in CI, such as Promptfoo or DeepEval, plus a spreadsheet-sized test set written with a clinician. Add tracing and production monitoring once real users arrive.
Can we use hosted eval platforms with patient data?
Only if the vendor's contracts, hosting region and retention meet your obligations, such as a business associate agreement under HIPAA. Many teams de-identify test data or self-host instead.
Sources
- promptfooGitHub
- DeepEvalConfident AI on GitHub
- Inspect AIUK AI Security Institute
- LangfuseGitHub
- PhoenixArize AI on GitHub
- LLM and agent evaluation platforms: 2026 comparisonArize AI · 2026
Found something out of date? Write to contact@brandlabs.app and we will fix it.