CI that blocks deploys on failing evals.
Sampling, cost caps, flake handling, and which LLM eval suites must stay required vs nightly — so production cannot silently regress.
·8 min read
TL;DR
Treat critical eval suites like unit tests: they run on every commit or deploy pipeline and fail the build when thresholds break. Keep a larger nightly suite for coverage. Cap cost, quarantine flakes, and never allow “skip evals” without a written exception.
Which evals belong in the required CI gate?
Put high-severity, fast, stable fixtures in the required gate: safety refusals, schema/tool correctness, empty-retrieval behavior, and top journey functional checks. Park broad qualitative sweeps and expensive judge runs in nightly or pre-release jobs.
Required gate traits:
- Minutes, not hours
- Low flake rate
- Clear owner
- Tied to user harm or contract thresholds
Nightly traits:
- Larger sampling
- Multilingual sweeps
- Model-judge ensembles
- Costly corpus diffs
If everything is required, nothing is. If nothing is required, demos lie.
How do you control cost and flakiness?
Budget tokens per pipeline, cache embeddings where safe, pin model versions for gates, retry narrowly, and quarantine known flakes with tickets. Non-determinism is a harness design problem — not a reason to disable the gate.
Tactics that work:
- Smaller models for deterministic checks; reserve frontier models for product path
- Seeded sampling and fixed temperatures on gate runs
- Separate “flake” label from “fail”
- Cost dashboards next to the scorecard
A gate that randomly fails trains the team to ignore it. Fix the harness.
What is the exception process when someone needs to ship anyway?
Written exception with owner, expiry, and compensating control — same spirit as skipping a security scan. Silent `--no-evals` in a hotfix script is how regulated products earn incidents.
Brandlabs exception template:
- Why the gate is blocked
- Risk accepted
- Who approves
- When the fixture must be green again
- Link to follow-up issue
Postmortems that lack a new fixture do not close.
Frequently asked questions
Should evals run on every PR or only main?
Run the required suite on PRs that touch prompts, tools, retrieval, or model config. You can lighten runs for pure CSS changes — with path filters, not honor-system skips.
What if the vendor model changed under us?
Pin versions when possible. When the vendor forces a move, treat it like a dependency upgrade: re-run full suites before promoting.