Three eval dimensions: functional, qualitative, safety.
The Brandlabs Discovery Sprint pattern for setting LLM eval targets in weeks 1–3 — before feature thrash eats the timeline.
·8 min read
TL;DR
Score AI products on three dimensions from day one: functional correctness, qualitative usefulness, and safety/compliance behavior. Set numeric targets in the Discovery Sprint so Friday demos and CI gates share one definition of done.
What are the three eval dimensions Brandlabs uses?
Functional asks whether the system did the task. Qualitative asks whether the answer is clear, complete, and fit for the user. Safety asks whether the system refused, escalated, and avoided harmful or non-compliant outputs. Shipping on only one dimension creates demos that fail real users.
Functional examples: correct tool call, valid JSON, grounded citation IDs, right language route, calendar slot booked.
Qualitative examples: tone for a patient vs clinician, completeness of a care-navigation answer, “helpful but not creepy” for sensitive topics.
Safety examples: no fabricated dosages, no PHI echo into logs, refusal on out-of-scope medical advice, escalation when confidence is low.
Precision (our lab eval product) exists because teams keep collapsing these into a single vibe score. Do not.
How do you set targets in a two-week Discovery Sprint?
Pick the top user journeys, write 20–50 fixtures per critical journey, assign dimension weights, and agree pass thresholds before build week four. Document what is in the required CI suite vs nightly. Ambiguous “we’ll know it when we see it” is how week 16 arguments start.
Discovery Sprint outputs we insist on:
- Journey list with owners
- Seed fixtures (including adversarial and empty-retrieval cases)
- Thresholds per dimension
- Decision on LLM-as-judge vs human rubric vs deterministic checks
- Cost budget for eval runs
If the buyer will not pick thresholds, we write a draft and force an explicit accept/reject. Silence is not alignment.
How do the three dimensions show up in Friday demos?
Each Friday scorecard shows deltas on functional, qualitative, and safety — not a single green check. A week that improved fluency but regressed refusals is not a successful week for healthcare AI.
Stakeholders love qualitative wins because they are visible. Safety regressions are quiet until they are catastrophic.
Make the quiet dimension loud: put safety on the first slide of the scorecard, not the appendix.
Frequently asked questions
Can one metric replace three dimensions?
You can roll up a weighted score for dashboards, but keep the underlying dimensions visible. Rollups hide the failure mode you most need to see.
Where does groundedness fit?
Usually functional (did we cite real retrieved evidence?) plus safety (did we answer when retrieval was empty?). Split the checks; do not bury groundedness inside “quality.”