Building a prompt regression dataset that ages well.
Golden sets, adversarial sets, and ownership so your LLM suite still means something after six months of prompt thrash.
·8 min read
TL;DR
A regression dataset dies when nobody owns it. Brandlabs splits golden journeys, adversarial safety cases, and production-derived fixtures — with labels, owners, and deletion rules — so CI stays trustworthy instead of becoming a junk drawer.
What belongs in a durable LLM regression dataset?
Three layers: golden cases for core journeys, adversarial cases for safety and abuse, and production-mined cases from real failures (redacted). Each row needs expected behavior, dimension tags, and an owner — not just an input string.
Minimum fields:
id,locale,journeyinput(redacted)expectedbehavior class or reference outputdimensions: functional / qualitative / safetyseverityandsuite: required vs nightlyownerandadded_from(ticket / postmortem)
If you cannot explain why a case exists, delete it.
How do you stop the suite from rotting?
Review quarterly, merge duplicates, quarantine flakes, and require a fixture for every severity-1 incident. Prompt changes that break goldens must update expectations deliberately — silent edits hide product drift.
Operating rules:
- Flakes get a ticket or get removed
- Goldens for retired features get archived
- Adversarial set grows from red-team notes
- Multilingual products keep locale parity
Precision-style eval products help teams share agents and datasets — the discipline still has to live in your repo and CI.
Who should own the dataset in an embedded pod?
The forward-deployed engineer owns the required suite; the client product owner approves threshold changes. Shared ownership without a name on the folder is how suites go stale.
In Brandlabs engagements, dataset ownership is written into the Foundation exit gate. No owner, no “green” claim on Friday.
Frequently asked questions
How big should the first dataset be?
Dozens of high-signal cases beat thousands of paraphrases. Grow from incidents and top journeys.
Can we store production prompts as-is?
Only in a controlled store with redaction and access rules. Default to minimized fixtures in the eng repo.