Every postmortem becomes a regression test.
TL;DR
AI incidents repeat when teams write prose postmortems and never encode the failure. Brandlabs requires each postmortem to produce at least one eval fixture and a CI gate so the same failure cannot silently return on the next model or prompt change.
Why do AI failures recur after a “thorough” postmortem?
Because the artifact is a document, not a test. Prompts drift, models upgrade, and retrieval corpora change. Without a fixture that fails when the bug returns, you are relying on memory — the least reliable component in the stack.
Classic postmortems end in action items: “improve prompt,” “add monitoring,” “train the team.” Those help. They do not pin the failure.
For LLM systems, the durable artifact is a case in the suite: inputs, expected behavior, and a scorer. Anything less is a story you will relearn in production.
What is the Brandlabs postmortem → eval loop?
Reproduce the failure, minimize a fixture, add it to the suite, set a gate, then ship the fix. The postmortem is closed only when CI proves the failure mode is covered — not when the Google Doc is polished.
Loop:
- Capture the real input (redacted)
- Write the expected refusal, escalation, or answer shape
- Land the fixture in the repo
- Confirm it fails on main before the fix
- Fix until green
- Keep it in the required suite
This is how “every postmortem becomes a regression test” stops being a slogan.
How do you keep the suite from becoming a junk drawer?
Curate ruthlessly. Merge duplicates, delete flaky cases, and tag by dimension (functional, qualitative, safety). Ownership sits with the pod — not a separate QA team that discovers failures after customers do.
A suite that nobody trusts gets skipped. A suite that blocks deploys for noise gets bypassed. Treat fixtures like production code: review, ownership, and deletion when obsolete.
Frequently asked questions
What if the failure is non-deterministic?
Run multiple trials with a threshold, or constrain temperature/tools for the test path. Non-determinism is a reason to engineer the harness — not a reason to skip coverage.
Should every minor UX complaint become an eval?
No. Encode failures that are unsafe, non-compliant, or repeatedly break user trust. Qualitative nits can live in a larger nightly set.