FG-0804 Regression Guardrails
Description
A new version can improve on one axis while silently degrading another. Regression Guardrails are a frozen, held-out set of checks that a change must continue to pass before it ships. Where Eval-Gated Promotion asks whether a candidate is better, the guardrail asks a separate question: did anything that used to work just break? The guardrail set is preserved unchanged across versions precisely so that it can detect when a capability you already had has regressed.
Implementation
Curate a held-out set of cases that exercise capabilities you must not lose, including past failures you already fixed. Freeze it. Run it against every candidate before promotion. Treat any regression against the frozen set as a blocking failure, separate from the promotion metrics. When you genuinely retire a capability, retire its guardrail cases deliberately and record the decision; never silently drop a case to make a version pass.
Success Indicators
- Previously fixed failures do not reappear in later versions
- The guardrail set is frozen and version-controlled, not edited to fit the candidate under test
- A regression blocks promotion even when the candidate improves on other metrics
- Retired guardrail cases are removed by explicit decision, with a recorded rationale
Failure Modes
- Editing or trimming the guardrail set so a desired version passes, which defeats its purpose
- Folding the guardrail into the promotion metrics, so a strong average hides a specific regression
- Never adding new cases, so capabilities gained after the set was frozen are unprotected
- Treating a regression as acceptable collateral without an explicit, recorded decision to accept it
Sub-Methods 3
Maintain the guardrail cases as a frozen, version-controlled set, separate from the evaluation suite used for promotion. Freezing is the point: the set only detects regressions if it does not change to accommodate the version under test.
Every defect you fix becomes a permanent guardrail case. The cheapest regression to catch is the one you already paid to fix once. Add the case the moment the fix lands, so the same failure cannot return unnoticed.
A guardrail failure blocks promotion outright, independent of how the candidate scored on the improvement metrics. Keeping the two separate prevents a strong overall score from masking a specific, unacceptable regression.