FG-0803 Eval-Gated Promotion
Description
No new version reaches production on intuition. Eval-Gated Promotion requires every candidate model, agent, prompt, or tool to pass a defined evaluation suite before it replaces the incumbent. The suite is a fixed set of prompts or tasks, scored on the dimensions that matter, run identically against the current production version and each challenger. The version that wins on the metrics advances; the rest are documented with their specific failure modes. This turns version selection into a measured comparison rather than a judgment call.
Implementation
Build a reusable evaluation suite covering the capabilities you depend on, with a scoring rubric for each. Define the promotion bar in advance: which metrics must improve, which must not regress, and by how much. Run the incumbent and every candidate through the same suite under the same conditions. Record results as structured data so comparisons are reproducible. Promote only the candidate that clears the bar, and archive the losers with their scores and failure modes as a decision record.
Success Indicators
- No version reaches production without a recorded eval result against the incumbent
- Promotion decisions trace to specific metric improvements, not impressions
- The eval suite is reused across versions, keeping comparisons consistent over time
- Regressions are caught at the gate rather than discovered in production
Failure Modes
- Evaluating on a suite that does not reflect real production use, so the gate passes versions that fail in the field
- Moving the promotion bar to fit a favored candidate instead of holding it fixed
- Scoring only the dimensions that improved while ignoring the ones that regressed
- Running the eval once and trusting it indefinitely as the workload and expectations drift
Sub-Methods 3
Define the pass criteria before running the evaluation: the metrics that must improve, the metrics that must not regress, and the minimum margin that counts as a real difference. Fixing the bar in advance prevents the bias of rationalizing whichever candidate you already preferred.
Always score the current production version on the same suite, in the same run, as every challenger. A candidate's absolute score is meaningless without the incumbent as the baseline. The comparison is the decision; the absolute number is not.
Every promotion produces a record: the suite version, the scores for each candidate, the bar that was applied, and why the winner won. Losers are documented with their specific failure modes. The record makes the selection auditable and informs the next round.