FG-0806 Corpus Curation
Description
The improvement corpus is the accumulated body of structured data the system learns from. Left to grow on its own, it collects duplicates, contradictions, and low-value entries that degrade the signal faster than volume improves it. Corpus Curation treats the corpus as a maintained asset: deduplicated, quality-gated at intake, versioned, and pruned. The quality of what the system learns is bounded by the quality of the corpus, so curation is the difference between a flywheel that compounds and one that drifts.
Implementation
Define a quality gate that every entry must pass before it enters the corpus: format validity, source attribution, and a minimum quality bar. Deduplicate on intake and on a schedule. Version the corpus so any model or capability can be traced to the exact data it was built from. Prune entries that are stale, superseded, or shown to degrade downstream results. Measure corpus quality alongside size, and treat a growing corpus with falling quality as a regression.
Success Indicators
- Every corpus entry passes a defined quality gate before intake
- Duplicates and contradictions are detected and resolved rather than accumulated
- The corpus is versioned, so any trained capability is traceable to the exact data behind it
- Corpus quality is measured, and growth never comes at the cost of declining signal
Failure Modes
- Accumulating volume without curation, so noise grows faster than signal
- No intake gate, so malformed or low-value entries dilute the corpus
- An unversioned corpus, making it impossible to trace a capability to its data or roll back a bad batch
- Never pruning, so stale and superseded entries keep shaping training long after they should
Sub-Methods 3
Every candidate entry passes a defined gate before it joins the corpus: valid format, attributed source, and a minimum quality bar. The gate is cheaper than the cleanup, and an ungated corpus degrades the moment volume outpaces attention.
Detect duplicate and contradictory entries on intake and on a schedule. Duplicates skew the signal toward whatever is repeated; contradictions teach the system both sides of a settled question. Resolve them rather than letting them accumulate.
Version the corpus so that any model or capability can be traced to the exact data it was built from. Versioning makes results reproducible and makes it possible to roll back a bad batch instead of contaminating every version that follows.