FG-0601 Testing Pyramid
Description
Implement a multi-tiered testing strategy that validates AI-generated output at every level. The Testing Pyramid applies traditional software testing principles to human-AI collaboration: unit-level validation of individual outputs, integration testing of combined outputs, system testing of complete deliverables, and acceptance testing against human expectations. More tests at the base, fewer at the top, comprehensive coverage throughout.
Implementation
Define 3-4 testing tiers appropriate to your domain. Typically: (1) Unit/Atomic: individual output validation, (2) Integration: combined output coherence, (3) System: complete deliverable functionality, (4) Acceptance: meets human expectations. Automate lower tiers. Maintain human review at the acceptance tier. Track test coverage and failure patterns.
Success Indicators
- Testing catches issues before they reach production
- Lower-tier tests run automatically and frequently
- Test failure patterns inform improvements to agent instructions
- Test coverage is comprehensive without being burdensome
Failure Modes
- Only testing at the acceptance level: issues found too late
- Inverted pyramid: manual testing heavy, automated testing light
- Tests that pass but don't validate meaningful properties
- Test maintenance overhead exceeding the value of testing
Sub-Methods 4
The pyramid base: high-volume, fast, automated tests that validate individual outputs in isolation. Structural checks, type validation, constraint enforcement, and format verification. These tests run on every change and catch the majority of issues. Optimize for speed; if atomic tests are slow, they will not run frequently enough.
The pyramid middle: tests that validate how individual outputs combine into larger deliverables. Cross-reference checks, consistency validation across related outputs, and interface contract verification. These catch issues that atomic tests miss: individually correct outputs that are collectively incoherent.
The pyramid upper tier: end-to-end tests that validate complete deliverables as a user would experience them. Navigate full workflows, verify user-facing behavior, and confirm that the assembled product works as intended. Fewer tests than lower tiers, but each validates a critical user path.
The pyramid apex: human judgment applied to deliverables that pass all automated tiers. Some qualities cannot be automated: aesthetic quality, narrative coherence, strategic alignment, and user experience require human evaluation. This tier focuses human attention on what only humans can judge, leaving verified items to automation.