FG-0104 Adversarial Design Thinking
Description
Apply red team methodology to all system design. Every architecture, every process, every governance structure should be evaluated through the lens of: how could this fail? How could this be misused? What happens when an agent misinterprets instructions? Adversarial design thinking is the discipline of building systems that survive contact with reality. It means designing for failure cases as deliberately as for the expected path.
Implementation
Before shipping any system or process, conduct a threat model. Ask: What is the worst thing an agent could do with this access? What happens if context is lost mid-operation? What if instructions are ambiguous? Build guardrails based on answers, not assumptions. Review guardrails periodically as capabilities change.
Success Indicators
- Systems fail gracefully rather than catastrophically
- Governance rules address actual failure modes observed in practice
- New capabilities are evaluated for risk before deployment
- The team can articulate what could go wrong with any process
Failure Modes
- Paranoia that prevents shipping: adversarial thinking should improve design, not block it
- Designing for theoretical attacks instead of observed failure modes
- Security theater: implementing visible controls that don't address real risks
- Static threat models: conducting adversarial analysis once at design time but never revisiting as agent capabilities and access patterns change