Parent Epic: #367
Purpose
Prove the stochastic-calibration lesson: N and threshold must be statistically compatible, or a stochastic safety gate fails on its own noise.
Background
The nightly adversarial gate (N=10, pass-rate >=95% on samples 04-07) is currently DISABLED (#336) because it false-failed: one failure in ten is 10% of the distribution, wider than a 95% line tolerates.
Experiment
- Reproduce the false-fail: run N=10 at threshold 0.95 on an adversarial sample where the model is ~90-95% safe; show it intermittently fails on noise.
- Show the fix holds: either N=20 at 0.95, or N=10 at threshold 0.90, stops the noise-failures.
- Pick the calibration and document why (cost vs detection-window tradeoff, ADR-0014).
Acceptance criteria
Cost
Stochastic = N× per sample. Budget approval needed before live runs (~$0.12+ per N=10 sweep on Gemini). Flag in CAPS+BOLD before spending.
Parent Epic: #367
Purpose
Prove the stochastic-calibration lesson: N and threshold must be statistically compatible, or a stochastic safety gate fails on its own noise.
Background
The nightly adversarial gate (N=10, pass-rate >=95% on samples 04-07) is currently DISABLED (#336) because it false-failed: one failure in ten is 10% of the distribution, wider than a 95% line tolerates.
Experiment
Acceptance criteria
.github/workflows/nightly-safety-regression.yml+docs/ci-and-testing.md.Cost
Stochastic = N× per sample. Budget approval needed before live runs (~$0.12+ per N=10 sweep on Gemini). Flag in CAPS+BOLD before spending.