[ToddHebebrand]

Summary

• The admission level now rises only under sustained pressure. A level rises after BREEZE_ADMISSION_RISE_TICKS (default 3) consecutive ticks of fresh evidence above its threshold. A single blip never raises it. The emergency path (BREEZE_ADMISSION_EMERGENCY_MS) is unchanged and still immediate.
• Level-1 thresholds go up to lag 200 ms / DB wait 150 ms (was 100 / 75). Level 2 and 3 are unchanged.
• The level-1 hold is shorter: BREEZE_ADMISSION_LEVEL1_HOLD_MS (default 5 s, was the shared 15 s). Levels 2 and 3 keep 15 s.
• New counter breeze_admission_rise_suppressed_total{to,trigger} counts each rise the gate held back. would_shed / shed semantics are unchanged, so the before/after comparison the issue asks for stays valid.

Root cause (why a tick gate alone would not have fixed it)

The controller already had a 2-tick rise gate (ADMISSION_ESCALATE_TICKS = 2). Single blips still got through because both inputs are smoothed over more than one tick:

• DB: the pool gate's grant-wait p95 used a 3 s window. At idle there are few grants, so p95 = the one slow grant. That grant was then re-read on three consecutive ticks, which matches the issue's trigger=db, dbWait=110ms, dbWaiting=0, dbInUse=0. Raising the gate to 3 ticks would still have let every one of these through. The new test a single slow pool grant at idle never raises the level uses a real gate and the real sampler. It fails with the 3 s window even at riseTicks=3 (mutation-checked).
• Lag: the median of three 1 s monitor samples moves when a blip lands in two adjacent samples. The monitor and the controller run on unaligned 1 s timers, so a tick that reads no new sample repeats the previous median. That repeat used to count as a second tick. This explains the issue's lag=251ms, peakLag=21..31ms: the median was high while this tick's newest sample was low.

Fix: the grant-wait window is now 1.5 ticks (POOL_WAIT_STATS_WINDOW_MS 3 s → 1.5 s). A grant is read on at most two ticks, fewer than the 3-tick gate, and the extra half tick covers late ticks, so grants are never dropped between ticks. The sampler now reports lagFresh. A tick whose only elevated reading is a stale lag median holds the streak: it neither advances nor resets it. Smoothing is now the rise gate's job, and the gate counts real evidence.

Decisions made on the brief (please check)

1. Level-1 thresholds are 200 / 150, not 250 / 200. 250 ms lag and 200 ms DB wait are exactly the unchanged level-2 entries. The config validator requires strictly increasing thresholds, so 250,250,600 would be rejected. It would also make level 1 unreachable, because every rise would go straight to shedding standard traffic. 200 / 150 is twice the old entry and the highest round values that still leave a level-1 band. To get the literal numbers, raise level 2 too, for example BREEZE_ADMISSION_LAG_MS=250,400,600 and BREEZE_ADMISSION_DB_WAIT_MS=200,350,600.
2. Level-1 hold is 5 s. The old level-1 dwell was already 15 s, so "e.g. 15 s" would not shorten it. 5 s works because each shed bulk request already carries a 120–180 s Retry-After, and W09 agents honour it for the whole loop. The shed load stays away for minutes whatever the level does. Under real pressure the signal stays above the 60 % exit line, so the hold never starts. 5 s is longer than the 3-tick rise, so a borderline signal cycles 0 → 1 → 0 at most about once every 9 s.
3. The streak's trigger is now the trigger of the lowest target in the streak, the level it actually enters. Before, it was the last tick's trigger.
4. A recent DB timeout now needs 3 ticks to pin level 2, where before it needed 2. The 30 s pin window means it cannot be missed, but this is a behavior change for a hard-failure signal. If you want that path to be immediate, like the emergency path, it is a one-line follow-up.

Linked Issues

Closes #9353
Refs #8139

Merge Danger

Door: two-way. Every default can be overridden by env. Setting BREEZE_ADMISSION_LAG_MS=100,250,600 BREEZE_ADMISSION_DB_WAIT_MS=75,200,600 BREEZE_ADMISSION_RISE_TICKS=2 BREEZE_ADMISSION_LEVEL1_HOLD_MS=15000 restores the old controller, except for the 1.5 s grant-wait window and the stale-lag hold. A revert restores everything.

Blast radius: agent bulk/standard admission on every API instance. In observe mode, which both regions run today, it changes counters only. Under enforce it decides which agent uploads get 503. Shedding now starts ~3 s into sustained pressure instead of ~2 s. Stalls ≥ 1 s still go straight to level 3.

Prod re-check (before flipping enforce)

After this deploys, run on both regions:
increase(breeze_admission_decisions_total{outcome="would_shed"}[1h])
The target is near zero at idle. Also check increase(breeze_admission_rise_suppressed_total[1h]) to see how many blips the gate absorbed. enforce stays off until both look right.

Type of Change

• Bug fix

Testing

Red first: 40 new and updated assertions failed against main before the implementation.

• cd apps/api && npx vitest run src/services/admission src/middleware/agentAdmission src/db/poolAdmission.test.ts src/routes/agentWs.test.ts src/system/connections src/config: 66 files / 2045 tests passed.
• Required cases, each a runtime test with a fake clock and sample feed: (a) one 300 ms lag sample at idle → level 0, suppressed counter +1; (b) 3 consecutive → level 1; (c) emergency still immediate; (d) level 1 decays within the 5 s hold; (e) old vs new config on the issue's trace: old rises to 2 and counts ≥ 16 bulk would_shed, new stays at 0 with 0 would_shed.
• Review follow-up tests cover a 1 → 2 rise under the gate, a stale-lag hold at level 1, the lowest-level trigger in mixed streaks, an emergency clearing a streak, a stale-only first tick, and a non-default level1HoldMs.
• Mutation checks: putting back the 3 s window turns the slow-grant test red. Ignoring lagFresh turns the 3 stale-lag tests red.
• Integration (vitest.integration.config.ts, real pg on a per-worktree test stack): admissionControl.integration.test.ts 3/3 passed, re-run after the review fixes. Its lag scenario was raised from 150 ms to 230 ms blocking slices, because 150 ms no longer reaches the new level-1 entry. The 150 ms version was confirmed to time out under the new defaults. The test stack was torn down afterwards.
• Typecheck: pnpm exec tsc --build apps/api/tsconfig.tests.json exit 0, 0 errors. Plain tsc --noEmit ran out of memory at 12 GB (exit 134), the known trap, so this build-mode config is what CI runs.
• Env plumbing: both compose files, .env.example, internalEnvVars.ts and the docs env table are updated. The env inventory and compose-coverage suites are green (included in the run above).

Checklist

• My code follows the project's code style
• I have reviewed my own changes
• I have tested on the relevant platforms
• No secrets, credentials, or personal data included
• Settings rule 9: N/A (no settings page)

🤖 Generated with Claude Code

[cloudflare-workers-and-pages]

Deploying breeze with    Cloudflare Pages

Latest commit:	27e506b
Status:	✅  Deploy successful!
Preview URL:	https://8b7ddda4.breeze-9te.pages.dev
Branch Preview URL:	https://fix-9353-admission-level-hys.breeze-9te.pages.dev

View logs

[ToddHebebrand]

Review run: /pr-review-toolkit:review-pr with code-reviewer, pr-test-analyzer and comment-analyzer, run in parallel. silent-failure-hunter and type-design were skipped: no error-handling changes, and only small additive types.

Findings: 0 critical, 0 important. 3 code suggestions, 8 test gaps and 6 comment/doc items were raised. All of them were addressed in 27e506b except the ones below.

• Fixed: the grant-wait window was exactly one tick wide, so grants landing between a late tick and the one before it were never read. It is now 1.5 s: a grant is read on at most two ticks, fewer than the 3-tick gate.
• Fixed: added 6 controller tests:
  • a 1 → 2 rise under the gate;
  • a stale-lag hold at level 1;
  • the lowest-level trigger in mixed streaks;
  • an emergency clears an unfinished streak;
  • a stale-only first tick is not reported as suppressed;
  • a non-default level1HoldMs.
• Fixed: the Retry-After rationale now names the /logs 2–4 s exception and the shipped-vs-W09 agent behaviour. Also corrected: the wording of the bounds claim, the integration stall comment, and the 60 % exit wording in the env docs. The new counter is added to the monitoring metrics table.
• Documented, not changed: a recent DB timeout now needs 3 ticks to pin level 2 (it was 2). This is listed in "Decisions" item 4.
• Not changed: emergency- and reset()-discarded streaks are not counted as suppressed, which is intentional because the level rose. A stale tick restarting a recovery dwell is intentional, since the prior fresh tick read the same elevated median. Banner log-line assertion skipped: low value.

Tests:

• npx vitest run src/services/admission src/middleware/agentAdmission src/db/poolAdmission.test.ts src/routes/agentWs.test.ts src/system/connections src/config: 66 files / 2045 tests passed.
• Integration admissionControl.integration.test.ts on a per-worktree pg stack: 3/3 passed, re-run after the fixes.
• pnpm exec tsc --build apps/api/tsconfig.tests.json: exit 0, 0 errors.

Status: review-clean, awaiting maintainer merge. Before flipping enforce, re-check increase(breeze_admission_decisions_total{outcome="would_shed"}[1h]) on both regions after deploy.