Repository navigation
feat(monitoring): Prometheus alerts for API saturation (#8840 G) - #8842
Merged
Merged
Conversation
Add five alert rules to the breeze-api-alerts group, each reading a series
the API already publishes, with promtool unit tests (one fires, one silent
per alert) and the test file wired into the perf-rules workflow.
- ApiDbPoolWaitersHigh: breeze_db_pool_admission_waiting > 20 (warning)
/ > 100 (critical) for 1m
- ApiDbPoolAcquireTimeouts: increase(breeze_db_pool_acquire_timeouts_total[5m]) > 0,
the counter that ticked on 2026-10-15
- ApiEventLoopLagHigh: breeze_nodejs_eventloop_lag_window_max_seconds > 0.5
for 3m (the gauge is a 2m high-water mark, so 3m outlasts one blip)
- ApiAdmissionDegraded: breeze_admission_level >= 2 for 2m
- ApiAdmissionShedding: increase(breeze_admission_decisions_total{outcome="shed"}[5m]) > 50
AgentWsMassClose is a commented placeholder: no WS close-by-reason counter
exists on main; #8838 plans breeze_agent_ws_closed_total{reason}.
Part of #8840
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Deploying breeze with
|
| Latest commit: |
68e46a7
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://e0f97e46.breeze-9te.pages.dev |
| Branch Preview URL: | https://feat-8840-saturation-alerts.breeze-9te.pages.dev |
perfWorkflowContract.test.ts asserts the image tag, 'test rules' and the capacity-kpis test file on one line; the saturation test file continues on the next line. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The level gauge reaches 2 in observe mode (the default) while decide() admits everything, so "shedding standard traffic" was false there. Say degraded, and point at ApiAdmissionShedding for actual refusals. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
github-merge-queue
Bot
removed this pull request from the merge queue due to failed status checks
Oct 16, 2026
JonathanPitre
pushed a commit
to JonathanPitre/breeze
that referenced
this pull request
Oct 18, 2026
…#8840) (LanternOps#9212) ## Summary Implementation plan for the LanternOps#8840 defensive layers. The goal: one tenant's bad network or workload degrades that tenant, not the shared API. It is docs only, with no code. Plan: `docs/superpowers/plans/platform-ci/2026-10-18-tenant-isolation-bulkheads.md` Refs LanternOps#8840 Refs LanternOps#8838 Refs LanternOps#8839 Refs LanternOps#8841 ## What it plans | Task / PR | Protection | |---|---| | 0 (ops) | Soak, then set `BREEZE_ADMISSION_MODE=enforce` (W08 is already deployed in observe) | | A | **Pool lanes + interactive reserve** inside the existing permit gate (no second pool). Lanes are `interactive`, `agent` and `background`, set by the authenticating middleware through a leaf AsyncLocalStorage. The reserve is ~20 % of `DB_POOL_MAX` (35 → 7). A contract test keeps lane coverage of the auth middlewares exhaustive | | B | **Agent-lane tenant fair share**: round-robin across orgs, with a per-org ceiling (default 50 % of shared capacity) that applies only under contention, so a sole tenant can still use all of it. A waiter held back only by its own org's ceiling stays out of W08's global pressure signals and gets `429 TENANT_POOL_SHARE_EXCEEDED`, so one org can't escalate admission for everyone | | C, D, E | **Request-long transaction cleanup**, from an audit of all 33 wrapped agent routes. Software ingest stops taking a second permit. Logs and changes move their Redis quota and decode before the context. Sessions, peripherals and message-events side effects, and command-result enqueues, publishes and dispatch, run after commit | | F | **WS close accounting**: an exhaustive close-reason enum counted once per socket by a latch, plus the `AgentWsMassClose` alert left as a TODO in LanternOps#8842 | | G, H | **Site agent-connectivity signal**: Redis only, from authenticated signals with a request denominator. Read by a partner route, an admin route and an AI tool, and shown in an org-record card and an admin page | | J | **Per-org agent error report** from `agent_logs` (receipt time, site-narrowed, cached) | | gated | **Per-site breaker enforcement** (429), designed but not scheduled. Go/no-go after ≥ 7 days of observe data | Each alert ships with the protection it watches, with promtool fire and silent cases. ## Settled by the advisor quorum Claude (author) and Codex (`gpt-6.1-sol`, xhigh, read-only). - **Agreed:** - one pool and one gate, with lanes as permit accounting; - round-robin plus a ceiling; - ALS lanes rather than a lane field on `DbAccessContext`; - `503` for global pressure and `429` for tenant throttles; - no wire change. - **Codex amendments adopted:** - ceiling waits are kept out of W08's signals; - the authenticated tenant is carried independently of RLS scope; - detached work loses interactive privilege once its request has settled (a `settled` flag); - agent-origin nested acquisitions are fenced out of the reserve. - **My draft changed to Codex's view:** - the WS lease TTL **stays 30 s** (a healthy subscriber does not prove a publish was delivered); - closes are counted through a once-per-socket latch; - breaker **enforcement is gated on evidence**. LanternOps#8850 removed the proven DB-pin mechanism, and auth failures must never trip a site, because agent ids are attacker-chosen. ## Invariants called out - **No new table and no migration.** Breaker and connectivity state live in Redis. - **No agent change, so no `MIN_AGENT_VERSION_*`.** Every shipped agent retries `429`/`503` and honours `Retry-After` inside the call. - **Three new env vars:** `DB_POOL_INTERACTIVE_RESERVE`, `DB_POOL_AGENT_TENANT_MAX_SHARE`, `BREEZE_AGENT_CONNECTIVITY_SIGNALS`. Each has a full inventory row. - **No BullMQ job.** The worker contract tests do not fire. `mcp-coverage`, the aiTools contracts and locale parity fire for G, H and J. ## Decisions for review - **Reserve default 20 %.** The lab gate in Task A pins it. - **Tenant ceiling share 0.5.** A starting value, to be tuned from `breeze_db_pool_agent_tenant_ceiling_waiting`. - **UI home for the partner signal.** The plan proposes a card on the organization record page plus a `/admin/agent-connectivity` page. - **Breaker enforcement go/no-go.** Decided after the Task G soak. ## Testing Docs only. An Explore pass checked every path, symbol and line ref the plan cites against `origin/main`: about 110 OK. It found 3 misses, all fixed: - the body-rejection metric's reason is `timeout`, not `stalled`; - the coverage-test regex now catches `authMiddleware` and `helperAuth`; - an unverified test-count baseline was removed. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Todd Hebebrand <todd@lanternops.io> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Five Prometheus alert rules for API saturation in
monitoring/rules/breeze-rules.yml(groupbreeze-api-alerts), withpromtool test rulesunit tests inmonitoring/rules/tests/breeze-saturation-alerts.test.yml(one fires + one silent case per alert) and the test file added to theperf-rulesworkflow.forApiDbPoolWaitersHighbreeze_db_pool_admission_waiting > 20ApiDbPoolWaitersHighbreeze_db_pool_admission_waiting > 100ApiDbPoolAcquireTimeoutsincrease(breeze_db_pool_acquire_timeouts_total[5m]) > 0ApiEventLoopLagHighbreeze_nodejs_eventloop_lag_window_max_seconds > 0.5ApiAdmissionDegradedbreeze_admission_level >= 2ApiAdmissionSheddingincrease(breeze_admission_decisions_total{outcome="shed"}[5m]) > 50Pool and event-loop rules select
job=~"breeze-(api|worker).*"and groupmax/sum by (job, instance), matchingDbPoolPermitsHeldByAbandonedTransactions; admission rules are API-only.Why
On 2026-10-15 a DB-pool stall from one tenant's uploads paged nobody: the waiter gauge climbed, the acquire-timeout counter ticked, sockets closed, and no rule read any of it (#8840 G).
Adjustments against the brief (every metric name verified in
apps/api/src)ApiAdmissionDegradedshed counter: the label isoutcomewith valuesshed/would_shed(services/admission/admissionMetrics.ts), notdecision=~"refuse|shed". Onlyshedcounts;would_shedis observe mode. Split into its own alert,ApiAdmissionShedding, so the level and the volume page separately.ApiEventLoopLagHighusesfor: 3mrather than 2m:_window_max_secondsis a high-water mark over the monitor's 2m retention window (DEFAULT_RETENTION_MS), so one stall holds it up for 2m on its own andfor: 2mwould page on a single blip.breeze_db_pool_acquire_timeouts_totalcarries atimerlabel (on_time/late); summed away in the alert.AgentWsMassClose: no WS close-by-reason counter exists on main (grep breeze_agent_wsfinds onlybreeze_agent_ws_connections). Left a commented placeholder namingbreeze_agent_ws_closed_total{reason}from agentWs: credential lease re-check fails closed on DB timeout and severs healthy sockets fleet-wide (close 4001) #8838.Test
Both
SUCCESSwith local promtool 3.15.0 and with the CI-pinnedprom/prometheus:v2.52.0image. Run againstmain's rules file, exactly the five fires-cases fail and every silent case passes.Part of #8840
🤖 Generated with Claude Code