Repository navigation
perf(dash): normalize the request body once for lifecycle signals - #500
Conversation
…y once Default --dashboard-lifecycle-signals to true (DASHBOARD_LIFECYCLE_SIGNALS=0 turns it off). The lifecycle scan now reuses the wait-cause normalization instead of running its own. Signed-off-by: Osher-Elhadad <Osher.Elhadad@ibm.com>
|
Review of the lifecycle default-on change. gofmt and vet clean; the flag test (default on, off with =false or =0, through the real main), the waitcause tests, and the proxy and dash lifecycle, wait-cause, retention, cascade and quota tests pass; attribution clean. Checked: (1) byte-identical behaviour: both consumers read one lazily computed normalization on the same writer goroutine (WaitCause runs first, then LifecycleFn, dash/capture.go), neither modifies the slice (no element assignment, append or sort on it in waitcause.observe or scanLifecycle), a panic in the first consumer leaves the memo unset so the second recomputes and recovers separately, and previous_response_id bodies skip normalization in both; the equivalence test compares the shared result with the old two-pass one on Anthropic, OpenAI chat, Responses, previous_response_id, no-tool-result and garbage bodies. (2) Flag-off path: LifecycleOn false registers no LifecycleFn, so only the wait-cause consumer normalizes, as before; TestLifecycleOffWritesNoRows covers it. (3) Cascade and quota: request_lifecycle rows are deleted with their request (ON DELETE CASCADE) and are not counted by the per-tenant row cap, both pinned by a test; the table is additive, and the upgrade test shows a database without it gets it with request rows intact. (4) Docs match the new default. Non-blocking: the public PR text quotes what 'production's writer logged' (dropped_total) over 24 hours; that is an operational fact about a deployment and belongs in the private notes, so drop that sentence. The benchmark table is fine. Default-on also means one extra row per keep-alive ping on every dashboard-enabled deployment; the PR says so in the storage line, but please also state it in the release note. Approved; ready-to-merge. |
Signed-off-by: Osher-Elhadad <Osher.Elhadad@ibm.com> # Conflicts: # cmd/context-guru-proxy/main.go # docs/reference/reference.md
OsherElhadad
left a comment
There was a problem hiding this comment.
Review: APPROVE at aa0c1ad. (Posted as a comment: GitHub does not let the author account approve.)
- Merge aa0c1ad. Its conflicts were the lifecycle flag's help text in
cmd/context-guru-proxy/main.goanddocs/reference/reference.md, both resolved to main, which already has the default-on flag. Nothing from main is reverted. The PR now carries the normalize-once change plus tests. - Change.
apply.NormalizeMessagesruns once per request, lazily, and is shared byscanLifecycleandwaitcause.ObserveWith. Both closures run one after the other on the dashboard writer goroutine (dash/capture.go), so the unsynchronised memo is safe. TestSharedNormalizationIsByteIdentical checks the lifecycle scan equals the old self-normalizing scan and that the shared closure is called at most once. - Checks.
gofmtandgo vetare clean.go testpasses for./proxy,./dash,./internal/waitcauseand./cmd/.... CI is green. Both commits are signed off with no AI attribution.
Non-blocking, tests:
- The waitcause half of the sharing is unpinned. Making
observeignorenormand normalize as Anthropic still passes every test. A case that runsObserveWithon an OpenAI-shaped body againstObservewould close that. - The production memo in
dashcapture.gois untested; removing it changes only cost.
amiddavid
left a comment
There was a problem hiding this comment.
Review round 1
Verdict: approve.
@OsherElhadad: no action needed. Optional points are at the end.
What I checked
- Merge: the PR merges cleanly onto current
origin/main(67c5f75). That includes #520, #524 and #527. No conflicts. The PR already holds a merge of main at 0ea12b6. - Same results: the shared normalization gives the same results as before.
- Only two callers use the normalized messages:
waitcause.observeandscanLifecycle. - Neither one writes to the slice or to its entries.
- The writer goroutine runs
WaitCausefirst andLifecycleFnsecond, one after the other. The lazy memo infinish()has no data race. - Both callers still read
waitBodyforprevious_response_idandis_error. Nothing needs the raw bytes after normalization.
- Only two callers use the normalized messages:
- Fail open: both
Observe*andscanLifecyclekeep theirrecover(). Ifnormpanics,normDonestays false, so the other caller retries. A panic in one scan does not block the other or the request. previous_response_id:normis not called. The test pins this.
Tests (eval box 1, merged tree)
go vetoninternal/waitcause,proxy,dash,cmd/...: clean.go test -race:internal/waitcauseok,cmd/...ok (24s),dash(Lifecycle, WaitCause) ok.go test -race ./proxy -run 'Lifecycle|SharedNorm|WaitCause|Capture': ok (80s). I did not run the full./proxypackage.
Benchmark (30 iterations x 3, 300-turn body, about 1.2 MB)
Twice = what main does today. Once = this PR.
| time/op | B/op | allocs/op | |
|---|---|---|---|
| main (Twice) | 35.2 - 36.9 ms | 12.4 MB | 31,220 |
| PR (Once) | 20.2 - 20.7 ms | 7.1 MB | 15,937 |
- -43% time, -43% bytes, -49% allocs. The claim is real.
- This runs on the writer goroutine, not the request path. The gain is writer throughput and less GC.
Mutations
waitcause: drop the last entry afternorm()-> caught (TestOpenTeammateWait,TestTeammateReplyClosesAnOpenWaitAndWakesTheThread,TestSharedNormalizationIsByteIdentical).scanLifecycle: empty the message slice -> caught (TestSharedNormalizationIsByteIdentical,TestScanLifecycle,TestLifecycleSignalsEndToEnd).- Remove the memo in
finish()(if !normDone || true): my run was contaminated by a leftover mutation, so I have no valid result. No test counts normalizations across the two closures. OnlyObserveis checked for "norm at most once".
Non-blocking
- Add a test that pins "one normalization per request" across the two closures in
finish(). A call counter onnormwould do. Today a regression there only shows in the benchmark. lifecycleflag_test.goanddash/lifecycle_retention_test.goare new tests that do not relate to the perf change. They pass. A separate PR would be cleaner.- The benchmark file keeps
scanLifecycleOldinsidelifecycle_equiv_test.go. Good as a reference. Add a short note that it must not be "fixed" to match new code.
What
The dashboard writer normalizes each request body once.
scanLifecycleused to callapply.NormalizeMessagesa second time after the wait-cause code (#453) had already done so; both now read one lazily-computed normalization (Tracker.ObserveWith). The tool_resultis_errorread, which the normalizer drops, is unchanged.The default-on change for
--dashboard-lifecycle-signalsis already on main (331c6ab); this PR no longer touches the flag.Cost
Benchmark on a ~1.2 MB (~300K-token) Anthropic transcript, 300 tool turns, writer-path scans (wait-cause + lifecycle),
go test -bench WriterBodyScans -benchtime 20x -count 3, shared loaded host:About 45% less time and 49% fewer allocations on the writer goroutine.
Tests
previous_response_id, no-tool-result and garbage bodies; wait-cause output is identical and normalization runs at most once.=false/=0) kept.