Skip to content

perf(dash): normalize the request body once for lifecycle signals - #500

Merged
amiddavid merged 2 commits into
mainfrom
feat/lifecycle-signals-default-on
Oct 15, 2026
Merged

amiddavid merged 2 commits into
mainfrom
feat/lifecycle-signals-default-on

Conversation

@OsherElhadad

@OsherElhadad OsherElhadad commented Oct 12, 2026 •

Copy link
Copy Markdown
Collaborator

What

The dashboard writer normalizes each request body once. scanLifecycle used to call apply.NormalizeMessages a second time after the wait-cause code (#453) had already done so; both now read one lazily-computed normalization (Tracker.ObserveWith). The tool_result is_error read, which the normalizer drops, is unchanged.

The default-on change for --dashboard-lifecycle-signals is already on main (331c6ab); this PR no longer touches the flag.

Cost

Benchmark on a ~1.2 MB (~300K-token) Anthropic transcript, 300 tool turns, writer-path scans (wait-cause + lifecycle), go test -bench WriterBodyScans -benchtime 20x -count 3, shared loaded host:

ns/op B/op allocs/op
normalize twice (before) 49.6-58.3 M 12.53 M 31,229
normalize once (after) 29.3-31.5 M 7.22-7.27 M 15,946-15,950

About 45% less time and 49% fewer allocations on the writer goroutine.

Tests

  • Rows from the shared normalization equal the old two-pass result on Anthropic, OpenAI chat, Responses, previous_response_id, no-tool-result and garbage bodies; wait-cause output is identical and normalization runs at most once.
  • Flag default test (on, off with =false/=0) kept.

…y once

Default --dashboard-lifecycle-signals to true (DASHBOARD_LIFECYCLE_SIGNALS=0 turns it off).
The lifecycle scan now reuses the wait-cause normalization instead of running its own.

Signed-off-by: Osher-Elhadad <Osher.Elhadad@ibm.com>
@OsherElhadad

Copy link
Copy Markdown
Collaborator Author

Review of the lifecycle default-on change. gofmt and vet clean; the flag test (default on, off with =false or =0, through the real main), the waitcause tests, and the proxy and dash lifecycle, wait-cause, retention, cascade and quota tests pass; attribution clean.

Checked: (1) byte-identical behaviour: both consumers read one lazily computed normalization on the same writer goroutine (WaitCause runs first, then LifecycleFn, dash/capture.go), neither modifies the slice (no element assignment, append or sort on it in waitcause.observe or scanLifecycle), a panic in the first consumer leaves the memo unset so the second recomputes and recovers separately, and previous_response_id bodies skip normalization in both; the equivalence test compares the shared result with the old two-pass one on Anthropic, OpenAI chat, Responses, previous_response_id, no-tool-result and garbage bodies. (2) Flag-off path: LifecycleOn false registers no LifecycleFn, so only the wait-cause consumer normalizes, as before; TestLifecycleOffWritesNoRows covers it. (3) Cascade and quota: request_lifecycle rows are deleted with their request (ON DELETE CASCADE) and are not counted by the per-tenant row cap, both pinned by a test; the table is additive, and the upgrade test shows a database without it gets it with request rows intact. (4) Docs match the new default.

Non-blocking: the public PR text quotes what 'production's writer logged' (dropped_total) over 24 hours; that is an operational fact about a deployment and belongs in the private notes, so drop that sentence. The benchmark table is fine. Default-on also means one extra row per keep-alive ping on every dashboard-enabled deployment; the PR says so in the storage line, but please also state it in the release note. Approved; ready-to-merge.

@OsherElhadad OsherElhadad added the ready-to-merge Program PR approved by the reviewer, CI green; needs maintainer approval label Oct 12, 2026
Signed-off-by: Osher-Elhadad <Osher.Elhadad@ibm.com>

# Conflicts:
#	cmd/context-guru-proxy/main.go
#	docs/reference/reference.md
@OsherElhadad OsherElhadad changed the title feat(dash): lifecycle signals on by default, normalize the body once perf(dash): normalize the request body once for lifecycle signals Oct 14, 2026

@OsherElhadad OsherElhadad left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review: APPROVE at aa0c1ad. (Posted as a comment: GitHub does not let the author account approve.)

  • Merge aa0c1ad. Its conflicts were the lifecycle flag's help text in cmd/context-guru-proxy/main.go and docs/reference/reference.md, both resolved to main, which already has the default-on flag. Nothing from main is reverted. The PR now carries the normalize-once change plus tests.
  • Change. apply.NormalizeMessages runs once per request, lazily, and is shared by scanLifecycle and waitcause.ObserveWith. Both closures run one after the other on the dashboard writer goroutine (dash/capture.go), so the unsynchronised memo is safe. TestSharedNormalizationIsByteIdentical checks the lifecycle scan equals the old self-normalizing scan and that the shared closure is called at most once.
  • Checks. gofmt and go vet are clean. go test passes for ./proxy, ./dash, ./internal/waitcause and ./cmd/.... CI is green. Both commits are signed off with no AI attribution.

Non-blocking, tests:

  • The waitcause half of the sharing is unpinned. Making observe ignore norm and normalize as Anthropic still passes every test. A case that runs ObserveWith on an OpenAI-shaped body against Observe would close that.
  • The production memo in dashcapture.go is untested; removing it changes only cost.

@amiddavid amiddavid left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review round 1

Verdict: approve.

@OsherElhadad: no action needed. Optional points are at the end.

What I checked

  • Merge: the PR merges cleanly onto current origin/main (67c5f75). That includes #520, #524 and #527. No conflicts. The PR already holds a merge of main at 0ea12b6.
  • Same results: the shared normalization gives the same results as before.
    • Only two callers use the normalized messages: waitcause.observe and scanLifecycle.
    • Neither one writes to the slice or to its entries.
    • The writer goroutine runs WaitCause first and LifecycleFn second, one after the other. The lazy memo in finish() has no data race.
    • Both callers still read waitBody for previous_response_id and is_error. Nothing needs the raw bytes after normalization.
  • Fail open: both Observe* and scanLifecycle keep their recover(). If norm panics, normDone stays false, so the other caller retries. A panic in one scan does not block the other or the request.
  • previous_response_id: norm is not called. The test pins this.

Tests (eval box 1, merged tree)

  • go vet on internal/waitcause, proxy, dash, cmd/...: clean.
  • go test -race: internal/waitcause ok, cmd/... ok (24s), dash (Lifecycle, WaitCause) ok.
  • go test -race ./proxy -run 'Lifecycle|SharedNorm|WaitCause|Capture': ok (80s). I did not run the full ./proxy package.

Benchmark (30 iterations x 3, 300-turn body, about 1.2 MB)

Twice = what main does today. Once = this PR.

time/op B/op allocs/op
main (Twice) 35.2 - 36.9 ms 12.4 MB 31,220
PR (Once) 20.2 - 20.7 ms 7.1 MB 15,937
  • -43% time, -43% bytes, -49% allocs. The claim is real.
  • This runs on the writer goroutine, not the request path. The gain is writer throughput and less GC.

Mutations

  • waitcause: drop the last entry after norm() -> caught (TestOpenTeammateWait, TestTeammateReplyClosesAnOpenWaitAndWakesTheThread, TestSharedNormalizationIsByteIdentical).
  • scanLifecycle: empty the message slice -> caught (TestSharedNormalizationIsByteIdentical, TestScanLifecycle, TestLifecycleSignalsEndToEnd).
  • Remove the memo in finish() (if !normDone || true): my run was contaminated by a leftover mutation, so I have no valid result. No test counts normalizations across the two closures. Only Observe is checked for "norm at most once".

Non-blocking

  • Add a test that pins "one normalization per request" across the two closures in finish(). A call counter on norm would do. Today a regression there only shows in the benchmark.
  • lifecycleflag_test.go and dash/lifecycle_retention_test.go are new tests that do not relate to the perf change. They pass. A separate PR would be cleaner.
  • The benchmark file keeps scanLifecycleOld inside lifecycle_equiv_test.go. Good as a reference. Add a short note that it must not be "fixed" to match new code.

@amiddavid
amiddavid merged commit 5d1becd into main Oct 15, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-to-merge Program PR approved by the reviewer, CI green; needs maintainer approval

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

3 participants