docs(integrations): add tracegauge Cost Evaluator for ADK agents - #2128
Open
gaurav-gandhi-2411 wants to merge 12 commits into
Open
docs(integrations): add tracegauge Cost Evaluator for ADK agents#2128gaurav-gandhi-2411 wants to merge 12 commits into
gaurav-gandhi-2411 wants to merge 12 commits into
Conversation
adk-tracegauge (https://pypi.org/project/adk-tracegauge/) registers a google.adk.evaluation metric that reports real per-invocation dollar cost, built on tracegauge's cost engine. ADK's built-in evaluation metrics cover quality (trajectory match, response similarity, safety, hallucination) but none report cost or token usage -- this fills that gap. Follows the standard integration template (frontmatter + Use cases / Prerequisites / Installation / Use with agent / Available metrics / Resources), closest in shape to the existing MLflow Scorers entry -- the only other catalog integration that also registers against ADK's MetricEvaluatorRegistry. catalog_tags: ["evaluation"] matches that precedent. The "Use with agent" code example was run against the actual published adk-tracegauge==0.1.0 package, not copy-pasted from untested source. 512x512 square icon included.
✅ Deploy Preview for adk-docs-preview ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
…ator/adk eval limitation Removes the "Use with agent" instructions that instruct wiring the plugin into an App and expecting AgentEvaluator/adk eval to pick it up automatically -- confirmed against google-adk==2.6.3 that this crashes AgentEvaluator.evaluate() and silently discards per-invocation output in adk eval, because LocalEvalService discards results for any metric reporting EvalStatus.NOT_EVALUATED (this metric's permanent status). Filed upstream: google/adk-python#6725. Replaces it with the hand-rolled Runner harness that actually works, pointed at the package README's full worked example, plus a caveated after_model_callback workaround.
Author
|
Correction to this PR's original "Use with agent" instructions, found and fixed before this could reach anyone following the docs. The original page instructed wiring Digging into why surfaced a second, independent issue in I've pushed a revision to this PR that:
The package itself ( Happy to adjust tone/length/placement to fit the catalog's normal style — flag anything that reads oddly for the format. |
…se 4 API The page still described the pre-Phase-2 state of adk-tracegauge: a Gemini-only price table, a metric that always reports NOT_EVALUATED with "no inverted-metric convention to plug into", and tracegauge as a pulled-in runtime dependency. All three are stale. - Hero path is now `tracegauge check` (the CI cost-regression gate), with the `adk eval` metric documented as a clearly labeled secondary path -- matching the package's own README repositioning (Phase 3 B6). - The metric now computes a real PASSED/FAILED verdict against a required threshold (Phase 2 W2); the stale "always NOT_EVALUATED" framing is removed and replaced with the real, narrower residual limitation (AgentEvaluator.evaluate()'s directionality bug, Phase 3 B3). - Documents `tracegauge check --mode paired` and the real pairing key (eval_case_id, authored and stable -- not session_id, which Phase 4 R2 found unreachable for the primary `adk eval` CLI path), including the in-process CliRunner mechanism required to capture usage at all. - Documents both residual ADK-side limitations (the AgentEvaluator inversion bug and adk eval's exit code) as "a fix has been prepared and is pending submission upstream" -- the corresponding adk-python PRs are prepared but not yet opened, so no PR link is given. - Pricing/install text updated for Phase 4 R5 (tracegauge PyPI dependency removed; arithmetic ported in-house). Every code block was executed for real against a freshly built wheel, installed into a clean venv, run from a directory outside both repos -- including the paired-mode block, which required correcting the mechanism (usage capture only works if the entrypoint runs `adk eval`'s own Click command in-process, not as a separate shell step) after the first draft's two-separate-shell-commands version was tested and found not to work. Sequencing: this PR must not merge before an adk-tracegauge release carrying the Phase 4 API (0.3.0 per CHANGELOG.md, not yet published -- PyPI still serves 0.2.0) is actually live, or the page would describe commands a reader's `pip install adk-tracegauge` cannot yet reproduce.
…gauge adk-tracegauge's console script was renamed from `tracegauge` to `adk-tracegauge` to resolve a name collision with the sibling `tracegauge` PyPI package (token-efficiency-scorer), which already installs a console script under the same name. Updates every CLI invocation, catalog title, and heading in this integration page to match (25 occurrences) -- not pushed yet per this branch's own gating (adk-tracegauge 0.3.0 must be live on PyPI first).
… (Phase 6 T5) The captured paired-mode output block still showed "95% CI" from before adk-tracegauge's Phase 5 S4 confidence retune (default 0.95 -> 0.98) -- this page was written in Phase 4 R3, before that retune landed, and was never re-captured. Re-verified live against a genuinely fresh 0.3.0 wheel (built and installed into a clean venv, run from outside both repos) using the exact --eval-history CLI flag sequence this page documents: real output is byte-identical to what's now shown, including "98% CI" [+0.001800, +0.001800] and exit code 1. Every other command/number on the page (google-adk pin, ADK_TRACEGAUGE_ASSUME_LOCAL, the sequencing constraint pinning this PR to after 0.3.0's real PyPI publish) was cross-checked against the 0.3.0 package and found still accurate -- this was the only stale spot found. Still not pushed, per this branch's own gating.
Rewrites the "Paired mode" section for adk-tracegauge's Phase 7 U1 change: --mode auto now prefers paired comparison whenever a pairing key resolves, falling back to two-sample only when it can't -- the section previously described paired as something opted into on top of an independent-samples default. Drops the now-unnecessary --mode paired flag from the CLI example and adds the shipped-configuration FPR/power numbers with Wilson 95% CIs, matching the standard the upstream README now uses. Re-verified against a freshly built adk-tracegauge wheel in a clean venv outside any repo checkout (google-adk 2.7.0): the no --mode-flag example reproduces the documented output byte-for-byte.
…nce from remote The remote-only commit (7cd1d91) this branch was rebased onto filed google/adk-python#6725 (LocalEvalService discards per-invocation results for any metric reporting overall_eval_status=NOT_EVALUATED) as part of a now-superseded framing (pre-Phase-2-fix, when this metric was permanently NOT_EVALUATED). That specific framing is contradicted by this page's current content -- the metric now reports real PASSED/FAILED -- but the underlying upstream issue is still open and still genuinely relevant: an individual invocation can still land NOT_EVALUATED (unresolved model, streaming anomaly, unpriced token category), and ADK still discards its rationale in that case. Preserved as a third "Known ADK-side limitations" entry plus a Resources link, per the reconciliation instruction to keep orthogonal-and-not-contradicted content from the remote commit.
… re-run example The adk eval PASSED example previously showed Score: 0.0007999999999999999, a figure carried forward from an earlier phase whose originating fixture was not committed or reproducible against the published 0.3.0 install (audit RELEASE_0_3_0.md W5.3, marked UNVERIFIED). Swapped in examples/01_minimal_cost_gate.py's PASSED output instead: a deterministic fixture (fixed fake-token-count model, no API key needed) already committed in adk-tracegauge and independently re-run this session against a fresh `adk-tracegauge==0.3.0` install from PyPI, producing byte-identical output (Score: 2.8, Threshold: 5.0). Cited the example file explicitly so the number is reproducible by any reader, not just asserted.
adk-tracegauge's own Phase 8 audit (docs/audit/FPR_ANOMALY.md in that repo) found the "paired mode's FPR is higher than two-sample's" claim was never significance-tested before being published, and does not hold up when tested -- corrected here to match, with the updated 5,000-trial figures and a brief explanation of the correction.
A real first-run failure on Windows (Python 3.14, default user-site pip install): adk-tracegauge --help raised CommandNotFoundException because the console script lands in a per-user Scripts directory not on PATH by default. Adding the note to the install step itself, not troubleshooting, since it's the first command a reader runs. Companion fix (python -m adk_tracegauge as a PATH-independent fallback) ships in adk-tracegauge 0.3.1 -- see gaurav-gandhi-2411/adk-tracegauge#8.
Mirrors adk-tracegauge's own README fix (gaurav-gandhi-2411/adk-tracegauge PR docs/ad1-ad2-cv-reframe): the single "99.22% at n=30" power figure assumed one specific, unmeasured cost-variance level. Replaced with an explanation of why paired/two-sample need a different CV measured against your own data, and a pointer to the package README's CV x n power table instead of restating one number here. FPR figures (unaffected by this finding) kept as-is.
Mirrors the package README's own AE1 fix (gaurav-gandhi-2411/ adk-tracegauge PR google#23): the apparent gap between the originally-published power figure and a later real measurement is not a bug in either -- it's two different, both-legitimate assumptions about within-case noise (fixed absolute dollar amount vs. proportional to cost). Frames both regimes, states which real workload each corresponds to, and points to the package README's own side-by-side tables rather than restating them. NOT PUSHED YET -- held locally per this session's explicit instruction: do not push the adk-docs branch until adk-tracegauge's next version (carrying this same fix) is live on PyPI.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
Adds
adk-tracegaugeto the integrations catalog. It registers agoogle.adk.evaluationmetric (adk_tracegauge_cost_usd) that reports real per-invocation dollar cost, built on tracegauge's cost engine. ADK's built-in evaluation metrics cover quality (trajectory match, response similarity, safety, hallucination) but none report cost or token usage — this fills that gap.Changes
docs/integrations/adk-tracegauge.md: new integration page, following the standard template (frontmatter + Use cases / Prerequisites / Installation / Use with agent / Available metrics / Resources). Closest in shape to the existing MLflow Scorers entry — the only other catalog integration that also registers against ADK'sMetricEvaluatorRegistry.catalog_tags: ["evaluation"]matches that precedent.docs/integrations/assets/adk-tracegauge.png: 512×512 square icon.Testing
adk-tracegauge==0.1.0package (not copy-pasted untested).Risk & rollback
New page, additive only — doesn't touch any existing catalog entry or navigation config (individual integration pages aren't listed in
mkdocs.yml; the catalog index auto-renders fromdocs/integrations/*.md). Revert is a two-file delete.