Data Schematic: Make it agent-friendly

[jqnatividad]

The webapp visual packaging is optimized for humans, make it so agents can more efficiently consume the DS.

We have all the artifacts embedded as machine-readable data, expose a simple "API" that both agents and traditional programs can use to get the DS metadata and the bundled sidecars (JSON Schema, stats, freq tables. etc.)

[jqnatividad]

Investigation findings

Scope: what's already embedded, where the real gap is, and how this relates to the
docs/DATA_SCHEMATIC.md conformance table and to the adjacent open issues. No code changes.

1. What the page already carries

The premise is right — the artifacts are there — but they are narrower and less uniform
than "all the artifacts embedded as machine-readable data" suggests.

Under --dict-info only, five sidecars ship as base64 data: URIs on
<a class="qsv-dict-export" download> anchors:

sidecar	short tag	when
<slug>.schema.json (the JSON Schema dictionary)	—	only if the dict is real JSON Schema (has properties); legacy fields dicts get no button
<stem>.stats.csv.data.jsonl	stats	stats anchor canonicalizes
<stem>.stats.csv.json	stats meta	same
frequency cache	freq	only when freq_from_cache actually hit
<stem>.viz-frequency.csv	charted freq	always — built in memory, never written to disk, works for stdin
<stem>.stats.bivariate.csv	bivariate	only when this run's moarstats subprocess wrote it

Built by collect_sidecar_downloads (src/cmd/viz.rs:18593), rendered at viz.rs:24777-24835
inside render_dict_page_html (viz.rs:24329). Supporting: SidecarDownload (viz.rs:18441),
sidecar_mime (viz.rs:18456), read_sidecar (viz.rs:18543), redact_sidecar_paths
(viz.rs:18504), MAX_SIDECAR_EMBED_BYTES = 4 MB (viz.rs:18438).

Two properties worth preserving in any API: the set is never speculative (a sidecar that
exists on disk but wasn't consumed is deliberately not offered), and absolute local paths are
redacted out of the embedded metadata.

Verified empirically against the committed gallery: smart_nyc311.html has 11 export markers;
smart_sales.html, smart_cpdb.html and smart_timeseries.html have zero.

Present on every run regardless of flags: #qsv-data-rows (gz-b64 or plain JSON),
#qsv-data-cols (always plain JSON), #qsv-tour-config (plain JSON).

One correction worth recording: there is no single machine-readable manifest blob today.
Two unrelated mechanisms coexist — gzip+base64 <script type="application/gzip-b64"> payloads
(plotly bundle, figure JSON, datatable rows) and plain-base64 data: download anchors.
date_generated / compute_duration_ms appear nowhere in viz.rs; they reach the page only
transitively, inside the embedded .stats.csv.json.

2. The actual gap is not the sidecars

The sidecars are at least in the artifact. The genuinely agent-hostile part is that
skipped columns and their reasons, panels not drawn, and GeoJSON binning coverage exist only
as stderr prose.

This is not a data-availability problem — it's a plumbing one. SkipRecord { column, field, reason } (viz.rs:19857) is already fully structured internally, and classify /
classify_measure / classify_with_semantics return Result<PanelKind, SkipReason>
specifically so the reason is recorded at the decision rather than re-derived. It just never
reaches the artifact in machine-readable form — only as drawer prose, and only under
--dict-info.

Two places in-tree already concede this:

• docs/DATA_SCHEMATIC.md:216, the MUST 2 row of qsv's own implementation table: "Skipped
  columns … are named in a stderr note. Per-column reasons are given only for the
  null-sentinel and non-numeric-measure diagnostics; the general skip list names columns
  without attributing a reason to each."
• .claude/skills/visual-data-dictionary/SKILL.md instructs the agent to scrape that stderr —
  lines 586, 606 ("that is authoritative"), 612, 614, 724, 731.

Suggested acceptance test for this issue: if those SKILL.md lines still say "scrape
stderr" after the change lands, #4453 isn't closed.

3. What an agent gets over MCP today

A filesystem path to a 1–22 MB HTML file, and nothing else.

viz is in both ALWAYS_FILE_COMMANDS and NON_TABULAR_COMMANDS
(.claude/skills/src/tool-constants.ts:72,88), HTML-only is enforced at
tool-handlers.ts:782-794, and file-operations.ts:676 sets
const forceFileArtifact = commandName === "viz" — so the artifact is never inlined even
when small. There is zero post-processing: no extraction, no metadata summary. To learn
anything about the dataset, the agent must read and parse multi-megabyte HTML itself.

This is the highest-leverage integration point — whatever this issue produces should surface
here, either inline in the MCP response or as a small companion file beside the HTML.

4. How this maps onto the spec

docs/DATA_SCHEMATIC.md already frames the work:

• MUST 2 (complete field inventory) — satisfied in prose, not in data. See above.
• MUST 8 (checkable) — --dict-info satisfies the bundling half; the spec notes Level 3
  is blocked solely because the bundled inputs carry no integrity digests.

So this issue is largely "finish MUST 2 and MUST 8 in a form a machine can read." If the
payload carries a per-sidecar digest, it also closes the stated Level 3 gap and gives #4448
a canonical object to sign — right now there is no serialized object for a TypedStandards
attestation to attach to.

5. Design axis (decision, not recommendation)

The open question is delivery, and the options are not equivalent:

1. Always-on embedded JSON manifest + a no-browser reader. Works on a forwarded file.
   Cost: ends the "a plain viz smart run is byte-identical" property — churns the gallery
   pages, and every external dashboard changes on its next refresh_external_dashboards.sh.
2. Flag-gated manifest. Zero churn; but agents can't rely on it being present.
3. Separate <stem>.schematic.json. House precedent exists — qsv profile writes
   <input>.metadata.json (src/cmd/profile.rs:335-356) with qsv_version / generated_at /
   input / stats / frequency. Cleanest for pipelines; a forwarded HTML carries nothing.
4. Extractor only, no page change. Works retroactively on every schematic ever produced;
   but leaves the skip reasons — the actual gap — still absent from the artifact.

Constraints that hold regardless of which is chosen:

• Keep it plaintext. If the payload lands inside a gz payload, an agent needs a JS runtime
  to read it, which defeats the purpose. It must be readable with QSV_VIZ_NO_COMPRESS unset.
• Reference the sidecars, don't re-embed them. Pages are already 4–22 MB; inlining bytes
  that already live in the data: anchors doubles them.
• Preserve redact_sidecar_paths. stats.csv.json carries arg_input; serializing
  provenance is exactly where path disclosure regresses.
• schema_version from day one — otherwise every later viz change is a silent breaking
  change.
• Read SkipRecord, don't re-derive it. A re-derived predicate drifts silently, and this
  is a provenance surface.
• Naming: viz's sidecars are stem-based (data.csv → data.stats.csv.data.jsonl), while
  qsv schema is full-filename-based (data.csv.schema.json). A new agent-facing surface
  should follow viz's stem convention, since that's what viz actually reads and writes.

There's a ready-made in-page idiom if the manifest is embedded: inline_json_script(id, json)
(viz.rs:40190), already used for #qsv-tour-config, #qsv-data-rows and #qsv-data-cols.

6. On WebMCP

Considered as an alternative. It is complementary, not a substitute, for three independent
reasons:

1. This issue asks for something "both agents and traditional programs" can use. WebMCP
   requires a browser and a browser-resident agent — a Python script, a CKAN harvester or
   curl | jq get nothing from it.
2. It is an Origin Trial, not shipped stable (Chrome 149+, Edge 150+; local dev needs
   about:flags#enable-webmcp-testing). Trial tokens are origin-bound via a
   <meta http-equiv="origin-trial"> tag, so a self-contained HTML that gets emailed,
   archived, or opened from file:// can't carry a valid one for wherever it lands. (For
   accuracy: file:// is a potentially-trustworthy origin per Secure Contexts — the token
   binding is the blocker, not secure-context gating.)
3. The API renamed itself mid-flight — navigator.modelContext → document.modelContext, old
   name deprecated in Chromium 150. A Data Schematic is an archival artifact and MUST 7 says
   it renders from the file alone; making an unshipped, still-renaming API the only
   machine-readable surface inverts that property.

That said, there's a real opportunity here worth its own issue. A Data Schematic is already
a tool-bearing app: window.__qsvDataSelect(indexes, extend) (viz.rs:23093),
__qsvDataRowCells(i) (viz.rs:23113), __qsvDataPageTo(i) (viz.rs:23154), __qsvDataDT
(viz.rs:23715) and qsvOpenDict() (viz.rs:22718) are tools in all but name. Wrapping them
in registerTool() is a small delta and a striking demo — but it buys actions on a live page
(filter, select, query the viewer), which is a different feature from this issue's ask (get
the metadata and the bundled sidecars). Those tools also need something to return, and that
something is the manifest.

Manifest is the substrate; WebMCP is one delivery channel over it — most valuable in the
hosted contexts already tracked in #4265 (CKAN) and #4390 (Data Concierge), and best landed
once it's out of Origin Trial.

7. Adjacent issues that want the same object

• Data Schematic: Replace Provenance free-text info with TypedStandards cryptographic signature #4448 — a TypedStandards signature needs a canonical serialized object to sign; per-input
  digests are the missing half of MUST 8. These two should agree on what that object is, or
  signing and extraction will diverge.
• Data Schematic: add visual cues denoting agent/machine/human produced/curated/verified data #4474 — agent/machine/human provenance cues; those fields belong in the same payload.
• viz: Data Schematics - Data Concierge aware #4390 / Data Schematic: deeper CKAN integration #4265 — Data Concierge and CKAN are downstream consumers of it.
• create suite-wide metadata cache, caching dataset-level statistics/metadata #2097 — suite-wide metadata cache, same metadata at a different scope.

8. Status

Greenfield — no branch, no PR, and nothing in CHANGELOG.md on this. The single commit
matching --grep=4453 (7c9e91c82, the --tour-audience work) mentions the number
incidentally in its body; it does not implement any part of this.