viz smart: a country choropleth panel is non-deterministically dropped (cause unknown)

[jqnatividad]

What happened

During a single examples/viz/refresh_external_dashboards.sh run, two committed gallery dashboards silently lost their country-level choropleth panel. No error, no warning on stderr, no non-zero exit — the pages just came out with one fewer panel and everything after it renumbered.

Page	Command	Panels before	after
smart_world_events.html	qsv viz smart world_events_dated.csv	8	7
smart_world_choropleth.html	qsv viz smart world_cities.csv --dictionary infer	13	12

In both cases the missing panel is the choropleth, and in both cases it sat at panel index 2:

smart_world_events.html
  before: indicator, scattergeo, choropleth, scatterpolar, scatter, scattergeo, violin
  after : indicator, scattergeo,             scatterpolar, scatter, scattergeo, violin

smart_world_choropleth.html
  before: indicator, scattergeo, choropleth, violin, bar, heatmap, bar, bar, bar, violin, box, violin
  after : indicator, scattergeo,             violin, bar, heatmap, bar, bar, bar, violin, box, violin

Both are country-level choropleths keyed by ISO3 codes (["MOZ","TUR","USA",...] and ["COD","NGA","EGY",...]).

Why it matters

examples/viz/*.html are committed artifacts published to GitHub Pages by viz-gallery-pages.yml. A silently-omitted panel gets committed and published looking entirely plausible — there is nothing in the output that says a panel was dropped. It was only caught because a version-bump diff showed an unexpected net-negative line count on those two files.

viz smart does report other omissions on the page itself (#4491-era "what it left out" chrome) and on stderr, so a dropped choropleth being completely silent looks like a gap rather than a design choice.

It is not deterministic

Re-running the generator restored the panel in both cases, and I could not reproduce it in isolation. Direct runs of the identical command produced the full 8 panels 7 times out of 7:

Run	Panels
pre-bump binary, bare command	8
post-bump binary, bare command	8
with --preview-threshold 500 (what the generator adds)	8
cold stats cache (sidecars deleted first)	8
warm stats cache	8
with QSV_CENSUS_API_KEY unset	8
gen_gallery.py re-run	8

Ruled out

• Not the DataTables 3.0.4 bump it was found during. Built the pre-bump binary and ran the identical command: 8 panels. Post-bump: 8 panels.
• Not --preview-threshold 500. The generator adds it in run_html; adding it by hand still gives 8. (world_events_dated.csv is 265 rows, so the threshold is inert anyway.)
• Not a stale or cold stats cache. Deleted world_events_dated.stats.* and ran cold, then warm — 8 both times.
• Not QSV_CENSUS_API_KEY. The refresh was run under env -u QSV_CENSUS_API_KEY; running with and without the key both give 8.
• Not generator parallelism. gen_gallery.py runs figures serially — no thread/process pool.

The one thing that distinguishes the bad run

Both bad outputs came from the same refresh_external_dashboards.sh invocation, which builds five large dashboards first (Pittsburgh 311 at 269 MB, Boston 311 at 146 MB, PA crashes at 396 MB) and only then runs gen_gallery.py. Every good output came from a gen_gallery.py run that did not follow those heavy builds. That is a correlation across one sample, not a diagnosis — but it is the only variable I did not manage to eliminate, and it points at state or resource pressure carried across invocations rather than at anything in the two figures' own inputs.

Suggested direction

Two independent things worth considering:

1. Find the non-determinism. Whatever decides that a country choropleth is worth a panel can evidently answer "no" for inputs where it normally answers "yes". Whether that is a geo-resolution step, a stats-cache read, or something order-dependent, the useful property is that it currently fails closed and silently.
2. Make it loud regardless. A dropped panel should say so — on stderr at minimum, and ideally in the page's own omissions note, the same way capped parcats combinations and out-of-region points already do. That turns this class of bug into something a regen diff or a CI log surfaces immediately, instead of something caught by eye months later.

A cheap guard for the artifacts specifically: have gen_gallery.py compare each regenerated page's panel count against the committed one and warn on a decrease, alongside its existing warn_stale_data_viewers / warn_stale_parcats_pages checks.

Environment

qsv 23.0.0-jemalloc-apply;fetch;foreach;geocode;Luau 0.736;magika;prompt;to;viz;viz_static;
polars-0.55.1:py-1.44.2;self_update-16-16 (aarch64-apple-darwin compiled with Rust 1.98;
macOS 27.0-Darwin 27.0.0;Apple M4 Max-16)

Found while regenerating the gallery for the DataTables dt-3.0.4/b-4.0.3/cc-2.0.2/sb-2.0.1 bump. The committed artifacts there are correct — every page was checked for panel-count parity against HEAD before staging — so this issue is about the underlying non-determinism, not about a bad artifact in the tree.

[jqnatividad]

Part 2 ("make it loud") landed in #4594 (squashed as f02ec4669). Part 1 — the non-determinism — is unresolved, and this issue stays open for it.

What changed

The panel can no longer vanish without saying so. build_smart_choropleth_panel reached the geocode engine through reverse_geocode_regions(..).ok()?, and because the builder returned a bare Option there was structurally nowhere for an error to go — so every hard setup failure was discarded and the panel simply ceased to exist. Now every way out either draws a panel or reports itself:

• the engine could not be loaded at all — reported with the engine's own error text, which names the layer that actually failed (index load, engine init, tokio runtime, download);
• the engine loaded but resolved no usable region code — a separate notice, for an index whose city records survive while its country table does not (a GeoRegion can carry a valid ISO-2 code with an empty ISO-3 one, and the tally skips empties);
• one resolved region stays silent, as before — that is the documented single-metro case.

A network failure is reported rather than aborting. That diverges from --geojson auto and viz choropleth, where a Network error propagates, and it was a deliberate reversal after measuring the alternative: aborting made qsv viz smart exit 3 on any globe-spanning CSV offline with no cached index — including quakes.csv, the fixture of the existing un-ignored viz_smart_with_coords_has_map_panel, which asserts success. The rule this panel doesn't inherit exists so network weather cannot decide which candidate column wins the region contest; there is no contest here, just one unrequested companion to the point map.

Separately, examples/viz/gen_gallery.py now records each page's panel count immediately before overwriting it and warns on a decrease, so the artifact-level symptom this issue was caught by shows up in the regen log rather than in a diff months later.

Measured on world_events_dated.csv against a deliberately poisoned index: 7 panels → 6 with no output whatsoever before; the same 6 panels plus a named reason after.

What the next occurrence will tell us

This is the diagnostic that was missing. Next time the panel disappears, stderr will carry one of the two notices above — and if it carries neither, that rules out the entire geocode-engine path and points somewhere else, which is itself the more valuable answer.

Ruled out, so nobody repeats these

• Not a misclassified 404. util::download_file already rejects non-success statuses, so a 404 was a real Err all along — it was being swallowed, not mislabelled.
• "State carried across invocations" has almost no channel. 4 of the 5 heavy dashboards in refresh_external_dashboards.sh pass an explicit --geojson, which takes the point-in-polygon branch and never loads the geocode engine at all; only --geojson auto does. And gen_gallery.py's cleanup_sidecars() only ever sweeps examples/viz, never ~/.qsv-cache.
• Not a missing version-keyed index. qsv-23.0.0-geocode-index.rkyv predates the version-bump commit, so the bad run was not a pre-release 404 against a release that did not exist yet.

One correction to the tables above

The tables in this report say 8 → 7 and 13 → 12, but both committed pages carry 7 and 12 Plotly.newPlot("qsv-viz-panel-N" calls respectively — which is also what the trace lists in this report come to (7 and 12 entries before, 6 and 11 after). The table rows look one high in both cases; the trace lists are right.