Skip to content

Trigger materialized forecasts when source runs are ready - #1177

Draft
aldenks wants to merge 1 commit into
materialized-update-planningfrom
materialized-triggers
Draft

aldenks wants to merge 1 commit into
materialized-update-planningfrom
materialized-triggers

Conversation

@aldenks

@aldenks aldenks commented Oct 7, 2026 •

Copy link
Copy Markdown
Member

Trigger materialized AIFS Single, GFS forecast, HRRR48 and GEFS35 updates when wxopticon records their source coverage ready, instead of always waiting for their fixed cron times. The virtual polling loop checks once per minute per phase, including after all references are present, within its existing deadline. It defers submission while any target Job is unfinished, then checks again, so an older cron with an empty/outdated plan cannot routinely consume the ready trigger. Expected summary and Kubernetes API/network errors log and defer the trigger without stopping virtual ingestion.

The existing CronJob clone helper supplies production-only, deterministic per-init/per-horizon Jobs. AIFS uses GCS through f360; GFS uses pgrb2 through f384; HRRR48 uses surface files through f48. GEFS has two independent phases: yesterday's f840 extension after the existing 28-hour gate, and today's f384 forecast. The day-16 phase catches late sources after the daily cron. All four targets opt into planning in this PR alongside their triggers. GEFS planning retains frontier reprocessing for extension repair.

Expected headroom before summary/polling overhead:

  • AIFS: recent median about 46 minutes, 29 matched runs.
  • GFS: about 11–12 minutes, one matched recent completion sample and corroborating retained Job times.
  • GEFS extension: about 2h35–2h45 before the 06:45 UTC cron; the day-16 phase adds smaller gains and late-source recovery.
  • HRRR48: modest and variable. All-family completion was near cron time, but this trigger needs only the surface subset; no guaranteed median gain is claimed.

Merge after #1176 has been deployed and verified. Coordinate with the separate GEFS virtual schedule change to 15 2 * * * and an 8h30m pod deadline; this PR changes no schedules or deadlines. The trigger also works under the prior schedule. Analysis cadence is unchanged: HRRR analysis's larger gain requires extra runs; GFS/GEFS analysis add dependency-window complexity for small gains. ICON-EU and HRDPS archivers run too late; existing IFS46 triggers and unpaired products remain unchanged.

The shared worker-zero plan handles older writers and repeated work. Unlike the optional virtual trigger, target planning fails loudly when the public source summary is unavailable or invalid; this is an explicit data-flow dependency. Falling back to partial writes would need a repair policy compatible with frontier deduplication. Its guard is non-atomic; deprecated Zarr replicas retain rare exposure. GEFS monitoring samples members, so an early extension may be incomplete until a reprocessing run. GEFS phases and its cron can roughly double routine update compute; no new completion watermark is introduced. Source availability after both cron and polling deadline waits until the next schedule, possibly without an immediate currency alert. Verify GEFS daily node-hours during rollout; the exact added cost has not been measured. No live rollout was performed.

Validation: repository-wide format, lint and type checks pass. The shared-path suite passes 343 tests; all affected dataset suites pass 327 tests, including slow tests. The infrastructure-only layer independently passes 247 tests from an exported index, plus its own format/lint/type checks. Three independent review passes are complete; no high- or medium-severity findings remain. Both commit signatures are verified by GitHub. Remote CI passes on amd64 and arm64, including Kubernetes admission and CodeQL checks.

@aldenks
aldenks added this pull request to stack #1178 October 7, 2026 20:54
@aldenks aldenks changed the title Trigger materialized forecasts from source-ready virtual updates Trigger materialized forecasts when source runs are ready Oct 7, 2026
@aldenks aldenks self-assigned this Oct 7, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant