Repository navigation
Conversation
aldenks
added this pull request to stack #1178
October 7, 2026 20:54
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Trigger materialized AIFS Single, GFS forecast, HRRR48 and GEFS35 updates when wxopticon records their source coverage ready, instead of always waiting for their fixed cron times. The virtual polling loop checks once per minute per phase, including after all references are present, within its existing deadline. It defers submission while any target Job is unfinished, then checks again, so an older cron with an empty/outdated plan cannot routinely consume the ready trigger. Expected summary and Kubernetes API/network errors log and defer the trigger without stopping virtual ingestion.
The existing CronJob clone helper supplies production-only, deterministic per-init/per-horizon Jobs. AIFS uses GCS through f360; GFS uses pgrb2 through f384; HRRR48 uses surface files through f48. GEFS has two independent phases: yesterday's f840 extension after the existing 28-hour gate, and today's f384 forecast. The day-16 phase catches late sources after the daily cron. All four targets opt into planning in this PR alongside their triggers. GEFS planning retains frontier reprocessing for extension repair.
Expected headroom before summary/polling overhead:
Merge after #1176 has been deployed and verified. Coordinate with the separate GEFS virtual schedule change to
15 2 * * *and an 8h30m pod deadline; this PR changes no schedules or deadlines. The trigger also works under the prior schedule. Analysis cadence is unchanged: HRRR analysis's larger gain requires extra runs; GFS/GEFS analysis add dependency-window complexity for small gains. ICON-EU and HRDPS archivers run too late; existing IFS46 triggers and unpaired products remain unchanged.The shared worker-zero plan handles older writers and repeated work. Unlike the optional virtual trigger, target planning fails loudly when the public source summary is unavailable or invalid; this is an explicit data-flow dependency. Falling back to partial writes would need a repair policy compatible with frontier deduplication. Its guard is non-atomic; deprecated Zarr replicas retain rare exposure. GEFS monitoring samples members, so an early extension may be incomplete until a reprocessing run. GEFS phases and its cron can roughly double routine update compute; no new completion watermark is introduced. Source availability after both cron and polling deadline waits until the next schedule, possibly without an immediate currency alert. Verify GEFS daily node-hours during rollout; the exact added cost has not been measured. No live rollout was performed.
Validation: repository-wide format, lint and type checks pass. The shared-path suite passes 343 tests; all affected dataset suites pass 327 tests, including slow tests. The infrastructure-only layer independently passes 247 tests from an exported index, plus its own format/lint/type checks. Three independent review passes are complete; no high- or medium-severity findings remain. Both commit signatures are verified by GitHub. Remote CI passes on amd64 and arm64, including Kubernetes admission and CodeQL checks.