Repository navigation
Conversation
aldenks
added this pull request to stack #1178
October 7, 2026 20:54
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add opt-in planning for materialized forecast updates before enabling earlier triggers. This first layer adds the shared planner and
jobs:listpermission but opts in no datasets, so deployed data-processing behavior remains unchanged. #1177 enables each target's planning together with its virtual trigger.Worker 0 can freeze a shared work/skip plan before writing: yield to an older unfinished Job of the same CronJob, remove init times already in the primary store, and trim the end to the newest source-ready candidate in wxopticon. Interior inits remain included. Replacement pods reuse the same template end and labels. Validation retries retain the frontier and ignore their finished-writing parent. A reprocessing option supports GEFS35's delayed extension.
The planned follow-up pairs are AIFS Single (about 46 minutes of launch headroom, 29 recent matched runs), GFS forecast (11–12 minutes, one recent complete sample plus retained Job timing), GEFS35 extension (2h35–2h45 before the daily cron), and HRRR48 (small, variable surface-file gains). These estimates precede source-summary/polling overhead; this infrastructure-only PR claims no latency improvement. Analysis cadence remains unchanged. HRRR analysis would require extra updates for its larger gain; GFS/GEFS analysis have small gains with dependency-window complications. ICON-EU and HRDPS archivers run too late to help; existing IFS46 triggers and unpaired products are unchanged.
When enabled, the Job guard is deliberately non-atomic: Icechunk rejects conflicting primary publication, while deprecated Zarr replicas retain rare exposure. A retry can still yield to an unrelated older writer, and a stuck Pending trigger can block later updates until resolved. Frontier deduplication removes routine re-download repair, so sub-threshold holes require a repair backfill. Source-summary errors fail target planning. Skip plans remain as small per-Job files for late pods.
Rollout: grant the deployer
jobs:listusing the updated bootstrap Role, deploy this infrastructure, verify the Role and unchanged cron behavior, then enable the follow-up PR. No live rollout was performed.Validation: repository-wide format, lint and type checks pass. The shared-path suite passes 343 tests; all affected dataset suites pass 327 tests, including slow tests. The infrastructure-only layer independently passes 247 tests from an exported index, plus its own format/lint/type checks. Three independent review passes are complete; no high- or medium-severity findings remain. Both commit signatures are verified by GitHub. Remote CI passes on amd64 and arm64, including Kubernetes admission and CodeQL checks.