Keep RRFS and REFS ingest running when source messages are missing

[aldenks]

RRFS and REFS 12Z source indexes on October 15 omit freezing-rain messages. Their completeness assertions abort ref construction for the entire polling batch, preventing unrelated variables and files from being committed.

Log missing RRFS/REFS messages and commit the available references, leaving omitted messages as fill values. This changes all four RRFS virtual datasets (18-hour, 84-hour, sub-hourly, ensemble) and both REFS virtual datasets (hourly and 3-hourly). The existing dataset hooks keep this policy scoped; GEFS retains its strict completeness checks and known-source exceptions.

Assume NOAA does not add messages to a file after publication. A partially populated file counts as ingested once its representative chunk exists; any later re-ingestion is a manual operation, with no automatic recovery mechanism added here. For all virtual datasets using the default polling loop, a file missing that representative chunk is instead logged and skipped before batching, so it cannot block sibling files. The writer still checks the representative invariant for custom generators.

Existing strict decode validation remains in place: it will fail on sampled missing freezing-rain/frozen-precipitation refs after the available data has committed, for as long as NOAA omits them. This is intentional; no validation waiver is added. Atomic commits preserve previously ingested data. After deployment, updates can fill blocked files within their operational windows; older gaps require a backfill. The normal window is 18 hours: the 12Z October 15 init ages out once the window starts after it (after about 06Z October 16; sub-hourly uses a one-hour-earlier window). No template metadata changes.

Validation: real-index regressions cover partial and complete RRFS/REFS messages; local Icechunk tests cover partial-file commits, isolated missing probes, and healthy siblings in both update and backfill modes. Format, lint, and type checks pass. All 148 tests in the changed test modules pass; the broader shared NOAA and dataset suites also passed. CI is green on amd64 and arm64, including Code Quality, CodeQL, and Kubernetes admission.

🤖 Generated with Claude Code