Skip to content

Add DynamicInputStreams: runtime field discovery and registration for IOStreams#422

Draft
xylar wants to merge 5 commits into
E3SM-Project:developfrom
xylar:omega/add-dynamic-input-streams
Draft

Add DynamicInputStreams: runtime field discovery and registration for IOStreams#422
xylar wants to merge 5 commits into
E3SM-Project:developfrom
xylar:omega/add-dynamic-input-streams

Conversation

@xylar

@xylar xylar commented May 28, 2026

Copy link
Copy Markdown

Omega's IOStreams mechanism previously required all fields to be pre-registered in the Metadata system before reading. This is appropriate for model state variables with names and dimensions fixed at compile time, but prevents Analysis operators from using user-supplied weight fields (e.g. MOC basin masks and transect edge-sign arrays) whose names and secondary dimensions vary by configuration.

This PR extends IOStreams with a DynamicFields: true per-stream option. When set, Omega inspects the input file at initialization time to discover each field's dimensions and native type, dynamically registers the necessary MetaDim entries and ArrayMetaData records, allocates storage, and reads the data with type promotion (I4/I8/R4 → R8). The resulting fields are placed in the same global field registry as all other Omega fields, so Analysis operators can list them in getInputFieldNames() without any special handling.

Changes

Source code (src/)

  • IO.cpp / IO.h: add IO::getVarInfo(), a wrapper around SCORPIO's PIOc_inq_var* functions that returns dimension names, global lengths, and native data type for a named variable in an open file.
  • IOStream.h / IOStream.cpp: add bool DynamicFields member parsed from YAML; add IOStream::registerAndReadDynamicField() that classifies dimensions (mesh vs. secondary), validates structure, registers new MetaDim entries (with conflict checking), creates an ArrayMetaData record and IOField, builds or reuses a SCORPIO decomposition, and reads with type promotion; add IOStream::readAllDynamic(ModelClock) static method that iterates all streams and reads every stream with DynamicFields: true (analogous to writeAll()); modify IOStream::readStream() to branch on DynamicFields.
  • OceanInit.cpp: call IOStream::readAllDynamic() between HorzMesh::init() and VertCoord::init() so that mesh dimensions are available and dynamic fields are in the registry before any downstream operators run.

Tests (test/)

  • DynamicInputStreamTest.cpp: CTest covering basic read of I4 fields on cells and edges with I4→R8 promotion, secondary-dimension deduplication across streams, and the readAllDynamic() path.
  • DynamicInputStreamBadFieldTest.cpp, DynamicInputStreamCollisionTest.cpp, DynamicInputStreamDimConflictTest.cpp: WILL_FAIL CTes covering the three hard-error conditions (field lacking a mesh dimension, field-name collision with an existing registry entry, and secondary-dimension size conflict between streams).

Checklist

  • Documentation:
  • Linting
  • Testing
    • Add a comment to the PR titled Testing with the following:
      • Which machines CTest unit tests
        have been run on and indicate that are all passing.
      • The Polaris omega_pr test suite
        has passed, using the Polaris e3sm_submodules/Omega baseline
      • Document machine(s), compiler(s), and the build path(s) used for -p for both the baseline (Polaris e3sm_submodules/Omega) and the PR build
      • Indicate "All tests passed" or document failing tests
      • Document testing used to verify the changes including any tests that are added/modified/impacted.
    • New tests:
      • CTest unit tests for new features have been added per the approved design.
      • Polaris tests for new features have been added per the approved design (and included in a test suite)

@xylar xylar self-assigned this May 28, 2026
@xylar xylar added the enhancement New feature or request label May 28, 2026
@xylar

xylar commented May 28, 2026

Copy link
Copy Markdown
Author

CTest unit tests:

  • Machine: chrysalis
  • Compiler: oneapi-ifx
  • Build type: Debug
  • Result: All tests passed
  • Log: /gpfs/fs1/home/ac.xylar/e3sm_work/polaris/main/build_omega/build_chrysalis_oneapi-ifx/ctests.log

xylar added 5 commits May 28, 2026 09:32
Omega's IOStreams mechanism previously required all fields to be pre-registered in the Metadata system before reading. This is appropriate for model state variables with names and dimensions fixed at compile time, but prevents Analysis operators from using user-supplied weight fields (e.g. MOC basin masks and transect edge-sign arrays) whose names and secondary dimensions vary by configuration.

This PR extends IOStreams with a DynamicFields: true per-stream option. When set, Omega inspects the input file at initialization time to discover each field's dimensions and native type, dynamically registers the necessary MetaDim entries and ArrayMetaData records, allocates storage, and reads the data with type promotion (I4/I8/R4 → R8). The resulting fields are placed in the same global field registry as all other Omega fields, so Analysis operators can list them in getInputFieldNames() without any special handling.
@xylar xylar force-pushed the omega/add-dynamic-input-streams branch from a14cc03 to 66035a1 Compare May 28, 2026 14:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant