Add DynamicInputStreams: runtime field discovery and registration for IOStreams#422
Draft
xylar wants to merge 5 commits into
Draft
Add DynamicInputStreams: runtime field discovery and registration for IOStreams#422xylar wants to merge 5 commits into
xylar wants to merge 5 commits into
Conversation
Author
CTest unit tests:
|
Omega's IOStreams mechanism previously required all fields to be pre-registered in the Metadata system before reading. This is appropriate for model state variables with names and dimensions fixed at compile time, but prevents Analysis operators from using user-supplied weight fields (e.g. MOC basin masks and transect edge-sign arrays) whose names and secondary dimensions vary by configuration. This PR extends IOStreams with a DynamicFields: true per-stream option. When set, Omega inspects the input file at initialization time to discover each field's dimensions and native type, dynamically registers the necessary MetaDim entries and ArrayMetaData records, allocates storage, and reads the data with type promotion (I4/I8/R4 → R8). The resulting fields are placed in the same global field registry as all other Omega fields, so Analysis operators can list them in getInputFieldNames() without any special handling.
a14cc03 to
66035a1
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Omega's IOStreams mechanism previously required all fields to be pre-registered in the Metadata system before reading. This is appropriate for model state variables with names and dimensions fixed at compile time, but prevents Analysis operators from using user-supplied weight fields (e.g. MOC basin masks and transect edge-sign arrays) whose names and secondary dimensions vary by configuration.
This PR extends IOStreams with a
DynamicFields: trueper-stream option. When set, Omega inspects the input file at initialization time to discover each field's dimensions and native type, dynamically registers the necessaryMetaDimentries andArrayMetaDatarecords, allocates storage, and reads the data with type promotion (I4/I8/R4 → R8). The resulting fields are placed in the same global field registry as all other Omega fields, so Analysis operators can list them ingetInputFieldNames()without any special handling.Changes
Source code (
src/)IO.cpp/IO.h: addIO::getVarInfo(), a wrapper around SCORPIO'sPIOc_inq_var*functions that returns dimension names, global lengths, and native data type for a named variable in an open file.IOStream.h/IOStream.cpp: addbool DynamicFieldsmember parsed from YAML; addIOStream::registerAndReadDynamicField()that classifies dimensions (mesh vs. secondary), validates structure, registers newMetaDimentries (with conflict checking), creates anArrayMetaDatarecord andIOField, builds or reuses a SCORPIO decomposition, and reads with type promotion; addIOStream::readAllDynamic(ModelClock)static method that iterates all streams and reads every stream withDynamicFields: true(analogous towriteAll()); modifyIOStream::readStream()to branch onDynamicFields.OceanInit.cpp: callIOStream::readAllDynamic()betweenHorzMesh::init()andVertCoord::init()so that mesh dimensions are available and dynamic fields are in the registry before any downstream operators run.Tests (
test/)DynamicInputStreamTest.cpp: CTest covering basic read of I4 fields on cells and edges with I4→R8 promotion, secondary-dimension deduplication across streams, and thereadAllDynamic()path.DynamicInputStreamBadFieldTest.cpp,DynamicInputStreamCollisionTest.cpp,DynamicInputStreamDimConflictTest.cpp: WILL_FAIL CTes covering the three hard-error conditions (field lacking a mesh dimension, field-name collision with an existing registry entry, and secondary-dimension size conflict between streams).Checklist
Testingwith the following:have been run on and indicate that are all passing.
has passed, using the Polaris
e3sm_submodules/Omegabaseline-pfor both the baseline (Polarise3sm_submodules/Omega) and the PR build