Repository navigation
fix: report the input paths that do not exist - #103
Merged
Merged
Conversation
`fetch_image_config` has always returned `(out_list, err)` with every missing path collected in `err`. Every caller threw it away -- generate_augmentations unpacked it as `_`, train_monai bound `err_train`/`err_val` and never read them -- so a data config naming 500 subjects of which 400 were missing augmented 100 of them and printed nothing at all. The filter also only checked IMAGE, so a missing LABEL passed it and failed much later inside a worker with an opaque nibabel error. Both paths are checked now. `report_missing` prints a count, the first few paths and how many more there are, from both scripts. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The defect
fetch_image_confighas always returned(out_list, err), whereerrcollects every path that is not on disk. Every caller discarded it:So a data config naming 500 subjects of which 400 are missing augments 100 of them, or trains on 100 of them, and prints nothing at all. Silent data loss is the failure mode that costs a whole run before anyone notices.
Second defect in the same filter: it only checked
IMAGE.A pair whose label was missing passed the filter and failed much later, inside a
process_mapworker, with an opaque nibabel error.The fix
Check both paths, and add
report_missing(err, split)— a count, the first few paths, and how many more — called from both scripts.Tests
unit_tests/test_missing_inputs.py: a pair with a missing label is rejected and named; a complete pair is accepted; the report names paths, stays silent when nothing is missing, and truncates a long list; and both scripts call it.Verified failing before the fix and passing after. Full suite unchanged against baseline;
ruff check .clean.🤖 Generated with Claude Code