Skip to content

fix: report the input paths that do not exist - #103

Merged
Hendrik-code merged 2 commits into
mainfrom
hm/fix-missing-input-reporting
Oct 3, 2026
Merged

Hendrik-code merged 2 commits into
mainfrom
hm/fix-missing-input-reporting

Conversation

@Hendrik-code

Copy link
Copy Markdown
Collaborator

The defect

fetch_image_config has always returned (out_list, err), where err collects every path that is not on disk. Every caller discarded it:

# generate_augmentations.py
data_list, _ = fetch_image_config(config_data=data_config, split="TRAINING")

# train_monai.py
train_list, err_train = fetch_image_config(...)   # err_train never read
val_list,   err_val   = fetch_image_config(...)   # err_val never read

So a data config naming 500 subjects of which 400 are missing augments 100 of them, or trains on 100 of them, and prints nothing at all. Silent data loss is the failure mode that costs a whole run before anyone notices.

Second defect in the same filter: it only checked IMAGE.

if not os.path.exists(input_img_path):
    err.append([input_img_path, "path error"])

A pair whose label was missing passed the filter and failed much later, inside a process_map worker, with an opaque nibabel error.

The fix

Check both paths, and add report_missing(err, split) — a count, the first few paths, and how many more — called from both scripts.

Tests

unit_tests/test_missing_inputs.py: a pair with a missing label is rejected and named; a complete pair is accepted; the report names paths, stays silent when nothing is missing, and truncates a long list; and both scripts call it.

Verified failing before the fix and passing after. Full suite unchanged against baseline; ruff check . clean.

🤖 Generated with Claude Code

`fetch_image_config` has always returned `(out_list, err)` with every missing
path collected in `err`. Every caller threw it away -- generate_augmentations
unpacked it as `_`, train_monai bound `err_train`/`err_val` and never read them
-- so a data config naming 500 subjects of which 400 were missing augmented 100
of them and printed nothing at all.

The filter also only checked IMAGE, so a missing LABEL passed it and failed
much later inside a worker with an opaque nibabel error. Both paths are checked
now.

`report_missing` prints a count, the first few paths and how many more there
are, from both scripts.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@Hendrik-code
Hendrik-code merged commit 91928d2 into main Oct 3, 2026
8 checks passed
@Hendrik-code
Hendrik-code deleted the hm/fix-missing-input-reporting branch October 3, 2026 08:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant