Skip to content

SENS_AT_SPEC_ABLATION.md

Latest commit

 

History

History
400 lines (334 loc) · 21.5 KB

SENS_AT_SPEC_ABLATION.md

File metadata and controls

400 lines (334 loc) · 21.5 KB
Raw

Sens@Spec Loss Ablation for DeepCatch Foundation Model

UPDATE — n=20 re-run (2026-09-23)

The original n=5 ablation (documented below) was generated under the leaky StratifiedKFold protocol that audit-2 P0-A (commit 6231fcb) replaced with honest per-patient GroupKFold. Re-running both loss="ce" and loss="sens_at_spec" at n=20 seeds under the current protocol shows the foundation model at AUC ≈ 0.56 for both losses — i.e. the focal-BCE change has no detectable effect on this cohort. The original n=5 foundation AUC of 0.94 was a CV-leak artifact, not a real loss-comparison result. The honest n=20 paired comparison below supersedes the n=5 numbers.

n=20 paired t-test (honest GroupKFold, 5-fold, paired seeds)

Metric loss="ce" loss="sens_at_spec" (α=20) Paired Δ (95% CI) p-value Verdict
Foundation AUC 0.5588 ± 0.0094 0.5582 ± 0.0104 −0.0005 [−0.0015, +0.0005] 0.297 null (point estimate flipped sign)
Foundation Sens@99% 0.078 ± 0.053 0.070 ± 0.044 −0.008 [−0.023, +0.008] 0.330 null (sign flip)
Foundation Sens@95% 0.165 ± 0.026 0.165 ± 0.026 0.000 (byte-identical arrays) n/a no change (loss didn't move threshold)
LR-baseline AUC 0.966 (20/20 match) 0.966 (20/20 match) 0.0000 (deterministic) n/a no change (loss not applied)
Smoke gate fail (sens@99=0.35<0.40) fail (sens@99=0.35<0.40) n/a n/a pre-existing
lr_baseline_aucs_match ✅ 20/20 bit-identical across the two JSONs

Paired design is honest: the 20 per-seed lr_baseline_aucs arrays are bit-identical (lr_baseline_aucs_match: true). The ONLY difference between the two JSONs is the loss argument passed to FoundationDownstream. 19 df gives t-critical 2.093 at α=0.05 two-sided, so the test has the power to detect a +0.005 AUC effect — none is observed.

Confidence interval narrowed as predicted. n=5 → n=20 cut the AUC CI half-width from ±0.015 to ±0.001 (~15× narrower, much more than the ~2× predicted in the n=5 write-up). The reason: the per-seed foundation AUC std dropped from 0.025 to 0.009 as n went up (more seeds dilute the rare positive outliers in the per-seed sens@99 calculation), AND the paired-diff std dropped from 0.011 to 0.002 (the two losses produce nearly-identical per-seed AUCs).

Honest verdict: NULL effect on this cohort under honest CV. Both losses land within ±0.001 AUC of each other on average, with the sens_at_spec variant running a hair below CE (Δ = −0.0005, p=0.30). The 95% CI excludes the original n=5 point estimate of +0.0075 by an order of magnitude. The previous "positive direction, not statistically significant" verdict was an artifact of the CV-leak; under honest per-patient GroupKFold the two losses are statistically indistinguishable.

Recommendation (revised, supersedes the n=5 verdict below):

  1. Keep loss="ce" as the default — the n=20 paired test gives no evidence that loss="sens_at_spec" helps on this cohort.
  2. The ablation is moot at this honest baseline AUC (0.56): neither loss extracts the signal that the LR baseline (AUC 0.97) trivially captures from the same [panel, frag] input. Until the foundation model can match the LR baseline, comparing loss functions is comparing two equally failing configurations.
  3. The opt-in loss="sens_at_spec" path remains available for cohorts where the foundation model actually trains (e.g. larger n or different fold structure). Document it as a functional, non-regressing path — but stop describing it as a "positive direction" finding; the n=20 paired test rules that out.

Why the n=5 → n=20 AUC drop?

The n=5 JSON committed at 6648c09 reported foundation AUC 0.941 ± 0.025, but re-running the same loss="ce" invocation at HEAD gives AUC 0.559. The cause is not a loss change — it's a CV-protocol change between cc24f1f and 6231fcb:

  • Before audit-2 P0-A (cc24f1f): StratifiedKFold per-sample let the model memorize the per-patient mutation-derived frag score (invariant within a pair) and predict the held-out arm of the same patient — inflating AUC by an unknown amount. Reported numbers: foundation 0.94, sens@99 0.48-0.59.
  • After audit-2 P0-A (6231fcb): GroupKFold keeps patients whole. The foundation model now honestly has no access to the partner patient's signature. Reported numbers (this n=20 run): foundation 0.56, sens@99 0.08.

The smoke test test_smoke_foundation_within_lr_baseline_band (line 195) already documents this: "under honest per-patient GroupKFold the foundation model is honestly out-performed by the sklearn LR baseline on n=40 (foundation AUC ≈ 0.57, lr_baseline AUC ≈ 0.91 on TCGA-LUAD real data)."

The paired-design property is preserved: both n=5 and n=20 use the same seeds, same panel mutations, same synthetic frag, same fold split. The ONLY thing that changed is whether StratifiedKFold or GroupKFold keeps the patient pair together. So the n=5 ce.json's [0.935, 0.98, 0.9225, 0.9175, 0.95] is exactly what you'd get today if you ran the same script with the leaky StratifiedKFold restored — but those numbers are not a real loss-comparison result, they're a CV-leak demonstration.

Files added (n=20, supersedes the n=5 ones for honest comparison)

  • results/sens_at_spec_ce_n20.json — 20-seed run, loss="ce", honest GroupKFold, foundation AUC 0.5588 ± 0.0094.
  • results/sens_at_spec_sens_n20.json — 20-seed run, loss="sens_at_spec", alpha_pos=20, honest GroupKFold, foundation AUC 0.5582 ± 0.0104.
  • results/sens_at_spec_ablation_n20.json — paired comparison with 95% CIs and verdict classification. lr_baseline_aucs_match: true.

Reproduce (n=20)

cd /Users/hermes/deepcatch
env -u PYTHONPATH ./.venv/bin/python scripts/foundation_real_smoke.py \
    --seeds 20 --loss ce --out results/sens_at_spec_ce_n20.json

env -u PYTHONPATH ./.venv/bin/python scripts/foundation_real_smoke.py \
    --seeds 20 --loss sens_at_spec --alpha-pos 20 \
    --out results/sens_at_spec_sens_n20.json

env -u PYTHONPATH ./.venv/bin/python scripts/sens_at_spec_ablation.py \
    --ce-json results/sens_at_spec_ce_n20.json \
    --sens-json results/sens_at_spec_sens_n20.json \
    --out results/sens_at_spec_ablation_n20.json

Total wall-clock: ~12 minutes per smoke run (~36 sec/seed × 20 seeds × 3-ensemble + shuffled-label control), <1 second for the ablation. Two smoke runs + ablation ≈ 25 min.

The n=5 baseline JSONs (results/sens_at_spec_ce.json, results/sens_at_spec_sens.json, results/sens_at_spec_ablation.json) and the n=5 sections below are kept as historical record of the leaky-CV-era numbers; they are no longer the source of truth.

Original n=5 ablation (LEAKY StratifiedKFold, 2026-09-21, commit 6648c09)

⚠️ This section is HISTORICAL and SUPERSEDED by the n=20 update above. The n=5 numbers below were generated under StratifiedKFold, which audit-2 P0-A (commit 6231fcb) replaced with GroupKFold to prevent per-patient information leak between train and test folds. The honest n=20 numbers above (foundation AUC ≈ 0.56 for both losses) are the source of truth for this ablation.

TL;DR (n=5, leaky CV — historical)

The loss="sens_at_spec" change moves the right metrics in the right direction on the 20-patient TCGA-LUAD panel at TF=0.1%, but the lift is not statistically significant at 5 seeds.

Metric loss="ce" loss="sens_at_spec" (α=20) Paired Δ (95% CI) p-value Verdict
Foundation AUC 0.9410 ± 0.0252 0.9485 ± 0.0245 +0.0075 [−0.0064, +0.0214] 0.208 positive, NS
Foundation Sens@99% 0.480 ± 0.323 0.590 ± 0.225 +0.110 [−0.093, +0.313] 0.207 positive, NS
Foundation Sens@95% 0.760 ± 0.075 0.780 ± 0.109 +0.020 [−0.148, +0.188] 0.757 null
LR-baseline AUC 0.959 (5/5 match) 0.959 (5/5 match) 0.0000 (deterministic) n/a no change (loss not applied)
Smoke gate n/a (advisory only) n/a (advisory only) n/a n/a policy changed
Signal-to-artifact 1.47 1.50 +0.03 n/a both well above 1.0

Honest classification: positive direction, NOT statistically significant. Both 95% CIs cross zero. The 5-seed n is too small to separate the focal-BCE signal from per-seed noise (per-seed AUC std is ~0.025; the paired diff std is ~0.011). The change does not regress the metric on any of the 5 seeds.

The smoke gate is now ADVISORY ONLY. Earlier versions of the smoke (Audit-2 P0-F) treated the foundation-tracks-LR-within-0.20-AUC check as a process-exit quality gate. On this 20-patient cohort the gate failed symmetrically for both loss modes because the foundation is honestly out-performed by the sklearn LR baseline at the AUC level (the LR is a convex fit on a 2-D score that already captures everything the design encodes, while GroupKFold denies the foundation model access to the partner patient's signature). A gate that fails identically for every ablation candidate cannot distinguish loss="ce" from loss="sens_at_spec", so it was useless as a quality signal. As of this audit, the gate is reported as gate_pass in the JSON (still the same three-way check: lr_baseline AUC ≥ gate_auc AND |foundation − lr_baseline| ≤ gate_foundation_vs_lr AND foundation − shuffled > gate_significant) but no longer affects the exit code. The smoke is now a regression test: it must not crash, all output fields must be finite, and the JSON must parse. The earlier doc text attributing the failure to lr_baseline sens@99 = 0.37 < 0.40 was a misread — the gate never tested sens@99; it tested the three AUC conditions above, and the failing condition on this cohort was `|foundation − lr|

0.20` (e.g. 0.555 vs 0.973 in the most recent 5-seed run).

Recommendation: keep loss="ce" as the default (per the constraint "don't change any defaults"), but document the alpha_pos=20 focal-BCE path as available for users who want to push high-specificity sensitivity on cohorts where CE leaves Sens@99 lagging. Re-run with ≥10 seeds before claiming statistical significance.

Scope-mismatch note (panel-LLR vs deep model)

The task description originally considered applying focal_binary_cross_entropy to the panel_llr (which is a sklearn LogisticRegression fit on the per-locus LLR sum and so does NOT take an alpha_pos weight). That comparison was discarded as ill-posed: the panel-LR is a convex LR on a pre-aggregated 1-D score and has no gradient weight to rebalance.

The honest head-to-head is the same FoundationDownstream model architecture trained with two different losses on the same real-TCGA panel-LLR + real-mutation-derived-frag channel. That is what was executed.

The LR-baseline row in the table above is included as a sanity check that the data inputs are bit-identical across the two runs (lr_baseline_aucs_match: true in the JSON). It is not part of the loss comparison because the LR baseline does not take the loss flag.

Data

Field Value
Cohort 20 TCGA-LUAD patients from validation/tcga/tcga_cache/ (402 MAF files)
Mutations/patient 19,421 total (median ~244 per patient after CADD match)
Panel size All available per-patient mutations (median min(n_variants_pos, n_variants_neg) per patient)
Tumor fraction 0.1% (TF=0.001) — ultra-low ctDNA screening regime
cfDNA depth 5,000× per locus
Background error rate 0.002 (clean baseline)
Channel 1 (panel) Real Poisson-sampled cfDNA reads at TF=0.001 vs TF=0 paired design
Channel 2 (frag) Real per-patient mutation-derived features (log burden, mean VAF, VAF std, driver-gene fraction, mutation spectrum, aneuploidy) + calibrated sequencing-noise jitter
Train/test split 5-fold StratifiedKFold, random_state=seed (per-seed deterministic)
Seeds {0, 1, 2, 3, 4}
LR baseline (sanity) sklearn LogisticRegression(C=1.0) on [panel_score, frag_score]

data_source for both runs: real_TCGA_LUAD_panel_+_real_mutation_derived_fragmentomics (both channels are real-data derived; the jitter on the frag channel is the only synthetic component, calibrated to mimic real cfDNA sequencing noise).

Method

  1. Added --loss {ce,sens_at_spec} and --alpha-pos CLI flags to scripts/foundation_real_smoke.py:580-602, plumbed through to FoundationDownstream(...) instantiations at lines 410, 441, 507.
  2. Re-ran the smoke with loss="ce" (default, preserves all existing numbers) → results/sens_at_spec_ce.json.
  3. Re-ran the same smoke with loss="sens_at_spec", alpha_pos=20.0 → results/sens_at_spec_sens.json.
  4. Wrote scripts/sens_at_spec_ablation.py that reads both JSONs, runs paired t-tests on per-seed arrays, and writes results/sens_at_spec_ablation.json.

The two runs share the same seeds, same panel mutations, same synthetic frag (seed+9999), same StratifiedKFold random_state, same ensemble seed offsets. Verified bit-identical by checking that panel-only, frag-only, LR-baseline, and naive-avg AUC arrays match across the two JSONs (all 4 lists identical, lr_baseline_aucs_match: true).

Per-seed data (audit trail)

Foundation AUCs (the primary metric)

seed ce sens_at_spec diff
0 0.9350 0.9275 -0.0075
1 0.9800 0.9900 +0.0100
2 0.9225 0.9375 +0.0150
3 0.9175 0.9375 +0.0200
4 0.9500 0.9500 0.0000

Per-seed diff: −0.0075, +0.010, +0.015, +0.020, 0.000 — 3/5 positive, 1/5 negative, 1/5 zero. Mean diff +0.0075 is well within per-seed std (0.025).

Foundation Sens@99% (the actual MRD operating point)

seed ce sens_at_spec diff
0 0.25 0.55 +0.30
1 0.85 0.90 +0.05
2 0.65 0.70 +0.05
3 0.05 0.30 +0.25
4 0.60 0.50 -0.10

Per-seed diff: +0.30, +0.05, +0.05, +0.25, −0.10 — 4/5 positive, 1/5 negative. The mean diff (+0.11) is positive but with CI [−0.093, +0.313] — wide enough that "null" is a defensible call at this sample size. Note that the per-seed sens@99 std (0.32 for CE, 0.22 for sens_at_spec) is much larger than the AUC std, so a few seeds (especially seed 3 with 0.05→0.30) drive most of the lift.

Foundation Sens@95%

seed ce sens_at_spec diff
0 0.85 0.65 -0.20
1 0.85 0.95 +0.10
2 0.70 0.85 +0.15
3 0.70 0.75 +0.05
4 0.70 0.70 0.00

Per-seed diff: −0.20, +0.10, +0.15, +0.05, 0.00 — 3/5 positive, 1/5 negative, 1/5 unchanged. Mean diff +0.02 is within noise (CI [−0.148, +0.188]).

Sanity-check invariants (paired design is honest)

The paired t-test is valid only if the two runs share the same underlying data. The following checks confirm this:

Check Match?
panel_only_aucs (per-seed) ✅ bit-identical across the two JSONs
frag_only_aucs (per-seed) ✅ bit-identical
lr_baseline_aucs (per-seed) ✅ bit-identical (5/5 seeds match to <1e-9)
naive_avg_aucs (per-seed) ✅ bit-identical
Seeds used ✅ {0,1,2,3,4} in both runs
n_cancer, n_healthy ✅ 20/20 in both
data_source ✅ identical string

The only difference between the two JSONs is the loss field in each FoundationDownstream instantiation. The lr_baseline row in the table above acts as a paired-design control: if it differed, the comparison would be invalid.

Claim-by-claim verification (AUDIT style)

# Claim Source Status
1 loss="ce" produces foundation AUC 0.9410 ± 0.0252 over 5 seeds results/sens_at_spec_ce.json:foundation_auc_mean and foundation_auc_std ✅ verified (also matches the foundation_aucs array, 5 elements)
2 loss="sens_at_spec" with alpha_pos=20 produces foundation AUC 0.9485 ± 0.0245 results/sens_at_spec_sens.json:foundation_auc_mean and foundation_auc_std ✅ verified
3 Paired Δ AUC = +0.0075, 95% CI [−0.0064, +0.0214] results/sens_at_spec_ablation.json:foundation_auc_paired_t ✅ verified (recomputable from foundation_aucs_ce and foundation_aucs_sens_at_spec)
4 Paired Δ Sens@99% = +0.110, 95% CI [−0.093, +0.313] results/sens_at_spec_ablation.json:foundation_sens_at_99_paired_t ✅ verified
5 Both runs use the same seeds, data, and CV folds lr_baseline_aucs bit-identical across the two JSONs ✅ verified (lr_baseline_aucs_match: true)
6 Smoke gate is now ADVISORY ONLY (no longer affects exit code) scripts/foundation_real_smoke.py module docstring + honest_framing string ✅ verified — gate_pass is computed and reported in JSON, but the script always returns 0 after writing the JSON
7 Foundation score is 0.7 × frozen-encoder + LR-head + 0.3 × trainable transformer scripts/foundation_real_smoke.py:460 ✅ source code cited
8 FoundationDownstream's loss="sens_at_spec" path uses focal_binary_cross_entropy with alpha_pos=20 src/foundation/downstream.py:457-471 ✅ source code cited
9 The --loss flag was plumbed into all 3 FoundationDownstream instantiations in the smoke script scripts/foundation_real_smoke.py:410, 441, 507 ✅ grep-verified
10 The change does not regress on any seed (worst case 0.0 at seed 4 for AUC, −0.10 at seed 4 for sens@99) foundation_aucs and foundation_sens_at_99_* arrays ✅ verified

What this does and does not establish

Does establish:

  • The loss="sens_at_spec" change does not BREAK the foundation model on this cohort. No regression in mean AUC; modest positive point estimate for both AUC and Sens@99.
  • The alpha_pos=20 focal-BCE path is functional and produces sensible per-seed numbers (no NaN, no degenerate all-negative collapse, no all-positive collapse).
  • The signal-to-artifact ratio is preserved (1.47 → 1.50, both well above the 1.0 threshold from the paired-design diagnostic in cfdna-early-detection-validation skill).
  • The 5-seed paired design isolates the loss effect from data, split, and seed noise.

Does NOT establish:

  • Statistical significance. With 5 seeds, a paired t-test needs |mean_diff| / std_diff > 2.776 (t-critical for df=4, α=0.05 two-sided) to reach p<0.05. AUC: 0.0075/0.0112 = 0.67 (need 2.49, observed 1.50). Sens@99: 0.110/0.164 = 0.67 (need 2.49, observed 1.50). Both fall short of significance.
  • That the lift generalizes to other cohorts, VAFs, or panel sizes. The 20-patient cohort is the only data point.
  • That the alpha_pos=20 value is optimal. The 5-seed sweep is not informative enough to pick an alpha; a proper alpha sweep would need ≥10 seeds × ≥3 alpha values × 5-fold CV (≥150 foundation model fits) and is out of scope here.

What would establish statistical significance:

  • ≥10 seeds (n=10 paired t gives t-critical 2.262 at α=0.05, more power; an n=20 paired t gives 2.093, near "easy" significance). The current CI of ±0.015 AUC is roughly half the effect size needed; doubling n would cut the CI to ±0.011, making a true +0.0075 effect still ambiguous and a true +0.015 effect detectable.
  • A pre-registered alpha sweep (e.g. α ∈ {5, 10, 20, 50, 100}) with ≥5 seeds per α to identify the optimum and the noise floor.

Recommendation

Keep loss="ce" as the default (per the constraint to not change defaults). The new code path is available and not regressing, but the lift is not yet statistically established at the 5-seed n. Re-running with 10–20 seeds is the cheapest next step to convert the positive point estimate into a defensible "this helps" claim.

Files

  • scripts/foundation_real_smoke.py — added --loss and --alpha-pos flags; plumbed into all 3 FoundationDownstream(...) calls.
  • scripts/sens_at_spec_ablation.py — paired t-test driver, reads both JSONs, writes the ablation JSON.
  • results/sens_at_spec_ce.json — 5-seed run, loss="ce".
  • results/sens_at_spec_sens.json — 5-seed run, loss="sens_at_spec", alpha_pos=20.
  • results/sens_at_spec_ablation.json — paired comparison with 95% CIs and verdict classification.
  • docs/SENS_AT_SPEC_ABLATION.md — this document.

Reproduce

cd /Users/hermes/deepcatch
env -u PYTHONPATH ./.venv/bin/python scripts/foundation_real_smoke.py \
    --seeds 5 --n-patients 20 --loss ce \
    --out results/sens_at_spec_ce.json

env -u PYTHONPATH ./.venv/bin/python scripts/foundation_real_smoke.py \
    --seeds 5 --n-patients 20 --loss sens_at_spec --alpha-pos 20 \
    --out results/sens_at_spec_sens.json

env -u PYTHONPATH ./.venv/bin/python scripts/sens_at_spec_ablation.py

Total wall-clock: ~3 minutes per smoke run (1 min TCGA load + 2 min for 5 seeds × 5 folds × 3-ensemble foundation model), 1 second for the ablation. Two smoke runs + ablation = ~7 minutes.