Add chance-aware detection metrics

[be-student]

Closes #736.

Problem

evaluate_print reports precision at rank n without its chance-level baseline, so the same precision value can mean different things at different outlier prevalences. Its old test passed scaled labels as scores, while seven time-series detector tests did not verify that scores separated their fixture labels.

Changes

• Add and publicly export detection_lift, defined as precision at rank n divided by outlier prevalence. Return 0.0 with no outliers and reject empty ground truth.
• Print chance level and lift from evaluate_print; make precision_n_scores return zero without an UndefinedMetricWarning when no positives are predicted.
• Test the printed output and metric edge cases.
• Add a shared assertion to seven time-series fixtures. It rejects scores that become constant after rounding to eight decimals and now requires detection_lift > 2.7.
• Align the KShape, MatrixProfile, and SAND fixtures with their subsequence scoring semantics by using mixed point/subsequence anomalies and 10-sample windows.
• Normalize pyod/utils/data.py line endings to avoid a mixed CRLF/LF diff.

Review follow-up

The reviewer measured that the former > 1.1 threshold accepted random scores too often. Commit 577f1ea raises it to > 2.7. In fresh seeded simulations with 10,000 random scorers per fixture size, acceptance fell from 54.57% to 3.36% at n=300 and from 35.49% to 3.02% at n=500. The three previously tabulated subsequence detector lifts are 6.75 (KShape), 10.91 (MatrixProfile), and 10.23 (SAND).

The first hosted run at 577f1ea found one further fixture mismatch: TimeSeriesOD::test_fit_univariate uses isolated point anomalies but had a 20-sample scoring window. Its lift sometimes fell below 2.7 on macOS/Python 3.9 and Ubuntu/Python 3.10. Commit 0d5ba5241108ff89b87bd18ce766a49ead3e6675 changes only that test's window to 10 samples, preserving the detector, labels, and strict bound. Across 100 seeded detector runs on each of local Python 3.9 and 3.10, the minimum lift was 4.0, with no bound failures. This keeps the quality assertion meaningful while matching its point-anomaly resolution.

An earlier head failed the KShape, MatrixProfile, and SAND tests across the hosted matrix because point-only fixtures with 20-sample windows produced broad tied score plateaus. The corrected subsequence fixtures passed all eleven Linux, Windows, and macOS matrix jobs, packaging, and CodeQL at head 4696fbe. Those hosted results predate the threshold increase and are not claimed for the new head.

Validation

• At current head 0d5ba52 on macOS/Python 3.14.3, 57 tests across the seven time-series detector modules and utility tests passed with PyTorch installed.
• The 11 focused lift/metric tests passed on local Python 3.9, 3.10, and 3.14 with PyTorch installed. A broader Python 3.12 local run reached 56 passes and one unrelated Pillow binary-signature import failure; the 11 focused tests passed there as well.
• One hundred seeded IForest runs on each of Python 3.9 and 3.10 yielded minimum lift 4.0 for the adjusted point-anomaly test fixture.
• The seeded random-score check ran 10,000 trials at each of n=300 and n=500, with the acceptance rates above.
• git diff --check passed.
• The replacement hosted run passes all 14 reported checks on 0d5ba52: eleven Linux, Windows, and macOS matrix jobs, packaging, CodeQL, and Python analysis. The earlier 577f1ea run's macOS/Python 3.9 and Ubuntu/Python 3.10 failures came from the 20-sample TimeSeriesOD fixture described above.

AI assistance disclosure

OpenAI Codex assisted with implementation, CI-log analysis, review-feedback remediation, and automated testing. No manual-testing or human-review claim is made.

[mateenali66]

the shared helper is weaker than the deepsvdd guard it generalises, and it lets the degenerate case through about half the time.

test_deepsvdd.py::test_scores_are_not_constant counts uniques after np.round(scores, 8), and the comment there says why, "a ROC floor can still be met by floating-point noise in the ordering". assert_scores_separate_labels drops the rounding and then uses auc > 0.5, which is that floor.

constant detector, 300 samples, 10% contamination:

s = np.full(n, 0.5) + rng.normal(0, 1e-17, n)

np.unique(s).size is 2, since float64 spacing near 0.5 is about 1.1e-16 and the noise nudges a couple of samples onto the next representable value. that clears > 1. auc then falls either side of 0.5 by luck. across 2000 seeds assert_scores_separate_labels accepts this detector on 970 of them, 48.5%. the deepsvdd guard rejects all 2000, because rounding to 8dp collapses it back to one unique value.

the fix is already in this PR. assert detection_lift with a margin instead of auc > 0.5. lift is prevalence-baselined, so one threshold means the same thing across all seven fixtures, and it is the metric the PR is adding. carrying over the np.round(scores, 8) from deepsvdd closes the constant check.

[cindysteward]

noticed this file isCRLF throughout, so every untouched or removed line carries a trailing \r. Butthe 3 new import lines here and the new lines further down in evaluate_print are LF-only. Worth normalising, otherwise the diff carries mixed line endings forward, future edits to this file could show spurious changes on every line

[cindysteward]

the shared helper is weaker than the deepsvdd guard it generalises, and it lets the degenerate case through about half the time.

test_deepsvdd.py::test_scores_are_not_constant counts uniques after np.round(scores, 8), and the comment there says why, "a ROC floor can still be met by floating-point noise in the ordering". assert_scores_separate_labels drops the rounding and then uses auc > 0.5, which is that floor.

constant detector, 300 samples, 10% contamination:

s = np.full(n, 0.5) + rng.normal(0, 1e-17, n)

np.unique(s).size is 2, since float64 spacing near 0.5 is about 1.1e-16 and the noise nudges a couple of samples onto the next representable value. that clears > 1. auc then falls either side of 0.5 by luck. across 2000 seeds assert_scores_separate_labels accepts this detector on 970 of them, 48.5%. the deepsvdd guard rejects all 2000, because rounding to 8dp collapses it back to one unique value.

the fix is already in this PR. assert detection_lift with a margin instead of auc > 0.5. lift is prevalence-baselined, so one threshold means the same thing across all seven fixtures, and it is the metric the PR is adding. carrying over the np.round(scores, 8) from deepsvdd closes the constant check.

reproduced this with afresh RNG (numpy Generator, 2000 seeds), same setup, n=300, 10% contamination, noise at 1e-17 around 0.5. Raw np.unique(scores).size > 1 passed 1114/2000, 55.7%. Rounding to 8dp first passed 0/2000, matching the deepsvdd guard. Numbers differ a bit from yours, likely RNG choice, mechanism and conclusion both hold. So yes, assert detection_lift with a margin plus the round(scores, 8) step, it's prevalence-baselined so one threshold means the same thing across all 7

[mateenali66]

The lift bound lets a random scorer through. Pure random scores clear > 1.1 on 54.4% of runs against the four n=300 fixtures and 36.0% against the three n=500 ones, 20k trials each. Lift is quantised by the outlier count, so > 1.1 is really "one hit in the top 15", or two in the top 25.

Measured what the detectors actually score: ts_od 3.0, kshape 6.8, sand 10.2, matrix_profile 10.9, and spectral_residual, lstm and anomaly_transformer all 20.0. So > 2.7 passes all seven and drops random to about 3%.

ts_od at 3.0 is the floor and 2.7 sits close under it. Raise the bound, or grow the fixtures so a bound has somewhere to sit?

The round(scores, 8) half is doing its job. Constant detector 0 of 2000, against 1109 and 1471 before.

[be-student]

Thanks for quantifying the false-accept rate. I raised the shared lift bound from > 1.1 to > 2.7 in 577f1ea, leaving the rounded-score guard and fixtures unchanged. All 57 affected detector/utility tests pass on the new head. A fresh seeded check of 10,000 random scorers at each fixture size reduced acceptance from 54.57% to 3.36% (n=300) and from 35.49% to 3.02% (n=500). The new-head hosted matrix is still pending; the PR description now distinguishes those results from the earlier green matrix.

[be-student]

The first new-head matrix exposed a real issue in the TimeSeriesOD quality test: on macOS/Python 3.9 and Ubuntu/Python 3.10, its 20-sample window spread isolated point anomalies enough for the lift to fall below the new 2.7 bound. I kept the strict bound and the same labels/detector, and changed only that test's scoring window to 10 samples in 0d5ba52. Across 100 seeded detector runs on each of local Python 3.9 and 3.10, the minimum lift was 4.0; the focused tests pass on Python 3.9, 3.10, 3.12, and 3.14, and all 57 affected tests pass on Python 3.14. The PR description distinguishes the failed prior run from the new head; hosted CI for 0d5ba52 is pending.

[coveralls]

Coverage Report for CI Build 36094929379

Warning

Build has drifted: This PR's base is out of sync with its target branch, so coverage data may include unrelated changes.
Quick fix: rebase this PR. Learn more →

Coverage increased (+0.08%) to 92.657%

Details

• Coverage increased (+0.08%) from the base build.
• Patch coverage: 21 uncovered changes across 1 file (314 of 335 lines covered, 93.73%).
• 48 coverage regressions across 16 files.

Uncovered Changes

File	Changed	Covered	%
pyod/utils/data.py	282	261	92.55%
Total (13 files)	335	314	93.73%

Coverage Regressions

48 previously-covered lines in 16 files lost coverage.

Top 10 Files by Coverage Loss	Lines Losing Coverage	Coverage
pyod/cli.py	20	27.59%
pyod/models/base_dl.py	5	93.57%
pyod/models/ts_kshape.py	4	90.66%
pyod/test/test_cli.py	4	97.86%
pyod/test/test_knn.py	3	98.88%
pyod/test/test_vae.py	2	98.35%
pyod/models/copod.py	1	90.0%
pyod/models/ecod.py	1	90.28%
pyod/test/test_base_dl.py	1	96.88%
pyod/test/test_deepsvdd.py	1	99.39%

---

Coverage Stats

Relevant Lines:	20862
Covered Lines:	19330
Line Coverage:	92.66%
Coverage Strength:	0.93 hits per line

---

💛 - Coveralls

[be-student]

The replacement run for 0d5ba52 has finished with all 14 reported checks passing, including macOS/Python 3.9 and Ubuntu/Python 3.10, the two platforms that failed on the previous head: hosted run. The PR description now records the final matrix result and the fixture correction separately.