Skip to content

Fix stale trainer reuse and fold leakage in recursive backtesting - #44

Open
LRod1on wants to merge 1 commit into
sb-ai-lab:mainfrom
LRod1on:fix/recursive-backtest-trainer
Open

LRod1on wants to merge 1 commit into
sb-ai-lab:mainfrom
LRod1on:fix/recursive-backtest-trainer

Conversation

@LRod1on

@LRod1on LRod1on commented Nov 11, 2026

Copy link
Copy Markdown

Problem

Strategy.back_test() repeatedly calls fit() and predict() on the same strategy instance. RecursiveStrategy.fit() appends each newly fitted trainer, while both prediction paths always use self.trainers[0].

The backtest generator processes the latest time window first, so subsequent, earlier windows are predicted using the first trainer, which has already seen those windows during training. This leaks future data across outer backtest folds and can make the reported quality overly optimistic. Repeated standalone fit() calls also keep using the original trainer, paired with the refitted pipeline.

This affects RecursiveStrategy (including reduced prediction), MIMOStrategy, and FlatWideMIMOStrategy through inheritance. It is separate from the trainer's internal validation splits.

Fix

Replace the stored trainer list with [current_trainer] after a successful fit. Both existing prediction paths then use the trainer from the latest fit, without retaining obsolete trainers. No prediction API changes are needed.

Changing both prediction sites to self.trainers[-1] would also select the latest trainer, but would retain every previous trainer without its corresponding fitted pipeline. Replacing the list matches the single-trainer design of RecursiveStrategy and the reset-on-fit behavior already used by DirectStrategy.

Regression tests

Use the real dataset, pipeline, MLTrainer, and backtest code with a deterministic estimator that predicts the maximum target seen during training. For an increasing series, each outer fold must predict its own training maximum rather than the first fold's maximum. A separate refit test shifts the training targets by 100 and checks that predictions change accordingly.

Both tests cover ordinary recursive prediction, reduced prediction, MIMO, and FlatWideMIMO. The test validator uses all supplied training samples to isolate the outer backtest lifecycle from internal validation.

Validation

  • Before the fix: all 8 new regression cases fail with stale predictions.
  • After the fix: all 8 regression cases pass.
  • Full suite on Python 3.11.1: 55 passed (existing pandas warnings remain).
  • Ruff 0.9.1 lint and formatting checks pass for both changed files.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant