Repository navigation
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Strategy.back_test()repeatedly callsfit()andpredict()on the same strategy instance.RecursiveStrategy.fit()appends each newly fitted trainer, while both prediction paths always useself.trainers[0].The backtest generator processes the latest time window first, so subsequent, earlier windows are predicted using the first trainer, which has already seen those windows during training. This leaks future data across outer backtest folds and can make the reported quality overly optimistic. Repeated standalone
fit()calls also keep using the original trainer, paired with the refitted pipeline.This affects RecursiveStrategy (including reduced prediction), MIMOStrategy, and FlatWideMIMOStrategy through inheritance. It is separate from the trainer's internal validation splits.
Fix
Replace the stored trainer list with
[current_trainer]after a successful fit. Both existing prediction paths then use the trainer from the latest fit, without retaining obsolete trainers. No prediction API changes are needed.Changing both prediction sites to
self.trainers[-1]would also select the latest trainer, but would retain every previous trainer without its corresponding fitted pipeline. Replacing the list matches the single-trainer design of RecursiveStrategy and the reset-on-fit behavior already used by DirectStrategy.Regression tests
Use the real dataset, pipeline, MLTrainer, and backtest code with a deterministic estimator that predicts the maximum target seen during training. For an increasing series, each outer fold must predict its own training maximum rather than the first fold's maximum. A separate refit test shifts the training targets by 100 and checks that predictions change accordingly.
Both tests cover ordinary recursive prediction, reduced prediction, MIMO, and FlatWideMIMO. The test validator uses all supplied training samples to isolate the outer backtest lifecycle from internal validation.
Validation