diff=True drops the checkpoint-boundary observation on get_state/set_state resume

[microprediction]

Summary

At current main commit 4b26c0c3e38e7fc7f868981c61a576880f970e86, every registered positional estimator with diff=True loses the return/difference that crosses a get_state() / set_state() checkpoint boundary.

The base class keeps the last raw observation in self._prev_x and needs it to form the next difference (base.py:50-62). But:

• get_state() serializes only self._state;
• set_state() restores only that dict and never restores or clears _prev_x (base.py:140-158);
• BlockCovariance's custom JSON state path has the same omission (block_covariance.py:95-138).

A fresh restored estimator therefore sees _prev_x is None, stores the first post-restore level, and skips it. Restoring into an already-used object is worse: its stale local _prev_x survives and can create a difference against an unrelated stream.

This contradicts the README's statement that state can be checkpointed mid-stream with get_state()/set_state() (README line 34) and the JOSS draft's claim that an entire stream can be serialized and resumed in another process (paper lines 89-94).

Deterministic reproduction

import json
import numpy as np
from precise import EmpiricalCovariance

# Differences are [1, 2, 100, 4, 5].
X = np.array([[0.], [1.], [3.], [103.], [107.], [112.]])

full = EmpiricalCovariance(diff=True).fit(X)

before = EmpiricalCovariance(diff=True).fit(X[:3])
state = json.loads(json.dumps(before.get_state()))
resumed = EmpiricalCovariance(diff=True).set_state(state)
resumed.partial_fit(X[3:])

print(full.n_samples_, full.location_, full.covariance_)
# 5 [22.4] [[1507.44]]

print(resumed.n_samples_, resumed.location_, resumed.covariance_)
# 4 [3.] [[2.5]]

The boundary difference 103 - 3 = 100 is silently omitted. I also ran this count check across all 20 classes returned by all_estimators(): each had full.n_samples_ == 5 and resumed.n_samples_ == 4 under diff=True.

A related reset failure is also exact:

e = EmpiricalCovariance(diff=True).fit([[0.], [10.]])
e.set_state(None)
e.partial_fit([20.])
print(e.n_samples_, e.location_)  # 1 [10.], from stale 20 - 10

A cleared estimator should merely remember the first new level and still have zero differenced observations. Instead, set_state(None) retains the previous stream's _prev_x.

Impact

For equity workflows using diff=True on levels/prices, the omitted observation is exactly the price change spanning a service restart. An overnight jump, split adjustment, or market shock at that boundary can disappear from the covariance history while the API reports a successful restore. The sample count also becomes wrong by one per restart.

This is not a numerical approximation or a concern about estimator definition; uninterrupted and resumed execution are observably different.

Existing behavior and test gap

Pickle round-trips correctly because __getstate__ explicitly includes prev_x (base.py:160-173). The advertised plain-dict path is the broken one.

The shared state tests compare the estimate immediately after restore but do not feed a subsequent observation, and instantiate every estimator with its default diff=False (tests/test_estimators.py:64-71). The JSON test has the same gap.

Suggested direction

Include the last raw observation in the JSON-friendly checkpoint whenever diff=True, restore it into self._prev_x, and always clear _prev_x in set_state(None). The custom BlockCovariance state path needs the same treatment. Then add a continuation-equivalence test:

1. split a level stream at a large boundary move;
2. JSON round-trip get_state();
3. continue both uninterrupted and restored estimators;
4. require equal n_samples_, location_, and covariance_ for every registered estimator under diff=True.

Old checkpoints that lack the raw previous observation cannot be resumed exactly; that compatibility limitation should be handled explicitly rather than silently skipping a difference.

I checked existing issues and PRs before filing. PR #77 mentions this defect under “not fixed here,” but there is no open or closed issue or implementation PR tracking it.