Zero-variance correlations break the unit diagonal and undo ConditionalCovariance's floor

[microprediction]

Summary

At current main commit 4b26c0c3e38e7fc7f868981c61a576880f970e86, cov_to_corrcoef does not preserve the package's documented unit-diagonal correlation contract when a feature has zero (or sufficiently underflowed) variance.

For a constant feature, 9 of the 20 registered estimators return a diagonal correlation entry of zero or nearly zero. This is more than a display convention: ConditionalCovariance consumes that result as (R) in (H=DRD), so it multiplies away its own volatility floor and can produce subnormal variances and enormous precision entries.

This is a follow-up boundary case to PR #79: that fix correctly made the denominator floor relative and repaired small-but-positive variances, but the exact-zero case still cannot produce a unit diagonal by division alone.

Relevant code and contracts

• precise/_linalg.py:22-39: cov_to_corrcoef floors the denominator but returns a / denominator. If a[i,i] == 0, the result is still 0 / floor == 0.
• precise/base.py:97-98: every positional estimator exposes that result as correlation_.
• precise/conditional.py:143-159: ConditionalCovariance floors each forecast variance at 1e-12, obtains R = self._corr_model.correlation_, and returns R * outer(d, d).
• precise/keyed.py:68-71, 87-91: the keyed adapters already call np.fill_diagonal(corr, 1.0), so keyed and positional views of the same covariance use different conventions.
• README lines 26-28 and the registry contract test describe correlation_ as unit-diagonal.

The constant-feature test at tests/test_correctness.py:273-282 checks only that correlation is finite; the unit-diagonal registry test uses full-rank random data.

Reproduction 1: registry contract

import numpy as np
from precise import all_estimators

rng = np.random.default_rng(20260925)
X = rng.standard_normal((400, 3))
X[:, 1] = 5.0

for cls in all_estimators():
    est = cls().fit(X)
    d = np.diag(est.correlation_)
    if not np.allclose(d, 1.0):
        print(cls.__name__, d, np.diag(est.covariance_))

Observed offenders (9/20):

EmpiricalCovariance                   [1. 0. 1.]
EwaCovariance                         [1. 0. 1.]
AdaptiveEwaCovariance                 [1.0 9.68e-19 1.0]
ShrunkCovariance                      [1. 0. 1.]
NonlinearShrinkageCovariance          [1. 0. 1.]
WindowedNonlinearShrinkageCovariance  [1.0 5.73e-19 1.0]
EwaNonlinearShrinkageCovariance       [1. 0. 1.]
SchurConditionalCovariance            [1.0 8.96e-09 1.0]
HuberCovariance                       [1. 0. 1.]

The other estimators happen to add a positive floor or shrinkage before normalization, rather than satisfying the normalization contract uniformly.

Reproduction 2: composition defeats its variance floor

import numpy as np
from precise import ConditionalCovariance, EwaCovariance

n = 10_000
X = np.column_stack([np.sin(np.arange(n)), np.full(n, 5.0)])

est = ConditionalCovariance(
    vol=EwaCovariance(r=0.05),
    corr=EwaCovariance(r=0.05),
).fit(X)

print("vol state:", est._state["var"])
print("R diagonal:", np.diag(est._corr_model.correlation_))
print("H diagonal:", np.diag(est.covariance_))
print("precision diagonal:", np.diag(est.precision_))

Observed:

vol state:          [5.29056524e-01, 1.00000000e-12]
R diagonal:         [1.00000000e+00, 5.63352662e-211]
H diagonal:         [5.29056524e-01, 5.63352662e-223]
precision diagonal: [1.89015720e+00, 1.77508702e+222]

Thus the explicit 1e-12 volatility floor is reduced by another factor of about (5.6 imes10^{-211}). A halted or stale equity that later moves can consequently generate an astronomically large Mahalanobis contribution even though the composition code appears to have bounded that risk.

Expected behavior / suggested direction

There are two defensible mathematical statements:

1. correlation involving a zero-variance random variable is undefined; or
2. for a matrix API, use the standard correlation-matrix convention: unit diagonal and zero off-diagonal entries for a zero-variance coordinate.

The package, README, tests, keyed adapters, and (H=DRD) composition already choose convention (2). The positional helper should apply it consistently, for example by setting the diagonal to exactly one after safe normalization. For a valid PSD covariance, a truly zero diagonal implies zero entries across that row/column, so this does not invent cross-correlation.

If cov_to_corrcoef is also intended to accept indefinite matrices, that should be handled separately; overwriting the diagonal should not be used to disguise an invalid covariance.

Suggested tests

1. Extend the registry constant-feature test to require diag(correlation_) == 1.
2. Test cov_to_corrcoef(diag([1, 0, 2])) explicitly: finite, symmetric, unit diagonal, zero correlations involving the constant coordinate.
3. Assert keyed and positional correlation conventions agree.
4. For ConditionalCovariance, verify diag(H) equals the stored/floored per-series variances even after a long constant stretch.
5. Verify a small post-halt move is bounded by the documented variance floor rather than a subnormal covariance.

Uncertainty

The unit diagonal at exactly zero variance is necessarily a convention because the probabilistic correlation is undefined. The verified correctness problem is that the library already advertises and relies on that convention, but the shared positional path violates it and breaks the explicit (H=DRD) variance floor.

Searches of current issues and pull requests found no existing report of this exact-zero boundary or its ConditionalCovariance consequence.