GMVVariance mishandles singular covariance via pseudoinverse normalization

[microprediction]

Summary

At current main commit 4b26c0c3e38e7fc7f868981c61a576880f970e86, GMVVariance forms portfolio weights as:

w = try_invert(cov) @ np.ones(len(cov))
w = w / w.sum()

try_invert falls back to the Moore–Penrose pseudoinverse for a singular covariance. For singular positive-semidefinite matrices, w ∝ Σ⁺1 does not generally solve the advertised constrained problem: minimize wᵀΣw subject to 1ᵀw = 1.

It can ignore a zero-risk feasible direction, is discontinuous as an eigenvalue reaches zero, and can return NaN or roundoff-driven leveraged weights. This affects the shipped economic assessor and the real-equity/recommender research that relies on it.

This is distinct from issue #88: the inputs here are valid PSD covariance matrices, not indefinite matrices.

Relevant code and callers

• precise/_linalg.py:110-118: try_invert uses np.linalg.pinv after inv fails.
• precise/assessment/assessors.py:140-151: GMVVariance normalizes try_invert(cov) @ 1 without a singular-case or zero-denominator policy.
• research/oos_equity.py:36-38: the real-equity rolling experiment uses the same construction.
• research/oos.py:15-17, 103+: the recommender validation describes GMV as its default economic judge.
• research/turnover.py:60-68 already uses a different policy: it adds a ridge and falls back to equal weight when normalization is unstable. Thus the research paths are inconsistent.
• tests/test_correctness.py:347-352 checks only that the pseudoinverse is finite, not that a downstream GMV portfolio is meaningful.
• tests/test_assessment.py tests only strictly positive-definite candidates.

Exact counterexample: discontinuity and wrong optimizer

Let Σ(ε) = diag(1, 1, ε).

For every ε > 0, the unique GMV weights are:

w(ε) = (ε, ε, 1) / (1 + 2ε)  →  (0, 0, 1).

At ε = 0, (0, 0, 1) remains feasible and has exactly zero variance, so it is a GMV solution. But the pseudoinverse drops the null direction:

Σ(0)⁺ 1 = (1, 1, 0),

and the implementation returns (1/2, 1/2, 0), whose estimated variance is 1/2.

Reproduction:

import numpy as np
from precise._linalg import try_invert

for eps in [1e-2, 1e-6, 1e-12, 0.0]:
    cov = np.diag([1.0, 1.0, eps])
    w = try_invert(cov) @ np.ones(3)
    w /= w.sum()
    print(eps, w, w @ cov @ w)

Observed:

0.01   [0.00980392 0.00980392 0.98039216]  0.0098039216
1e-06  [9.99998e-07 9.99998e-07 9.99998e-01] 9.99998e-07
1e-12  [1e-12 1e-12 1.0]                  1e-12
0.0    [0.5 0.5 0.0]                      0.5

The output jumps at the exact boundary rather than converging.

Other verified failures

Zero covariance

import numpy as np
from precise.assessment import GMVVariance

cov = np.zeros((3, 3))
score = GMVVariance().score(cov, X_test=np.ones((4, 3)))
print(score)

Observed: nan, with an invalid-divide warning, because both Σ⁺1 and its sum are zero.

Nullspace contains the budget direction

For:

cov = np.array([[1.0, -1.0],
                [-1.0, 1.0]])

the equal-weight portfolio is in the nullspace and has zero variance. Numerically, pinv(cov) @ ones is approximately [-8.3e-17, 5.6e-17]; normalizing that roundoff produced weights [3, -2] in the executed reproduction. The result can therefore depend on LAPACK-level rounding rather than the optimization problem.

Impact for equity work

Singular covariance estimates are not exceptional here:

• p >= n empirical windows;
• newly listed, halted, stale, or constant-price assets;
• duplicate share classes or nearly replicated instruments;
• factor/residual panels with exact linear constraints;
• estimators that deliberately preserve structural null directions (the nonlinear-shrinkage documentation explicitly says they do not invent variance there).

A NaN assessor score can contaminate rankings. A finite but discontinuous pseudoinverse portfolio can sharply mis-rank two estimators or make the real-equity validation dependent on whether an eigenvalue is exactly zero versus 1e-12.

Suggested direction

The intended singular policy needs to be explicit. Two defensible choices have different meanings:

1. Exact unconstrained GMV: solve the equality-constrained PSD quadratic problem including its nullspace. If the nullspace contains a vector with nonzero sum, a normalized nullspace vector is a zero-variance solution. Otherwise the pseudoinverse formula applies. Non-uniqueness needs a tie-break rule.
2. Regularized economic probe: apply an explicit, scale-relative ridge/shrinkage and solve the SPD system. This deliberately avoids zero-variance overfit and is probably more useful for estimator assessment, but the regularization is part of the assessor and should be parameterized/documented.

Either choice should guard the normalization denominator and guarantee a finite score or raise a clear validation error. A fixed absolute ridge such as the 1e-10 in research/turnover.py would reintroduce unit dependence; a trace/eigenvalue-relative ridge would preserve scale equivariance.

It may be better to introduce one shared gmv_weights helper and use it consistently in GMVVariance, research/oos_equity.py, and research/turnover.py.

Suggested tests

1. Continuity or explicitly documented regularization behavior for diag(1, 1, ε) through ε = 0.
2. All-zero covariance returns documented finite weights/score or a clear error, never NaN.
3. [[1, -1], [-1, 1]] does not generate roundoff-driven leveraged weights.
4. A p > n empirical covariance yields deterministic finite behavior.
5. Global rescaling of Σ does not change weights.
6. All research and assessment callers use the same declared policy.

Uncertainty

There is no unique economically correct portfolio for every singular covariance: exact nullspace exploitation may be mathematically correct but statistically undesirable. The verified defect is narrower—the current pseudoinverse-and-normalize rule is neither a general solution of the stated GMV problem nor a stable documented regularization, and it demonstrably returns discontinuous, NaN, or roundoff-driven results.

Searches of current issues and pull requests found no existing report for this singular-GMV behavior.