Skip to content

fix: restrict Hessian API to second-order DDScalar - #72

Merged
oberbichler merged 1 commit into
mainfrom
fix/hessian-api-order-2
Jul 26, 2026
Merged

oberbichler merged 1 commit into
mainfrom
fix/hessian-api-order-2

Conversation

@oberbichler

Copy link
Copy Markdown
Owner

Problem

The Hessian accessors index into m_data behind the gradient block:

return self.m_data[1 + self.size() + i];

A first-order scalar has no such storage — its data length is 1 + size. The header exposed h / set_h / hm / set_hm unconditionally, and common.h bound them for both orders (only from_gradient/from_arrays were guarded by order). So every Hessian access on a first-order type read or wrote past the end of the object, reachable from pure Python:

v = hj.D3Scalar.variable(0, 1.0)   # data length 4
v.hm()
# [[0.00e+000, 5.83e-320, 9.88e-324],
#  [5.83e-320, 2.57e+151, 1.06e-153],   <- foreign memory
#  [9.88e-324, 1.06e-153, 8.19e+247]]

hj.DScalar.variable(0, 1.0, 3).set_hm(np.full((3, 3), 7.0))
# writes 6 doubles past a 4-element heap buffer, no error

The asserts in h() are compiled out in release wheels, so nothing caught it.

It also surfaced through the documented API: hj.dd() probes hasattr(v, 'hm') (main.cpp:69) and therefore returned foreign memory for first-order values instead of an empty Hessian.

Fix

  • Header — requires (order() == 2) on h(i), h(i, j), set_h(i, v), set_h(i, j, v), hm(mode), hm(mode, out), set_hm(value).
  • Bindings — the four defs moved under if constexpr (T::order() == 2).

Putting the constraint on the header is what makes the guarantee: the binding can no longer drift away from it, and misuse in C++ becomes a compile error instead of silent UB. It also replaces one more static_assert-style guard with a real constraint (cf. TODO.md → Concepts).

Tests

Both were written first and observed failing.

C++ (test/src/test.cpp) — concepts probe the constraint directly at the header level, independent of the bindings. Before the fix all 8 first-order checks reported true:

test.cpp:513: ERROR: CHECK_FALSE( HasHessianEntry<DDScalar<1, double, 3>> ) is NOT correct!
  values: CHECK_FALSE( true )
... 8 failures

Python (python/tests/test_DDScalar.py) — test_hessian_access asserts AttributeError for first-order types (matching the existing test_pad_right convention) and keeps asserting real values for second order; test_dd_of_first_order pins hj.dd() to np.empty((n, 0, 0)), consistent with test_dd_of_scalar. The hj.dd failure showed the leak plainly:

(shapes (2, 2, 2), (2, 0, 0) mismatch)
 ACTUAL: array([[[ 0.      ,  1.414214],    <- u2's value, from the neighbouring object
        [ 1.414214, -0.353553]], ...

After: C++ 40/40 test cases, 400 assertions · Python 147 passed · clang-format --Werror and ruff format --check clean.

Second-order behaviour is unchanged — the existing test_values / test_ndarray / test_dd assertions on h, set_h, hm, set_hm all still pass untouched.

Note

Found while reviewing the whole project. Related follow-ups, in rough order of severity, that this PR deliberately does not touch:

  • h(i, j) / set_h(i, j, v) are still unbounded in release builds — v.h(-5, -5) reads arbitrary memory from Python. The binding layer needs a real range check; assert is the wrong mechanism at a language boundary.
  • DDScalar<1, double, Dynamic>{1.0, 2.0, 3.0} segfaults (m_data is an empty vector when std::copy writes into it).
  • x *= x and x /= x produce wrong derivatives (self-aliasing) in both DDScalar and SScalar.
  • resize() silently scrambles the Hessian layout for order 2; pad_right() does it correctly.

The Hessian accessors index into m_data behind the gradient block. On a
first-order scalar that storage does not exist, so every one of them read
or wrote past the end of the object.

The C++ header exposed h/set_h/hm/set_hm unconditionally, and common.h
bound them for both orders — only from_gradient/from_arrays were guarded
by order. Reachable from pure Python:

    v = hj.D3Scalar.variable(0, 1.0)   # data length 4
    v.hm()                             # reads 6 doubles past the array
    hj.DScalar.variable(0, 1.0, 3).set_hm(np.full((3, 3), 7.0))
                                       # writes 6 doubles past the heap buffer

The asserts in h() are compiled out in release wheels, so nothing caught
it. It also surfaced through the documented API: hj.dd() probes
hasattr(v, 'hm') and therefore returned foreign memory for first-order
values instead of an empty Hessian.

Constrain the accessors with `requires (order() == 2)` in the header and
bind them under `if constexpr (T::order() == 2)`. The constraint on the
header is what makes the guarantee — the binding can no longer drift away
from it, and misuse in C++ is now a compile error rather than silent UB.
@oberbichler
oberbichler merged commit 83a6a15 into main Jul 26, 2026
16 checks passed
@oberbichler
oberbichler deleted the fix/hessian-api-order-2 branch July 26, 2026 19:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant