fix: validate indices and sizes instead of relying on asserts

[oberbichler]

Problem

Every index and size that reaches the dynamic code path was validated only via assert, which is compiled out in release wheels. From pure Python that meant out-of-bounds reads, out-of-bounds writes and two hard crashes. All of the following were reproduced against the installed release build:

Entry point	Before
u.h(-5, -5)	1.5e-323 — arbitrary memory
u.h(1000, 1000)	arbitrary memory
u.set_h(1000, 1000, 42.0) (static)	OOB write, returns None
d.set_h(1000, 1000, 42.0) (dynamic)	SIGABRT — malloc detected the corruption
d.eval([])	SIGSEGV
d.eval([1]) at size 2	silent wrong result
d.set_hm(np.ones((2, 2))) at size 3	reads the 2×2 input as 3×3
hj.DDScalar(f=1, g=[1,2,3], hm=np.ones((2,2)))	silently constructed, Hessian is garbage
hj.DDScalar.variable(i=100, f=1.0, size=3)	OOB write via g(i)
hj.DDScalar.constant(1.0, 3).pad_right(1)	OOB write, silently truncates
hj.DDScalar.empty(size=-5)	object with size == -5

Static types were mostly protected already, because pybind11 rejects the wrong shape at the caster level (FixedSize(n), [n, n]) — which is why only the dynamic variants crash.

Fix

The split follows what each function is:

• Element accessors — g(i), h(i), h(i, j) are called in loops, so they keep precondition semantics like std::vector::operator[] and stay free of branches. The bindings validate the index and raise IndexError, because that is where the caller is untrusted:

  IndexError: index (1000, 1000) is out of range for size 3
  

• Everything that creates, resizes or evaluates — empty, zero, variable, resize, pad_left, pad_right, eval, hm, set_hm validate their arguments in the header via the existing throw_exceptions() mechanism, reusing check_valid_size / check_equal_size plus check_valid_index and check_pad_size. This also covers the C++ API, and HYPERJET_NO_EXCEPTIONS still degrades to assert.

from_arrays is covered transitively through set_hm.

Performance

empty(size) sits on the hot path of every dynamic operation, so the added check was measured rather than assumed — 2M iterations of a * b + a.sin(), dynamic order 2 size 8, -O3 -DNDEBUG, three alternating runs:

run	with check	without
1	166.8 ns/op	165.2 ns/op
2	164.1 ns/op	168.2 ns/op
3	164.0 ns/op	167.7 ns/op

Noise — the checked build is faster in two of three runs. The compare is free next to the allocation it guards. The AD kernels (unary/binary/ternary) and the operators are untouched.

Tests

Written first and observed failing.

Python — 12 new parametrized cases. test_hessian_index_out_of_range probes the boundary (2, -1 on a size-2 scalar) for both h and set_h; test_eval_with_wrong_number_of_values, test_set_hm_with_wrong_shape, test_init_by_array_with_inconsistent_shape, test_variable_with_index_out_of_range, test_negative_size and test_pad_below_current_size cover the rest. The existing tests already pinned the static size-mismatch paths, so these fill in the dynamic ones. The eval test took down the whole pytest process before the fix:

Extension modules: numpy._core._multiarray_umath, numpy.linalg._umath_linalg (total: 2)
[SIGSEGV]

C++ — TEST_CASE("Dynamic size checks") covers the header-level checks that Python cannot reach directly, including hm(mode, out) with an undersized output matrix. RED was 4 × did NOT throw at all!.

After: C++ 41/41 test cases, 406 assertions in both Debug and Release · Python 166 passed · clang-format --Werror and ruff format --check clean · all 21 entry points now raise a proper Python exception, and valid usage is unchanged.

Note

ruff check reports two pre-existing unused imports in test_SScalar.py (numpy, copy.copy). Not touched here — ruff check is not part of the CI gate.

Remaining items from the same review, unaffected by this PR:

• DDScalar<1, double, Dynamic>{1.0, 2.0, 3.0} segfaults — m_data is an empty vector when std::copy writes into it.
• x *= x and x /= x produce wrong derivatives (self-aliasing) in both DDScalar and SScalar.
• resize() silently scrambles the Hessian layout for order 2; pad_right() does it correctly. This PR only stops resize() from accepting a negative size.