Skip to content

API: Raise when mixing SLEEF and longdouble backends - #122

Open
SwayamInSync wants to merge 22 commits into
numpy:mainfrom
SwayamInSync:raise-mixed-backends
Open

SwayamInSync wants to merge 22 commits into
numpy:mainfrom
SwayamInSync:raise-mixed-backends

Conversation

@SwayamInSync

Copy link
Copy Markdown
Member

Addresses #113 for now, until #112 makes casts between backends exact.

Values of the SLEEF and longdouble backends currently move between each other in two flawed ways. Every conversion goes through float64 (#112), so it silently loses precision in both directions. Operations also disagree on the common backend (#113): np.result_type and np.concatenate pick SLEEF, while ufuncs pick longdouble.

This PR makes the policy consistent by refusing every such operation with a TypeError until there is an exact conversion and one agreed promotion rule. Everything that does not combine the two backends is unchanged, and NumPy's own dtypes, including np.longdouble, work with both backends as before.

Behaviour

Upstream main (888042c) vs this branch, on x86-64 Linux:

Operation main This PR
np.result_type(sleef_dt, ld_dt) QuadPrecDType(backend='sleef') TypeError
(sleef_arr + ld_arr).dtype QuadPrecDType(backend='longdouble') TypeError
np.concatenate([sleef_arr, ld_arr]).dtype QuadPrecDType(backend='sleef') TypeError
SLEEF 1 + 2**-112 .astype(ld_dt), minus 1 0.0 TypeError
longdouble 1 + 2**-60 .astype(sleef_dt), minus 1 0.0 TypeError
ld_arr[0] = QuadPrecision(1.5) stores 0.0 TypeError
np.sqrt(sleef [4.0], out=ld_arr, casting="unsafe") writes 0.0 TypeError
QuadPrecision(q) == q for q = 1 + 2**-112 False True
ld_arr + 1 [2.5] TypeError
ld_arr + np.float64(1) [2.5] [2.5]
ld_arr.astype(np.longdouble) [1.5] [1.5]

The error is TypeError: Cannot mix QuadPrecision backends 'sleef' and 'longdouble'. Where NumPy checks the casting rule before calling our resolver (gufuncs, astype(..., casting="same_kind")), it is NumPy's own Cannot cast ... TypeError instead.

Backwards-incompatible changes

  • Ufuncs whose operands or out= arrays have different backends, astype, np.copyto, slice and item assignment, np.result_type, np.promote_types, np.concatenate, and QuadPrecision(value, backend=...) given a QuadPrecision of the other backend now raise. np.array of mixed QuadPrecision scalars gives an object array.
  • Python int and float use the default SLEEF descriptor, so they raise with longdouble arrays too. This includes NumPy functions that pass Python scalars internally: np.ones, np.full, np.ones_like, mean, var, std, np.median, np.isclose, np.allclose and np.round. The workarounds are NumPy scalars or arrays, e.g. ld / np.float64(2), np.full(3, np.longdouble(1), dtype=ld.dtype) or np.ones(3).astype(ld.dtype). SLEEF arrays are unaffected.

Bug fixes

  • Assigning a QuadPrecision into an array of the other backend, or casting an object array of them, stored the bytes under the wrong backend (0.0 instead of 1.5 above).
  • Unary ufuncs with an out= array of the other backend and casting="unsafe" wrote values in the wrong format.
  • QuadPrecision(q) rounded a same-backend q through float64; it now copies exactly.

Implementation

  • quad_to_quad_resolve_descriptors is the single point that refuses a cross-backend cast. The cast loop's cross-backend branch and quad_to_quad_same_value_check are kept as they are, so BUG: inter-backend casts lose precision by converting through double #112 only has to replace the float64 conversions and allow the cast in the resolver.
  • common_instance, the binary, comparison and unary resolvers, setitem and the scalar paths raise through shared helpers in dtype.h (quad_set_backend_mismatch_error, quad_loop_backend, quad_out_loop_descr).
  • The float64-based same_value tests for cross-backend casts are kept as strict xfails tagged gh-112. Once casts between backends are allowed, they XPASS and fail CI until the markers are removed.

Tests and docs

  • New tests/test_backend_mixing.py covers every path above, the literal limitation, the workarounds, and that NumPy dtypes are unaffected. The new tests fail on main.
  • Docs: a "Mixing backends" section in docs/api/core.md, and the changes above in the README changelog under "Unreleased".

CI

big_endian.yml skips pandas 3.1.0 on s390x. Its sdist does not build there (pandas-dev/pandas#71548, fix planned for 3.1.1), and s390x has no pandas wheels. pandas!=3.1.0 keeps test_pandas_strrep running on 3.0.6 and moves to 3.1.1 automatically.

AI Disclosure

Used claude code as assistant for testing and writing PR/docs

@SwayamInSync SwayamInSync added bug Something isn't working v1.1 labels Nov 3, 2026
@SwayamInSync

Copy link
Copy Markdown
Member Author

I believe the earlier behaviour was pretty non-deterministic and buggy, while I will be working on adding support of proper intermediate (f64) free inter-backend working (planned for future releases). We can strictly error-out the any inter-backend use for now instead of being sloppy with the implementation

@ngoldbaum

Copy link
Copy Markdown
Member

Sorry for taking a little while to look at this! I agree that fixing this is worthwhile. Unfortunately the current approach is overly broad and breaks valid operations like this:

a = np.array([1, 2], dtype=QuadPrecDType("longdouble"))
a + 1  # Raises “Cannot mix QuadPrecision backends”

I think adding this restriction needs to wait for a NumPy fix. NumPy materializes 1 using the default SLEEF descriptor before calling the loop resolver. The resolver correctly selects longdouble, but the new cast restriction rejects that temporary.

Another problem is this:

x = QuadPrecision(1, backend="longdouble")
QuadPrecision(x)  # Defaults to SLEEF and raises

If a backend is already selected, the constructor should use that instead of the default.

The prototype below defers conversion of the original Python scalar until the input descriptor is resolved, avoiding the intermediate SLEEF conversion. NumPy already uses this ordering in its private scalar-aware resolver path. An explicit DType opt-in preserves existing user dtypes’ behavior. With quaddtype opted in, literals work with either backend, while actual mixed-backend arrays still raise. The copyto adjustment also restores ones and full.

NumPy patch: opt-in conversion of Python scalars using resolved descriptors
diff --git a/numpy/_core/include/numpy/dtype_api.h b/numpy/_core/include/numpy/dtype_api.h
index 047cb861b9..89bf9c2059 100644
--- a/numpy/_core/include/numpy/dtype_api.h
+++ b/numpy/_core/include/numpy/dtype_api.h
@@ -411,6 +411,9 @@ typedef int (PyArrayMethod_PromoterFunction)(PyObject *ufunc,
 #define NPY_DT_PARAMETRIC 1 << 2
 #define NPY_DT_NUMERIC 1 << 3
 
+/* Opt in to packing Python numeric scalars after descriptor resolution. */
+#define NPY_DT_REPACK_PYSCALARS 1 << 4
+
 /*
  * These correspond to slots in the NPY_DType_Slots struct and must
  * be in the same order as the members of that struct. If new slots
diff --git a/numpy/_core/src/multiarray/multiarraymodule.c b/numpy/_core/src/multiarray/multiarraymodule.c
index 516986d2d5..b17d029c42 100644
--- a/numpy/_core/src/multiarray/multiarraymodule.c
+++ b/numpy/_core/src/multiarray/multiarraymodule.c
@@ -2033,6 +2033,10 @@ array_copyto(PyObject *NPY_UNUSED(ignored),
         if (descr == NULL) {
             goto fail;
         }
+        if (is_pyscalar && (dst_DType->flags & NPY_DT_REPACK_PYSCALARS) &&
+                NPY_DTYPE(descr) == dst_DType) {
+            Py_SETREF(descr, (PyArray_Descr *)Py_NewRef(PyArray_DESCR(dst)));
+        }
         int res = npy_update_operand_for_scalar(&src, src_obj, descr, casting);
         Py_DECREF(descr);
         if (res < 0) {
diff --git a/numpy/_core/src/multiarray/public_dtype_api.c b/numpy/_core/src/multiarray/public_dtype_api.c
index ebf0440fc7..f2c5633eab 100644
--- a/numpy/_core/src/multiarray/public_dtype_api.c
+++ b/numpy/_core/src/multiarray/public_dtype_api.c
@@ -57,11 +57,13 @@ PyArrayInitDTypeMeta_FromSpec(
     }
 
     /* Check and handle flags: */
-    int allowed_flags = NPY_DT_PARAMETRIC | NPY_DT_ABSTRACT | NPY_DT_NUMERIC;
+    int allowed_flags = NPY_DT_PARAMETRIC | NPY_DT_ABSTRACT | NPY_DT_NUMERIC
+            | NPY_DT_REPACK_PYSCALARS;
     if (spec->flags & ~(allowed_flags)) {
         PyErr_SetString(PyExc_RuntimeError,
                 "invalid DType flags specified, only NPY_DT_PARAMETRIC, "
-                "NPY_DT_ABSTRACT, and NPY_DT_NUMERIC are valid flags for "
+                "NPY_DT_ABSTRACT, NPY_DT_NUMERIC, and NPY_DT_REPACK_PYSCALARS "
+                "are valid flags for "
                 "user DTypes.");
         return -1;
     }
diff --git a/numpy/_core/src/umath/ufunc_object.c b/numpy/_core/src/umath/ufunc_object.c
index 67ad4da489..f1e797cc5a 100644
--- a/numpy/_core/src/umath/ufunc_object.c
+++ b/numpy/_core/src/umath/ufunc_object.c
@@ -4456,6 +4456,7 @@ resolve_descriptors(int nop,
     /* number of original_descrs entries to clean up, including NULLs */
     int n_cleanup = 0;
     PyArray_Descr *original_descrs[NPY_MAXARGS];
+    npy_bool deferred_scalars[NPY_MAXARGS] = {0};
 
     NPY_UF_DBG_PRINT("Resolving the descriptors\n");
 
@@ -4544,6 +4545,15 @@ resolve_descriptors(int nop,
             if (new_descr == NULL) {
                 goto finish;
             }
+            if (input != NULL &&
+                    (PyArray_FLAGS(operands[i]) & NPY_ARRAY_WAS_PYTHON_LITERAL) &&
+                    NPY_DTYPE(new_descr) == signature[i] &&
+                    (signature[i]->flags & NPY_DT_REPACK_PYSCALARS)) {
+                original_descrs[i] = new_descr;
+                n_cleanup = i + 1;
+                deferred_scalars[i] = NPY_TRUE;
+                continue;
+            }
             int res = npy_update_operand_for_scalar(
                 &operands[i], input, new_descr, casting);
             Py_DECREF(new_descr);
@@ -4601,6 +4611,14 @@ resolve_descriptors(int nop,
                 ufunc_get_name_cstr(ufunc), npy_casting_to_string(casting));
         goto finish;
     }
+    /* Pack deferred Python scalars once the input descriptors are resolved. */
+    for (int i = 0; i < ufunc->nin; i++) {
+        if (deferred_scalars[i] && npy_update_operand_for_scalar(
+                &operands[i], inputs[i], dtypes[i], casting) < 0) {
+            goto finish;
+        }
+    }
+
     retval = 0;
 
   finish:
diff --git a/numpy/_core/tests/test_custom_dtypes.py b/numpy/_core/tests/test_custom_dtypes.py
index e6c5d0583c..7e27113961 100644
--- a/numpy/_core/tests/test_custom_dtypes.py
+++ b/numpy/_core/tests/test_custom_dtypes.py
@@ -198,6 +198,12 @@ def test_sfloat_cast_internal_errors(self, aligned):
                 match="error raised inside the core-loop: non-finite factor!"):
             a.astype(SF(2e-300))
 
+    def test_copyto_python_scalar_keeps_existing_cast_path(self):
+        # Without opt-in, scalar assignment must still execute the cast.
+        target = np.zeros(1, dtype=SF(1e-310))
+        with pytest.raises(TypeError, match="non-finite factor"):
+            np.copyto(target, 1e-310)
+
     def test_sfloat_promotion(self):
         assert np.result_type(SF(2.), SF(3.)) == SF(3.)
         assert np.result_type(SF(3.), SF(2.)) == SF(3.)

Quaddtype opts in by adding NPY_DT_REPACK_PYSCALARS to its DType spec flags, guarded by build-time and runtime NumPy support. The proposed flag name/API would need upstream agreement.

We should delay the blanket rejection until this support is available in the NumPy versions quaddtype supports. Representation-corruption fixes can proceed separately. The constructor regression affecting scalar-returning reductions also needs its own fix.

@SwayamInSync

Copy link
Copy Markdown
Member Author

I’ll take your NPY_DT_REPACK_PYSCALARS idea to NumPy as a PR, and come back to this one once it’s available, gated on the NumPy version. converting this to draft until then.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working v1.1

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants