Conversation
The buffer types hard-coded `Cint` counts, so `MPI.Bcast!` on an array of more than
`typemax(Cint)` elements died in `Cint(length(arr))` -- and the large-count `MPI_*_c`
entry points the API layer now exposes were called by nothing.
Type the counts and displacements the four buffer types carry with `API.Count`,
`API.Displ` and `API.TypeDispl`, and route the 54 affected call sites to the `MPI_*_c`
entry points. Those aliases follow the MPI library, so large messages work with no
special handling on MPI 4.0 or later and raise `InexactError` rather than truncating on
anything older. Three aliases and not one because the widenings differ: `MPI_Alltoallv_c`
widens counts to `MPI_Count` but displacements only to `MPI_Aint`, while
`MPI_Type_create_struct_c` widens its byte displacements all the way to `MPI_Count`.
They coincide on 64-bit ABIs and come apart on 32-bit ones.
A large count is only usable where addresses are wide enough to describe the data:
`MPI_Aint` is pointer-sized, so on a 32-bit build neither a 2 GiB datatype extent nor a
2 GiB displacement can exist whatever `MPI_Count` says, and a 32-bit address space could
not hold the buffer anyway. test_largecount.jl gates on that rather than on
`HAS_LARGE_COUNT`, which answers a different question.
Left alone deliberately:
- Ranks, roots, tags, keyvals, flags, request indices and everything in topology.jl are
still `int` in MPI 5.0 -- the absence of `_c` variants for `MPI_Cart_create`,
`MPI_Dist_graph_*` and `MPI_Waitsome` says so.
- User-defined reduction operators keep using the narrow `MPI_Op_create`. Which callback
signature applies is fixed by the creator, not by the reduce entry point consuming the
handle, and `OpWrapper`'s loop is element-wise, so an implementation is free to chunk a
large-count reduction into calls the `int *len` callback can express. Confirmed on
MPICH 5.0.1 with a 2 GiB `Allreduce!`, behind JULIA_MPI_TEST_LARGECOUNT.
Two behaviour changes worth knowing about. A `VBuffer` built from a `Vector{Cint}` now
copies rather than aliasing the caller's array, because the conversion is no longer the
identity. And `MPI.bcast` of an arbitrary object broadcasts its serialized length as
`Int64` rather than `Cint` -- part of the wire format, so fixed rather than
build-dependent, and it lifts the 2 GiB cap that `Cint` imposed.
`UBuffer.nchunks` becomes `Int`: it is never passed to MPI, being only validated against
`Comm_size`, yet narrowing it through `Cint` meant `UBuffer(arr, 1)` threw `InexactError`
for an array of more than `typemax(Cint)` elements.
The GPU extensions now let `Buffer` convert the count instead of writing `Cint`
themselves, so they stay out of the way of any future change of this kind.
eschnett
force-pushed
the
eschnett/mpi-large-counts
branch
from
September 23, 2026 16:02
c4cda6d to
14da0d2
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is the second part of #989 (large count functions). It is stacked on #990, which updates the auto-generated API to support MPI 5.0, including large-count functions.
The important design choice in #990 was that the large count functions, if not supported (because the MPI implementation is too old), will automatically fall back to the narrow count functions. This means that the high-level API can always call the large count functions. Recall that the fallback happens when MPI.jl is precompiled, and there is no run-time check necessary. The only visible difference is that e.g.
MPI_Send_ceither takes anInt64or anInt32as argument. These argument types are explicitly declared in Julia asAPI.CountandAPI.Displfor counts and displacements, respectively.This PR updates MPI.jl to actually call the large-count functions. Most of the changes are mechanical, mostly using
API.CountorAPI.Displinstead ofCint.User-defined reduction operators keep using the narrow
MPI_Op_create. Which callback signature applies is determined by the function which creates the user-defined operator, not by the function that calls the operator. It turns out that all MPI implementations we tested (MPICH, Open MPI) will automatically chunk large arrays (> 2 GB) when a narrow operator is used, so this is fine.There may by two small performance impacts:
VBuffernow stores aVector{API.Count}andVector{API.Displ}instead ofVector{Cint}. Depending on what types the caller passes, this may now require copying these arrays.MPI.bcastto broadcast the lengths when sending arbitrary objects. This now broadcasts the length as 64-bit integer, not as 32 bits.Both performance impacts should be negligible compared to the actual data exchange.