test_io_subarray: Synchronize the GPU before handing buffers to MPI - #993
Merged
Merged
Conversation
The collective read in `test_io_subarray.jl` flaked on the ROCm Buildkite agent (Julia 1.10), with `data_read` coming back all zeros. There `ArrayType` is `ROCArray`, and the `fill!(data_read, 0)` just before `MPI.File.read_all!` is an asynchronous kernel launch; without a `synchronize()` the zero-fill can land after MPI has written the data. The noncollective read above it has no `fill!` and passed. Every other GPU-aware test already synchronizes after device-side initialization; this one did not. This is not a file-consistency race. Each rank accesses only the bytes it wrote itself through the same handle, so MPI guarantees sequential consistency regardless of which ROMIO aggregator physically performs the read, and the existing comment saying no barrier is needed stands.
vchuravy
approved these changes
Sep 23, 2026
Contributor
Author
|
I'm waiting for the GPU CI tests to run (and hopefully be green!) before merging. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
test/test_io_subarray.jl:52(theMPI.File.read_all!check) flaked on the ROCm Buildkite job (Julia 1.10, build 1891), withdata_readall zeros. The same job passed on the builds before and after it.On that agent
ArrayTypeisROCArray, and thefill!(data_read, 0)right beforeread_all!is an asynchronous kernel. Nothing synchronized before the buffer went to MPI, so the zero-fill could finish after MPI had written the data. The noncollective read above it has nofill!, which is why it passed. The other GPU tests already callsynchronize()after device-side initialization; this PR adds the missing calls here.This is not a file-consistency race: each rank reads only the bytes it wrote itself, so the existing comment saying no barrier is needed is correct.
It's a flake, so this can't be reproduced on demand. The evidence will be the ROCm job staying green. On CPU
synchronize()does nothing.