Repository navigation
Conversation
fhagemann
commented
Sep 25, 2026
Collaborator
Author
There was a problem hiding this comment.
Copying #623 (comment):
The code snippet @claudiaalvgar posted is:
_ghost_ops_φ(ax::DiscreteAxis{T, :periodic, :periodic}, n::Int, gbf) where {T} = _ghost_ops(ax, n, gbf)
_ghost_ops_φ(ax::DiscreteAxis{T, :reflecting, :reflecting}, n::Int, gbf) where {T} = _ghost_ops(ax, n, gbf)
_ghost_ops_φ(::DiscreteAxis{T, BL, BR}, n::Int, gbf) where {T, BL, BR} =
throw(ArgumentError("boundary condition ($BL, $BR) is not supported for the φ axis"))
Comment on lines
+168
to
+171
| # Workgroup shapes tuned on an A100; both are 2.4-2.6x faster than the | ||
| # KernelAbstractions default across 2D/3D cylindrical and Cartesian grids. | ||
| # Dimension 1 is the memory-contiguous red-black dimension. | ||
| _sor_kernel_workgroupsize(ndrange) = ndrange[2] == 1 ? (128, 1, 1) : (32, 4, 1) |
fhagemann
commented
Sep 25, 2026
Comment on lines
+14
to
+18
| kernel = if backend isa _KA_Backend && !(backend isa KernelAbstractions.CPU) | ||
| get_sor_kernel(S, backend, Val(via_KernelAbstractions), _sor_kernel_workgroupsize(ndrange)) | ||
| else | ||
| get_sor_kernel(S, backend, Val(via_KernelAbstractions)) | ||
| end |
Collaborator
Author
There was a problem hiding this comment.
Copying #623 (comment):
Manually verify that this is passed correctly via args... in src/PotentialCalculation/SuccessiveOverRelaxation/GPU_kernel.jl
Collaborator
Author
There was a problem hiding this comment.
From PR623:
Manually check that these tests make sense.
- Drop the host-side backend synchronization between the red/black kernels and the boundary-condition updates on GPU backends: stream ordering already guarantees correctness, and the synchronizations stalled the queue twice per iteration. The task-based KA CPU backend keeps its explicit synchronization. - Replace the r = 0 azimuthal-average boundary condition on GPU arrays (getindex + CUBLAS gemv + broadcast per color, allocating temporaries and forcing host round trips every iteration) with one small kernel. Per-iteration profile of a 3.4M-point cylindrical grid on an A100: host-to-device copies 8011 -> 11, pool allocations 12016 -> 16 per 1000 iterations; the update loop is now bound by the SOR kernels themselves (~2.6e9 point updates/s, up from ~1.0e9 end-to-end before). CPU and GPU results agree to convergence tolerance (verified with and without depletion handling). Created by generative AI.
On GPU backends the ghost-face updates ran as ~10 view broadcasts per red-black color; together with per-call dynamic launch configuration this left the iteration loop host-bound. Encode each face update as (active, source row, factor) - a 1:1 transcription of the existing broadcast methods - and apply all six faces in a single kernel launch after the r = 0 kernel. Instantiate the SOR kernels with a static workgroup size, which removes the per-launch configuration cost. The iteration loop of a 3.4M-point cylindrical grid on an A100 drops from ~3.4 ms to ~0.37 ms per iteration (~9x) and is now bound by the device time of the SOR kernels. Verified against the CPU backend for cylindrical grids (2D, full-2pi periodic with r = 0, reflecting phi-wedge) and Cartesian grids, with and without depletion handling; Cartesian and 2D results are bit-identical, 3D cylindrical agrees to convergence tolerance. The multi-ghost corner entries the fused faces may write concurrently are never read by the SOR stencil. Created by generative AI.
Add JLArray-based testsets: as a real AbstractGPUArray on CPU memory, JLArray runs the fused boundary-condition kernels, the r0 kernel and the device convergence checks that the abstract pseudoGPUArray type never reached (Adapt leaves its arrays as plain Arrays). Cylindrical and Cartesian solves reproduce the CPU results bit-identically. Add test_boundary_conditions.jl pinning every ghost-row method exactly per axis role and boundary-type combination, the equivalence of the CPU broadcast methods with the GPU ghost-face op tables, and the r0 kernel against its CPU reference. The r-axis end-2 source row is pinned as current behavior (issue JuliaPhysics#617). Also cover temperature-model selection with the vacuum no-op scaling, virtual drift volumes (dead and arbitrary, including the Val-id user hook and its error path), and the geometrical axis weights interface. Created by generative AI.
fhagemann
force-pushed
the
improvements-gpu
branch
from
September 25, 2026 09:19
67eceac to
8c3c4f9
Compare
Collaborator
Author
|
I moved more commits from #623/#644 here:
|
fhagemann
commented
Sep 25, 2026
| # precomputed weights and broadcast the result back to those columns. Replaces | ||
| # a getindex + gemv + broadcast sequence (a temporary and several kernel | ||
| # launches per SOR iteration) on GPU backends. | ||
| @kernel function r0_bc_kernel!(rbpot, rbi::Int, nnz::Int, gw, even_points::Bool) |
Comment on lines
+72
to
+83
|
|
||
| function apply_boundary_conditions_at_r0!( | ||
| rbpot::GPUArrays.AbstractGPUArray{T, 4}, | ||
| rbi::Int, | ||
| nnz::Int, | ||
| gw::AbstractArray{T}, | ||
| update_even_points::Val{even_points} | ||
| ) where {T, even_points} | ||
| kern = r0_bc_kernel!(_ka_get_backend(rbpot)) | ||
| kern(rbpot, rbi, nnz, gw, even_points, ndrange = rbidx(nnz) - 1) | ||
| nothing | ||
| end |
Collaborator
Author
There was a problem hiding this comment.
From PR623: cross-check this
Collaborator
Author
|
Marking this as draft for now, because it looks like it needs the changes in #644 to be merged first. |
fhagemann
marked this pull request as draft
September 25, 2026 10:01
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cherry-picked from #623, addressing GPU boundary conditions.
Might require some manual checks before merging.