API to get just the nth of N chunks? #64

Activity

1. lmiq commented on Jun 25, 2026

   

   lmiq

   on Jun 25, 2026

   Collaborator

   More actions

   You can chunk the indices, assuming they are the same for all arrays, something like this may be what you want:

   julia> x = ones(20); y = ones(20); z = zeros(20);
   
   julia> @sync for inds in index_chunks(x; n=4)
              @spawn begin
                  for i in inds
                      z[i] = x[i] + y[i]
                  end
              end
          end
   
   julia> z
   20-element Vector{Float64}:
    2.0
    2.0
    2.0
    2.0
   ...
2. ojwoodford commented on Jun 25, 2026

   

   ojwoodford

   on Jun 25, 2026

   Last edited by ojwoodford

   Author

   More actions

   Thanks. That won't work for me though. I need something like this:

   A = [rand(1e6), rand(Int, 1e5)]
   n = nthreads()
   @sync for n in 1:N
       @spawn begin
           for a in A
               myfun(getchunk(A, n, N))
           end
       end
   end
3. lmiq commented on Jun 25, 2026

   

   lmiq

   on Jun 25, 2026

   Collaborator

   More actions

   Actually you can do that, because the chunking is lazy, so you can do:

   julia> A = [ ones(10), ones(15) ];
   
   julia> A_chunks = chunks.(A; n=N);
   
   julia> A_chunks[1][1]
   3-element view(::Vector{Float64}, 1:3) with eltype Float64:
    1.0
    1.0
    1.0
   
   julia> A_chunks[1][3]
   2-element view(::Vector{Float64}, 7:8) with eltype Float64:
    1.0
    1.0
   
   julia> A_chunks[2][2]
   4-element view(::Vector{Float64}, 5:8) with eltype Float64:
    1.0
    1.0
    1.0
    1.0

   such that you can do:

   julia> @sync for ichunk in 1:N
              @spawn begin
                  for i in eachindex(A)
                      A_chunks[i][ichunk] .+= 1
                  end
              end
          end
4. ojwoodford commented on Jun 25, 2026

   

   ojwoodford

   on Jun 25, 2026

   Last edited by ojwoodford

   Author

   More actions

   That’s nice. Good to know. Is there a way to construct the chunk inside the task, though? If the data structure A isn’t iterable, chunk.() won’t work.
5. lmiq commented on Jun 25, 2026

   

   lmiq

   on Jun 25, 2026

   Collaborator

   More actions

   You can just do:

   julia> x = 1:10;
   
   julia> x_chunks = chunks(x; n=4);
   
   julia> x_chunks[2]
   4:6

   inside the task. You will be repeating the chunking for each task, but very likely that is not an issue, as it is very cheap.

   👍React with 👍1Reacted by Oliver J. Woodford
6. ojwoodford commented on Jun 25, 2026

   

   ojwoodford

   on Jun 25, 2026

   Author

   More actions

   Thanks. I guess this relies on the fact that the chunking is deterministic, so never changes between calls to chunks.
7. lmiq commented on Jun 25, 2026

   

   lmiq

   on Jun 25, 2026

   Collaborator

   More actions

   Oh yes. It is.

   👍React with 👍1Reacted by Oliver J. Woodford
8. ojwoodford

   closed this as completedon Jun 25, 2026