Title: [TACO-26] CPU-Parallel ACO: Persistent Thread Pool and Concurrency Fixes#186
Open
HornetSys wants to merge 30 commits into
Open
Title: [TACO-26] CPU-Parallel ACO: Persistent Thread Pool and Concurrency Fixes#186HornetSys wants to merge 30 commits into
HornetSys wants to merge 30 commits into
Conversation
…ll be done at a differnt time
…nges and now its not working.
…sult in any malloc errors, runtime errors nevertheless
…t create new adjusted data structures and function calls for the SchedInstruction clas
…ated adjustment. Problem remains.
- random.hip.cpp: replace std::scoped_lock{m} (no-op temporary) with
std::scoped_lock lock(m) in all six call sites; the anonymous temporary
was immediately destroyed, leaving j/k/y[] unprotected and causing
sporadic out-of-bounds SIGSEGV under concurrent RNG calls
- aco.hip.cpp: pre-build instScsrs_ before spawning PCPU threads to
eliminate the GetFrstScsr/GetNxtScsr cursor race on scsrLst_->rtrvEntry_
- aco.hip.cpp: fix integer division IScore*9/10→IScore*=0.9 and guard
globalBestStalls_==0 before divisions in SelectInstruction and
PCPU_SelectInstruction
- aco.hip.cpp: select best schedule across all CPU threads in
FindManyCPUSchedule instead of always taking thread 0's result
- aco.hip.cpp: restore numAntsTerminated_++ in PCPU_FindOneSchedule
- sched_region.hip.cpp, OptimizingScheduler.hip.cpp: guard HIP device
allocation behind hipGetDeviceCount check so DEV_ACO NO runs cleanly
without a GPU present
- sched.ini: set DEV_ACO NO, HOST_ANTS 8; increase NO_CPU_THREADS to 8
Bug 1 fix — numAntsTerminated_ data race:
FindManyCPUSchedule (aco.hip.cpp:2469): introduces a std::atomic<int> localAntsTerminated{0}, captures it by reference in each thread lambda, and adds it to numAntsTerminated_ after joining.
PCPU_FindOneSchedule signature (aco.h:157, aco.hip.cpp:2517): takes std::atomic<int> &antsTerminated instead of touching the shared numAntsTerminated_ directly.
The early-exit increment (aco.hip.cpp:2638): changed from numAntsTerminated_++ to antsTerminated.fetch_add(1, std::memory_order_relaxed).
Bug 2 fix — shared RNG serializes all ants:
PCPUACOSchedVars (aco.h:94): added std::mt19937 rng member.
AllocPCPUACOSchedVars (aco.hip.cpp:2921): seeds each thread's RNG with random_seed_ ^ (thread_index * large_prime) so each ant gets a distinct, reproducible sequence.
PCPU_SelectInstruction (aco.hip.cpp:2804): replaced RandDouble(...) calls (which contend on a global mutex) with std::uniform_real_distribution draws from pcpu_sched_vars.rng — fully lock-free per thread.
Replace FindManyCPUSchedule's per-batch thread create/join with a persistent CPUThreadPool that lives for the duration of each occupancy iteration, eliminating the spawn/join overhead on every ACO iteration. Threads are kept alive between iterations using a generation-counter condition variable pattern. All per-thread state (SchedInstruction pcpu_inst_vars_, Register pcpu_crntUseCnt_, ParallelCPUVars fields and bitvectors, and the shared Register::crntUseCnt_ that IsLive() reads) is explicitly reset at the top of each iteration via new Reset methods, replacing the implicit reset that Free+Alloc in FindManyCPUSchedule previously provided. New methods added: - CPUThreadPool: persistent pool struct (aco.hip.cpp) - ACOScheduler::PCPU_ResetSchedInsts: resets pcpu_inst_vars_ per iter - SchedInstruction::InitPCPUVars: implements previously empty reset body - Register::ResetParallelCPURegs: zeroes per-thread crntUseCnt_ array - BBWithSpill::ResetParallelCPUVars: resets all ParallelCPUVars state including shared crntUseCnt_ to fix "used without being defined" fatal
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR completely overhauls the CPU-based Ant Colony Optimization (ACO) scheduling fallback.
The architecture has been upgraded to eliminate per-iteration thread lifecycle overhead, remove global mutex bottlenecks, and resolve severe memory safety and data race bugs that previously caused
SIGSEGVcrashes and fatal errors during host-only execution.Technical Details
1. Persistent Thread Pool Architecture (Performance)
FindManyCPUSchedulewith a persistentCPUThreadPoolthat lives for the duration of each occupancy iteration. Threads are kept alive between iterations using a generation-counter condition variable pattern.Free/Alloc) to explicit, per-iteration state resets to support thread persistence. IntroducedPCPU_ResetSchedInsts,ResetParallelCPURegs, andResetParallelCPUVarsto clear thread state and bitvectors safely (resolving "used without being defined" fatal errors).2. Lock-Free Concurrency (Scaling)
std::mt19937member toPCPUACOSchedVars, seeding each thread deterministically (random_seed_ ^ (thread_index * large_prime)).PCPU_SelectInstructionnow utilizes lock-freestd::uniform_real_distributiondraws.numAntsTerminated_. Replaced it with a localstd::atomic<int>inFindManyCPUSchedule, updated via lock-freefetch_add, and cleanly aggregated after threads synchronize.3. Memory Safety & Logic Corrections (Stability)
SIGSEGVin RNG: Replaced the no-op temporarystd::scoped_lock{m}with a namedstd::scoped_lock lock(m)across all six call sites inrandom.hip.cpp, preventing array bounds violations during legacy concurrent calls.instScsrs_inaco.hip.cppprior to spawning threads, resolving theGetFrstScsr/GetNxtScsrrace condition.FindManyCPUSchedulenow correctly selects the best schedule across all CPU threads (instead of defaulting to thread 0). Corrected integer truncation (IScore*9/10→IScore*=0.9) and added guards against division-by-zero (globalBestStalls_==0).4. Environment & Configuration
hipGetDeviceCountcheck, allowingDEV_ACO NOto run natively on host systems without GPU presence.sched.ini): SetDEV_ACO NO,HOST_ANTS 8, andNO_CPU_THREADS 8.Logger::Info.