Wrong suggestions for OpenMPI and GCC(core)

[Flamefire]

Describe the bug

1. LMod suggest only GCCcore instead of GCC for loading OpenMPI together with a GCCcore module
2. It suggests incompatible combinations of GCCcore and GCC

To Reproduce

Reproducer for /tmp/modules: modules.tar.gz

$ ml CuPy OpenMPI
Lmod has detected the following error:  These module(s) or extension(s) exist but cannot be loaded as requested: "CuPy", "OpenMPI"
   Try: "module spider CuPy OpenMPI" to see how to load the module(s).
   Or load any one of these options:
      module load release/24.04 GCCcore/13.2.0 CuPy OpenMPI
      module load release/24.04 GCCcore/14.2.0 CuPy OpenMPI
      module load release/24.10 GCCcore/13.2.0 CuPy OpenMPI



$ module load release/24.04 GCCcore/13.2.0 CuPy OpenMPI
Lmod has detected the following error:  These module(s) or extension(s) exist but cannot be loaded as requested: "OpenMPI"
   Try: "module spider OpenMPI" to see how to load the module(s).
   Or load any one of these options:
      module load release/24.04 GCC/13.2.0 GCCcore/13.2.0 CuPy OpenMPI
      module load release/24.04 GCC/14.2.0 GCCcore/13.2.0 CuPy OpenMPI



$ module load release/24.04 GCC/14.2.0 GCCcore/13.2.0 CuPy OpenMPI
Lmod Warning: 
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
The following dependent module(s) are not currently loaded: GCCcore/14.2.0 (required by: GCC/14.2.0)
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------



Custom hook:
Module GCCcore/14.2.0 unloaded.
Modules GCC/14.2.0, GCCcore/13.2.0, CuPy/14, OpenMPI/5.6 and 1 dependency loaded.

Expected behavior

Only correct suggestions

Additional context

LMod 9.2.3 release version used.

Are the 3 empty lines at the end of each error expected?

Possibly related to #805 and #823

[mrcawood]

Hi @Flamefire, thanks again for the reproducer; I used your modules.tar.gz layout for the regression test.

I have a branch ready for you to try:

Branch: IS828-toolchain-suggest
https://github.com/TACC/Lmod/tree/IS828-toolchain-suggest

This branch is the current integration point for the load-failure suggestion work. It builds on the earlier fixes for related issues and adds the #828 fix on top:

Issue	What it addressed
#805	Disjoint toolchains (e.g. Python under GCCcore, QGIS under GCC/OpenMPI), suggest paths from the right hierarchy, not invalid cross-product commands
#823	Shorter suggestions when dependencies are already loaded; stack-root swap paths still listed when appropriate
#828	GCC / GCCcore toolchain handling, no GCCcore-only suggestions when GCC is required, and no mismatched GCC + GCCcore pairs in one command

#828 fix (your report)

When modules sit under different but related toolchain layers (CuPy → GCCcore, OpenMPI → GCC), Lmod now:

1. Treats same-version GCC and GCCcore as one toolchain when merging spider parent paths
2. Prefers GCC over GCCcore in suggested commands when both apply
3. Normalizes spider parent paths (drops self-referential tail entries that skewed path counts)
4. Omits an already-loaded / requested GCCcore from a suggestion when the path implies a different matching GCCcore via GCC
5. Validates multi-module suggestions against all failing modules, not only the one with the fewest paths

So for your scenarios:

ml CuPy OpenMPI
# → suggests release/24.04 GCC/13.2.0 … / GCC/14.2.0 … (not GCCcore-only)

ml release/24.04 GCCcore/13.2.0 CuPy OpenMPI
# → suggests GCC/13.2.0 … / GCC/14.2.0 … (not GCC/14.2.0 GCCcore/13.2.0)

Tests

New RT test using your module tree:

• rt/load_suggest_828, your three scenarios (including a positive load of release/24.04 GCC/14.2.0 CuPy OpenMPI/5.6)

Regression coverage on the same branch:

• rt/load_suggest_cmd_805, Followup on "Show command to load module on failure" #804 #805 disjoint hierarchies
• rt/load_suggest_823, Overcomplicated (wrong?) suggestions for loading failed module #823 minimal / preloaded suggestions (uses the same reproducer family as your tarball)

All three pass on my side.

Please try

If you can, check out IS828-toolchain-suggest and re-run:

ml CuPy OpenMPI
ml release/24.04 GCCcore/13.2.0 CuPy OpenMPI
ml release/24.04 GCC/14.2.0 CuPy OpenMPI   # or your preferred valid combo

Confirm that:

1. Suggestions use GCC/… where OpenMPI needs it (not GCCcore-only)
2. No suggested command mixes GCC/x.y with a non-matching GCCcore/a.b
3. The suggested commands actually load without the dependency warning you saw before

Once you confirm, I’ll merge to main and cut a new point release.

[Flamefire]

I'm a bit worried about the hard-coded GCC/GCCcore there. I mean the suggestion itself was impossible to use, so somewhere in LMod there must be information that the modules cannot be loaded together.
I fear that there are similar cases given the >= 3 toolchains (GCC, Intel, Clang) and deeper toolchains (GCC-OpenMPI, GCC-IntelMPI, ...) so if this case can be handled generically we won't see the same there.
Or is the information not there or to hard to collect?

[mrcawood]

You're right to push back on the hardcoding; that was the wrong approach.

The information is there. Your modulefiles already declare the relationships (GCC depends_on GCCcore, CuPy depends_on GCCcore, etc.). The suggestion code walks spider parent paths but never had access to depends_on metadata in the spider cache, so I took a shortcut with literal GCC/GCCcore names. That doesn't generalize to Intel, Clang, or deeper stacks like GCC-OpenMPI vs GCC-IntelMPI.

I've reworked IS828-toolchain-suggest to use the generic path:

• During the spider walk, depends_on is now recorded into the cache (it was effectively discarded before).
• When merging parent paths from multiple modules, compatibility is decided from those dependencies — e.g. {release, GCC/13.2.0} and {release, GCCcore/13.2.0} merge because GCC/13.2.0 depends on GCCcore/13.2.0, with no compiler names in the logic.
• When paths diverge at the same position, we prefer the element that depends_on the other (outer toolchain over inner prerequisite).
• If a path element has a fixed depends_on for module X but the suggestion would use a different version of X, that suffix gets dropped — same idea as the mixed GCC/GCCcore case, but driven from metadata.

The same RT tests still pass (load_suggest_828 with your module layout, plus the #805 and #823 regressions). I've pushed the updated branch, pull latest IS828-toolchain-suggest before re-testing.

Your three scenarios are still what I'd like you to check:

ml CuPy OpenMPI
ml release/24.04 GCCcore/13.2.0 CuPy OpenMPI
ml release/24.04 GCC/14.2.0 CuPy OpenMPI

Mainly: suggestions should use GCC/… where OpenMPI needs it, no mixed GCC/x + wrong GCCcore/y in one command, and the suggested loads should actually work without the dependency warning.

[Flamefire]

Your three scenarios are still what I'd like you to check:

IIUC your change includes a new "format" of the spider cache saved on disk? That would mean I cannot test that without "damaging" the spider cache used by the production LMod version.

What would be the easiest way to temporarily move the spider cache? Or does LMOD_IGNORE_CACHE ensure that the files won't be written to and the feature can still be tested without a "physical" cache?