Repository navigation
tilecpp: key the cubin disk cache on the nvcc that built it - #227
Conversation
| """ | ||
| if NVCC_PATH not in _compiler_fingerprints: | ||
| try: | ||
| identity = subprocess.run([NVCC_PATH, "--version"], capture_output=True, text=True, timeout=10).stdout |
There was a problem hiding this comment.
Could we add check=True to subprocess.run()? Without it, a nonzero exit code does not enter the exception handler.
There was a problem hiding this comment.
Done in 6150200. With check=True, a nonzero exit raises CalledProcessError, which the existing except (OSError, subprocess.SubprocessError) already catches, so no change to the handler was needed. A failing nvcc --version now falls back to the path instead of fingerprinting whatever it printed. A working nvcc exits 0, so existing fingerprints and cache entries don't change. Added a test with a stub that prints a version and exits 1.
| identity = subprocess.run([NVCC_PATH, "--version"], capture_output=True, text=True, timeout=10).stdout | ||
| except (OSError, subprocess.SubprocessError): | ||
| identity = "" | ||
| _compiler_fingerprints[NVCC_PATH] = identity or os.path.realpath(NVCC_PATH) |
There was a problem hiding this comment.
Could we resolve NVCC_PATH with shutil.which() before applying os.path.realpath() in the fallback? The default is "nvcc", and realpath("nvcc") resolves relative to the working directory rather than finding the executable on PATH.
There was a problem hiding this comment.
Done in 6150200. The fallback now resolves shutil.which(NVCC_PATH) before realpath, so it finds the same binary the subprocess call would, matching selector._nvcc_version_supported. If nothing is found on PATH, the fingerprint is the NVCC_PATH string as given rather than a path in the working directory, so the key doesn't depend on where the process runs. Tests cover a bare nvcc found on PATH (through a symlink) and a missing one.
The cache key covered the kernel source, template parameters and arch but not the compiler. After switching TILECPP_NVCC_PATH or upgrading the CUDA toolkit, previously cached cubins were silently reused, so compiler fixes (or regressions) were not picked up until the cache was cleared or TILECPP_DISABLE_CACHE=1 was set. Add a per-process fingerprint of the nvcc in NVCC_PATH (its --version output, or its resolved path when it cannot run) to the source hash. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de>
…print Address review on the compiler fingerprint: - Pass check=True to the nvcc --version probe, so a nonzero exit is treated as a failed probe and falls back to the path instead of fingerprinting whatever it printed. The existing except clause already covers CalledProcessError (a SubprocessError). - Resolve NVCC_PATH with shutil.which() before realpath() in the fallback. The default is the bare name "nvcc", which realpath() resolved against the working directory rather than PATH. When nvcc is not found at all, fingerprint the NVCC_PATH string as given. A working nvcc exits 0, so its fingerprint and existing cache entries are unchanged. Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
6150200 to
249764d
Compare
|
Thanks @hannahli-nv, rebased onto main (759fb1a). The branch now has only this PR's two commits, unchanged. |
|
/ok to test 249764d |
Description
The tilecpp cubin disk cache key covers the kernel source, the template parameters and the target arch, but not the compiler. After switching
TILECPP_NVCC_PATHor upgrading the CUDA toolkit, cubins built by the previous nvcc are silently reused. A compiler fix or regression is then not picked up until the cache is cleared orTILECPP_DISABLE_CACHE=1is set. This also matters when retesting compiler-related reports such as #225 with another toolkit.This change adds a per-process fingerprint of the nvcc in
NVCC_PATHto the source hash: itsnvcc --versionoutput, or, when that fails or exits nonzero, its path as found onPATH. The fingerprint is computed once per compiler path, so the extra cost is onenvcc --versionper process.Test:
test_cubin_cache_is_keyed_on_compilercompiles the same kernel with two stub compilers that report different versions and checks that the second compile does not return the first cubin. Fails on main and passes with this change. Three more tests cover the fallback: a stub that prints a version but exits 1, a barenvccfound onPATHthrough a symlink, and annvccthat is not found. Each fails without the second commit. After rebasing onto main, the whole file gives 18 passed on Linux (GH200,CUTILE_TUTORIALS_BACKEND=tilecpp).I emailed my signed CLA (individual) to TileGym@nvidia.com on 2026-11-03.
CI Configuration
Checklist
./format.sh)