fix: skip CUDA_PATH / HIP_PATH DLL directories that do not exist - #2349
Open
emptyngton wants to merge 1 commit into
Open
fix: skip CUDA_PATH / HIP_PATH DLL directories that do not exist#2349emptyngton wants to merge 1 commit into
emptyngton wants to merge 1 commit into
Conversation
os.add_dll_directory() raises FileNotFoundError [WinError 3] when the directory does not exist, so a CUDA_PATH or HIP_PATH left pointing at an uninstalled or upgraded toolkit makes "import llama_cpp" fail outright on Windows, before any user code runs. Check each candidate directory with os.path.exists() before registering it. Directories are tested individually, so a partially removed toolkit still contributes whichever of bin/lib remain rather than raising. Reported previously in abetlen#1077, where a stale CUDA_PATH pointing at an uninstalled CUDA 11.2 broke the import on a machine with 12.3 installed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
On Windows,
os.add_dll_directory()raisesFileNotFoundError: [WinError 3]if the path does not exist. Inllama_cpp/_ctypes_extensions.pytheCUDA_PATHandHIP_PATHsub-directories are passed to it unconditionally.So if either variable points at a toolkit the user has since uninstalled, upgraded, or moved,
import llama_cppfails outright — before any user code runs, and regardless of whether that backend is even in use:Repro on any Windows machine:
A stale toolkit variable is common — CUDA upgrades in particular tend to leave the old
CUDA_PATHbehind — and because the failure surfaces inside the loader rather than at the offending variable, it reads as a broken wheel or a broken install.This was reported in #1077:
CUDA_PATHpointed at an uninstalled CUDA 11.2 while 12.3 was actually installed. That issue was closed after the reporter corrected their environment, but the loader itself was never changed, so it still happens today.Solution
Check each candidate directory with
os.path.exists()before registering it.Both variables use the same
bin/libsub-directories, so the two near-identical blocks collapse into one loop. Each directory is tested individually, which also means a partially removed toolkit contributes whichever ofbin/libsurvive instead of raising.No behaviour change for a valid toolkit: every directory registered today is still registered.
Notes
os.environ.get(...) is Noneis used rather thanin os.environso an empty-string value is handled as well.