Honestly, this shouldn’t even be an issue, but since it actually happened to me, it’s worth writing down.

In Arch Linux, a kind soul (let’s call them A) packaged CUDA, and another kind soul (B) packaged NVHPC. Probably to save disk space, B pointed NVHPC’s internal CUDA directory directly to the standalone version provided by A. This means if you install CUDA via pacman and then install NVHPC, running nvcc inside the NVHPC environment actually just invokes the standalone nvcc from the cuda package.

You can see right through this trick with a single command:

readlink -f $(which nvcc)

The official full NVHPC bundle from NVIDIA ships with its own self-contained version of CUDA, which obviously minimizes conflicts with its other components. Arch Linux’s aggressive space-saving approach, however, introduces a rather confusing trap: when the standalone CUDA package upgrades to a newer version (say, 13.5), while NVHPC is still built against an older one (like 13.3), the resulting binaries can become buggy. Put simply, nvcc has evolved, while the rest of the toolchain (like nvfortran) is still stuck a couple of generations behind. I ran into this exact problem recently, when compiling my OpenACC code using both nvfortran and nvcc, the CUDA version mismatch caused a massive 2x performance drop.

Of course, you could manually downgrade the cuda package, but that drags a whole chain of dependent packages down with it, such as ollama-cuda. Plus, it creates a huge mental overhead, having to double-check versions and dependencies every single time you run a system update is exhausting.

The more straightforward route is to ditch the pacman version of NVHPC altogether and just download the official installer from NVIDIA. Unsurprisingly, this eats up several extra tens of gigabytes of disk space.

At the end of the day, it’s a trade-off between disk space and maintenance cost. And honestly, saving my brainpower is the top priority-the hard drive can take the hit.