Distributed training with NCCL often encounters a frustrating issue: the 'CUDA-capable device busy or unavailable' error leading to handshake timeouts and startup hangs. This problem typically arises when multiple processes compete for GPU resources or when CUDA contexts are not properly initialized. The solution involves several steps: first, ensure that all GPUs are visible and not occupied by other processes using 'nvidia-smi'. Second, set the CUDA_VISIBLE_DEVICES environment variable correctly for each process. Third, adjust NCCL parameters such as NCCL_IB_DISABLE and NCCL_SOCKET_IFNAME to avoid network conflicts. Additionally, using 'torch.distributed' with proper 'init_method' and 'world_size' configuration can prevent deadlocks. For persistent issues, consider adding a small delay before initializing NCCL to allow GPU resources to settle. This guide is essential for ML engineers deploying large-scale models across multiple GPUs, as it addresses a common bottleneck in distributed training workflows.
This article provides a solution for the 'CUDA-capable device busy or unavailable' error that causes NCCL handshake timeouts in distributed training setups. It offers practical debugging steps and configuration fixes, making it a valuable resource for engineers working with multi-GPU environments.