[Bug]: Driver container starts Fabric Manager against the wrong fabric path and reports Ready when NVLink5 prerequisites (mlx5_ib/ib_umad) are missing on HGX B200

[shivam-dubey-1]

1. Quick Debug Information

• OS/Version: Canonical Ubuntu 24.04 EKS AMI (linux-modules-extra not installed by default), kernel 6.x
• Container Runtime Type/Version: containerd
• K8s Flavor/Version: AWS EKS 1.35
• GPU Operator Version: v26.3.3 (driver container 580.126.20, Fabric Manager 580.126.20)

2. Issue or feature description

On HGX B200 (p6-b200.48xlarge, AWS EKS), the NVSwitches are managed through the
InfiniBand kernel stack. The driver container detects NVLink5 systems via
_assert_nvlink5_system() (scanning /sys/class/infiniband/*/device/vpd for
SW_MNG). On AMIs that do not load mlx5_ib/ib_umad (e.g. the Canonical
Ubuntu EKS AMIs, which do not include linux-modules-extra by default), no IB
devices are present, so NVLink5 detection fails.

The startup script then falls through to the legacy NVSwitch path
(_assert_nvswitch_system) and starts Fabric Manager against an interface that
has no devices on this platform. FM errors and exits, yet the driver pod reports
Ready (1/1 Running). The GPU fabric stays "In Progress" on all 8 GPUs
indefinitely and all CUDA initialization fails with error 802 ("system not yet
initialized"), while nvidia-smi otherwise looks healthy. The failure is silent
at every operator-visible layer.

Reproduced deterministically on 4 of 4 nodes; unaffected by pod restarts, OS
reboot, or instance replacement (expected, since the prerequisite is missing on
every boot of the AMI).

Related: #1595 (error-handling gap acknowledged there).

3. Steps to reproduce the issue

GPU Operator v26.3.3 (driver container 580.126.20, driver.enabled=true,
toolkit.enabled=true) on AWS EKS, managed node group of p6-b200.48xlarge,
Canonical Ubuntu 24.04 EKS AMI (EKS 1.35). Boot a node and run any CUDA workload.

4. Expected behavior

• If the platform is HGX B200/NVLink5-class hardware and the required IB stack
  (mlx5_ib/ib_umad) is absent, the driver container should fail loudly (pod not
  Ready, explicit log naming the missing prerequisite) rather than start FM
  against the wrong fabric path and report Ready.
• The NVLink5 host prerequisites for custom/minimal AMIs should be documented in
  the GPU Operator platform requirements.

5. Diagnostic output

Fabric Manager log (/var/log/fabricmanager.log in the driver container):

[INFO] Fabric Manager version 580.126.20 is running with the following configuration options
[INFO] Fabric Mode = 0
[INFO] Abort CUDA jobs when FM exits = 1
[INFO] Disabling RPC mode for single node configuration.
[ERROR] request to query NVSwitch device information from NVSwitch driver failed with error:WARNING Nothing to do [NV_WARN_NOTHING_TO_DO]

Manual FM start reproduces the same error.

cuda-validator (init container):

Failed to allocate device vector A (error code system not yet initialized)!
[Vector addition of 50000 elements]

Per-GPU fabric state, all 8 GPUs identical, indefinitely:

Fabric
    State                    : In Progress
    Status                   : N/A
    CliqueId                 : N/A

InfiniBand class lists only the EFA device, no mlx5 entries:

$ ls /sys/class/infiniband/
efa_0

NVSwitch: control node only, no per-switch devices, module not loaded:

$ ls /dev/nvidia-nvswitch*
/dev/nvidia-nvswitchctl
$ lsmod | grep -i nvswitch
(no output)

GPUs and NVLinks otherwise healthy (why the failure is silent):

$ nvidia-smi nvlink --status    # all 8 GPUs
GPU 0: NVIDIA B200 ... Link 0-17: 53.125 GB/s
$ nv-fabricmanager --version
Fabric Manager version is : 580.126.20

Driver pod status throughout: nvidia-driver-daemonset 1/1 Running, no error surfaced.

[tariq1890]

Thanks for raising this @shivam-dubey-1. We are almost done releasing gpu-operator v26.7.0 which will default to the latest driver versions with some improvements that you'd benefit. The new driver containers will error out when fabric manager fails to start up.

Please upgrade to the gpu-operator v26.7.0 and let us know if you still face issues.