SXM5 (H100) over PCIE Nightmare

[IonutZ]

Did you ever figure this out? @MildHotSauce

 

[MildHotSauce]

Yeah, was a boot sequence issue mostly, the BCM needs time to take over the sxm5 card

 

[gsrcrxsi]

Yeah, was a boot sequence issue mostly, the BCM needs time to take over the sxm5 card

Are you also stuck with using the older Ubuntu 20.04/kernel?

 

[MildHotSauce]

I'm using Ubuntu 24.04.3 LTS, NVIDIA-SMI 570.195.03, Driver Version: 570.195.03, CUDA Version: 12.8

 

[krich]

Yeah, was a boot sequence issue mostly, the BCM needs time to take over the sxm5 card

So, what exactly did you need to do? have you disabled initializing the kernel modules automatically when the machine boots? Is you setup the same Pro WS WRX90E-SAGE SE and threadripper amd processor? Do you think you can run this on X870 MSI gaming boards with ryzen processors?

 

[krich]

Yeah, was a boot sequence issue mostly, the BCM needs time to take over the sxm5 card

these are my logs:

sudo dmesg | grep nvidia
[ 4.674546] nvidia: loading out-of-tree module taints kernel.
[ 4.674551] nvidia: module verification failed: signature and/or required key missing - tainting kernel
[ 4.719369] nvidia-nvlink: Nvlink Core is being initialized, major device number 511
[ 4.720085] nvidia 0000:01:00.0: enabling device (0000 -> 0002)
[ 4.749968] nvidia-modeset: Loading NVIDIA UNIX Open Kernel Mode Setting Driver for x86_64 570.211.01 Release Build (dvs-builder@U22-I3-AM27-29-5) Tue Nov 25 16:11:49 UTC 2025
[ 4.806113] [drm] [nvidia-drm] [GPU ID 0x00000100] Loading driver
[ 4.806116] [drm] Initialized nvidia-drm 0.0.0 for 0000:01:00.0 on minor 1

sudo dmesg | grep gsp
[ 67.812295] NVRM: kfspSendBootCommands_GH100: version=0x1, size=0x35c, gspFmcSysmemOffset=0x178f00000
[ 67.812297] NVRM: kfspSendBootCommands_GH100: gspBootArgsSysmemOffset=0x130bb4000
[ 67.812319] NVRM: nvCheckOkFailedNoLog: Check failed: Call timed out [NV_ERR_TIMEOUT] (0x00000065) returned from kfspSendBootCommands_HAL(pGpu, pKernelFsp) @ kernel_gsp_gh100.c:834

so, this "Call timed out" I assume kills the process. My understanding H100 refuses to boot, have no clue how to force it to boot.

 

[MildHotSauce]

So, what exactly did you need to do? have you disabled initializing the kernel modules automatically when the machine boots? Is you setup the same Pro WS WRX90E-SAGE SE and threadripper amd processor? Do you think you can run this on X870 MSI gaming boards with ryzen processors?

Give the BCM more time to take control before powering on your computer and/or try using the open drivers.

 

[krich]

Give the BCM more time to take control before powering on your computer and/or try using the open drivers.

you mean I need to power the SXM5 to PCIe adapter and let it run like this for several minutes and ONLY after that boot the server?

 

[MildHotSauce]

About 12 good Mississippi seconds should do it

 

[CyklonDX]

Let me know if you get this working - i'm quite curious.

 

[Layla]

So @MildHotSauce you never were able to update firmware, but it didn't matter and is working just fine anyway? How much did you pay for the H100? I've been looking for some, and it seems I missed all the good deals that existed from May-July

 

[MildHotSauce]

So @MildHotSauce you never were able to update firmware, but it didn't matter and is working just fine anyway? How much did you pay for the H100? I've been looking for some, and it seems I missed all the good deals that existed from May-July

Oh man I forget maybe 6--7k was a while ago. I don't think you will see those prices again. yeah the boot sequence matters a lot.