Proxmox with aoc-s25g-b2s based on STH review

[cyrus104]

So I went with a Supermicro AOC-S25G-B2S for my small home Proxmox cluster based on the STH review for 25Gb cards and well... I should have gone with my gut and stuck with the AOC-S25G-I2S for the Intel version because Proxmox doesn't like the card. I'm working on a reboot and will post some logs but it's something to do with the RoCE and Infiniband support causes a kernel panic that leaves the system operational but the card in a unregistering the nics state.

 

[NablaSquaredG]

Even better... Go with Mellanox

 

[NPS]

I have two Dell 57414 cards and one of them is in my proxmox server. I've got no problems at all. Those cards have no Infiniband. You can disable RoCE in the boot options of the card.

 

[cyrus104]

Disabling RoCE might be helpful. I ran into a known issues with the 6.5 kernel causing some kernel panics. I did install the 6.2 PVE kernel and while the card comes up it never populates in Proxmox or IP Link.

[    1.286682] bnxt_en 0000:01:00.0 eth0: Broadcom BCM57414 NetXtreme-E 10Gb/25Gb Ethernet found at mem 6062310000, node addr ac:1f:6b:07:a8:48
[    1.286698] bnxt_en 0000:01:00.0: 63.008 Gb/s available PCIe bandwidth (8.0 GT/s PCIe x8 link)
[    1.306513] bnxt_en 0000:01:00.1 eth1: Broadcom BCM57414 NetXtreme-E 10Gb/25Gb Ethernet found at mem 6062300000, node addr ac:1f:6b:07:a8:49
[    1.306518] bnxt_en 0000:01:00.1: 63.008 Gb/s available PCIe bandwidth (8.0 GT/s PCIe x8 link)
[    1.336349] bnxt_en 0000:01:00.1 enp1s0f1np1: renamed from eth1
[    1.421500] bnxt_en 0000:01:00.0 eno1np0: renamed from eth0

Those interfaces don't show up anywhere in Proxmox, this is a really new install too and I've done nothing besides enable the non-subscription official repo.

@NablaSquaredG, I just took out a CX456A on my other node in the cluster because it was having issues with VLANs.