Hi all,
This is largely a copy/paste from a couple other product specific forums as well as with the FreeBSD folks. I figured I'd also pop this here to see if someone has some experience with this particular issue, or if I'm just spinning my wheels here.
I've been using OPNSense and TrueNAS in a home lab environment for some time now, but I'm in the midst of building a new box and the setup is killing me here.
I'll try to be brief here. Firstly, the relevant bits:
I have a Broadcom P225P/BCM57414 NIC with SR-IOV enabled. The initial configuration seemed fine. The latest drivers from VMWare and the latest FW for the NIC has been applied. This at the moment seems to be a general FreeBSD issue rather than anything product specific.
The initial deployment of a new OPNSense VM (of which is on FreeBSD 13) was met with some issues regarding the drivers for the cards. It showed:
...similar to another thread I'd seen on the OPNSense forums. I was able to manually load the drivers via "kldload if_bnxt" with success, and added it to loader.conf.local, also with success upon a reboot.
However, upon rebooting the host, it all went to pot.
After having configured some of the new interfaces within the firewall UI, and rebooting, all the newly created interfaces disappeared. I was met with the same "none0@" output above. However, upon checking to see if something didn't run as expected:
I rebooted the host again, only to be met with the VM not starting up at all as the NIC seemed to drop out of having SR-IOV enabled. It indicated it was enabled but required a reboot. I rebooted once more, and one port became active, one still required a reboot.
The second reboot seemed to resolve the issue, but I was still experiencing these interfaces not loading. I deleted the loader.conf.local file to prevent the drivers from being loaded, rebooted, manually tried to load the drivers again but was met with the same message above. I don't quite know how that would have been the case as there was nothing to explicitly call that driver to load.
The only thing I could see from dmesg was
In other attempts to boot, I saw:
Which seemed to repeat endlessly.
I now have no idea why or how that occurred and am at a loss here. I'm guessing this is a guest issue or that FreeBSD, even though they explicitly say the virtual function is supported, actually isn't. I see similar behavior with TrueNas, which is on FreeBSD 12.
I've taken the OPNSense software above and did a fresh install to try to mimic the original steps, but it's still in the same boat. I've also installed a fresh vanilla instance of FreeBSD 13 to see if it happened to be something specific to the other builds and it's the exact same thing. I've not yet tried to do a full passthrough of the NIC yet, but plan to for testing.
I'm sure I'm missing something here, but this side of the house is not my forte. Happy to provide any additional info required. Also apologies if this isn't the best place to pop this, but it seemed this subforum was the best all encompassing place.
TIA
This is largely a copy/paste from a couple other product specific forums as well as with the FreeBSD folks. I figured I'd also pop this here to see if someone has some experience with this particular issue, or if I'm just spinning my wheels here.
I've been using OPNSense and TrueNAS in a home lab environment for some time now, but I'm in the midst of building a new box and the setup is killing me here.
I'll try to be brief here. Firstly, the relevant bits:
Build’s Name: TEH
Operating System/ Storage Platform: ESXi 7 (enterprise plus license)
CPU: Epyc 7282
Motherboard: Supermicro H12SSL-i
RAM: 256GB @3200
Add-in Cards:
-Broadcom BMC57414/P225p
-Broadcom 9305-24i
-GTX 1660 ti
-Asus Hyper M.2 (4x4x4x4 bifurcated slot)
I have a Broadcom P225P/BCM57414 NIC with SR-IOV enabled. The initial configuration seemed fine. The latest drivers from VMWare and the latest FW for the NIC has been applied. This at the moment seems to be a general FreeBSD issue rather than anything product specific.
The initial deployment of a new OPNSense VM (of which is on FreeBSD 13) was met with some issues regarding the drivers for the cards. It showed:
none0@pci0:11:0:0: class=0x020000 rev=0x00 hdr=0x00 vendor=0x14e4 device=0x16dc subvendor=0x14e4 subdevice=0x16d7
vendor = 'Broadcom Inc. and subsidiaries'
device = 'NetXtreme-E Ethernet Virtual Function'
class = network
subclass = ethernet
none1@pci0:19:0:0: class=0x020000 rev=0x00 hdr=0x00 vendor=0x14e4 device=0x16dc subvendor=0x14e4 subdevice=0x16d7
vendor = 'Broadcom Inc. and subsidiaries'
device = 'NetXtreme-E Ethernet Virtual Function'
class = network
subclass = ethernet
none2@pci0:27:0:0: class=0x020000 rev=0x00 hdr=0x00 vendor=0x14e4 device=0x16dc subvendor=0x14e4 subdevice=0x16d7
vendor = 'Broadcom Inc. and subsidiaries'
device = 'NetXtreme-E Ethernet Virtual Function'
class = network
subclass = ethernet...similar to another thread I'd seen on the OPNSense forums. I was able to manually load the drivers via "kldload if_bnxt" with success, and added it to loader.conf.local, also with success upon a reboot.
However, upon rebooting the host, it all went to pot.
After having configured some of the new interfaces within the firewall UI, and rebooting, all the newly created interfaces disappeared. I was met with the same "none0@" output above. However, upon checking to see if something didn't run as expected:
kldload if_bnxt
kldload: can't load if_bnxt: module already loaded or in kernelI rebooted the host again, only to be met with the VM not starting up at all as the NIC seemed to drop out of having SR-IOV enabled. It indicated it was enabled but required a reboot. I rebooted once more, and one port became active, one still required a reboot.
The second reboot seemed to resolve the issue, but I was still experiencing these interfaces not loading. I deleted the loader.conf.local file to prevent the drivers from being loaded, rebooted, manually tried to load the drivers again but was met with the same message above. I don't quite know how that would have been the case as there was nothing to explicitly call that driver to load.
The only thing I could see from dmesg was
bnxt0: <Broadcom NetXtreme-E Ethernet Virtual Function> mem 0xffa04000-0xffa07fff,0xff900000-0xff9fffff,0xffa00000-0xffa03fff at device 0.0 on pci5
bnxt0: Timeout sending HWRM_VER_GET: (timeout: 1000) seq: 0
bnxt0: attach: hwrm ver get failed
bnxt0: IFDI_ATTACH_PRE failed 60
device_attach: bnxt0 attach returned 60In other attempts to boot, I saw:
bnxt0: Timeout sending HWRM_RING_ALLOC: (timeout: 2000) seq: 225
bnxt0: Timeout sending HWRM_FUNC_RESET: (timeout: 2000) seq: 226
bnxt0: Timeout sending HWRM_FUNC_RESET: (timeout: 2000) seq: 227
bnxt0: Timeout sending HWRM_FUNC_RESET: (timeout: 2000) seq: 228
bnxt0: Timeout sending HWRM_FUNC_RESET: (timeout: 2000) seq: 229
bnxt0: Timeout sending HWRM_FUNC_RESET: (timeout: 2000) seq: 230
bnxt0: Timeout sending HWRM_FUNC_RESET: (timeout: 2000) seq: 231
bnxt0: Timeout sending HWRM_FUNC_RESET: (timeout: 2000) seq: 232
bnxt0: Timeout sending HWRM_FUNC_RESET: (timeout: 2000) seq: 233
bnxt0: Timeout sending HWRM_FUNC_RESET: (timeout: 2000) seq: 234
bnxt0: Timeout sending HWRM_FUNC_RESET: (timeout: 2000) seq: 235
bnxt0: Timeout sending HWRM_FUNC_RESET: (timeout: 2000) seq: 236
bnxt0: Timeout sending HWRM_FUNC_RESET: (timeout: 2000) seq: 237
bnxt0: Timeout sending HWRM_FUNC_RESET: (timeout: 2000) seq: 238
bnxt0: Timeout sending HWRM_PORT_PHY_QCFG: (timeout: 2000) seq: 239
bnxt0: Timeout sending HWRM_PORT_PHY_QCFG: (timeout: 2000) seq: 240
bnxt0: Timeout sending HWRM_PORT_PHY_QCFG: (timeout: 2000) seq: 241
bnxt1: Timeout sending HWRM_PORT_PHY_QCFG: (timeout: 2000) seq: 15
bnxt1: Timeout sending HWRM_PORT_PHY_QCFG: (timeout: 2000) seq: 16
bnxt1: Timeout sending HWRM_PORT_PHY_QCFG: (timeout: 2000) seq: 17
bnxt2: Timeout sending HWRM_PORT_PHY_QCFG: (timeout: 2000) seq: 15
bnxt2: Timeout sending HWRM_PORT_PHY_QCFG: (timeout: 2000) seq: 16
bnxt2: Timeout sending HWRM_PORT_PHY_QCFG: (timeout: 2000) seq: 17
bnxt0: Timeout sending HWRM_PORT_PHY_QCFG: (timeout: 2000) seq: 242
...Which seemed to repeat endlessly.
I now have no idea why or how that occurred and am at a loss here. I'm guessing this is a guest issue or that FreeBSD, even though they explicitly say the virtual function is supported, actually isn't. I see similar behavior with TrueNas, which is on FreeBSD 12.
I've taken the OPNSense software above and did a fresh install to try to mimic the original steps, but it's still in the same boat. I've also installed a fresh vanilla instance of FreeBSD 13 to see if it happened to be something specific to the other builds and it's the exact same thing. I've not yet tried to do a full passthrough of the NIC yet, but plan to for testing.
I'm sure I'm missing something here, but this side of the house is not my forte. Happy to provide any additional info required. Also apologies if this isn't the best place to pop this, but it seemed this subforum was the best all encompassing place.
TIA
Last edited: