FreeBSD/pfSense guest 10GbE SR-IOV VF successes?

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

arglebargle

H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈
Jul 15, 2018
657
244
43
I've been researching FreeBSD supported SR-IOV capable network cards for a couple of days and I'm not finding much information on support. Has anyone here built a system that passes SR-IOV VFs to a FreeBSD guest and had it work reliably? The big gotcha seems to be driver VF support on the BSD guest, passthrough of the whole nic seems successful in a lot of cases but passthrough of VFs only seems pretty niche (even though that's the entire point of SR-IOV, sheesh.)

I'd like to host pfSense VMs on Proxmox and use SR-IOV to pass nic VFs to the guests.

I've tested Mellanox CX2 and CX3 cards fairly extensively and as of FreeBSD 12 VF passthrough does work, but the driver has issues and leaves the VF in an indeterminate state on guest shutdown requiring a host reboot before the VF can be used again.

My preference would be for a Chelsio card but I'll consider literally anything 10GbE that has a confirmed functional driver for SR-IOV VFs at this point.

Thanks for your time!
 
Last edited:
  • Like
Reactions: gigatexal

WANg

Well-Known Member
Jun 10, 2018
1,302
967
113
46
New York, NY
Hm...give Solarflare a try - I know that their Linux drivers are usually pretty solid - passing PCIe VFs to it should not be a major headache - although I am not sure how the VFs will be treated by the Solarflare drivers in FreeBSD. Then maybe give the Intel ixgbe (I350) stuff a try as well. I don't have my hands on a Chelsio Terminator so I can't tell you whether that's doable or not, either.
 
Last edited:

arglebargle

H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈
Jul 15, 2018
657
244
43
Yeah, I was planning to give the I350 a try in the next little bit.

I kinda splurged on eBay this morning ordering test cards so I've got Chelsio, Solarflare and Qlogic coming in the next week. I didn't realize how comparatively overpriced Chelsio cards are, wow. I can literally pick up 2-3 comparable Mellanox or Solarflare cards for the price of a Chelsio T420-CR.

I've heard through the grapevine, and by that I mean google searches for SR-IOV and Intel turn up a ton of problem reports, that Intel 10GbE support is possibly not so great right now. I've been told someone broke the driver a couple of months back and it hasn't been fixed yet; I don't have confirmation of that, however.

If you have a chance to spin up a FreeBSD 11.1 or 11.2 install with VF passthrough from one of your solarflare cards I'd love to hear the results. If not I'll run tests when my cards arrive sometime next week. (Seriously, don't waste your time if it's a bother.)

I'm going to have quite the stack of spare nics to resell when this is done, oh man.
 
Last edited:

WANg

Well-Known Member
Jun 10, 2018
1,302
967
113
46
New York, NY
Nah, it's no bother - I actually have to bring the ESXi instance down this weekend due to a FreeNAS upgrade anyways. I'll toss up a Promox VE installer ISO, a spare 256GB M2 SSD, and see how far I can go with the SolarFlare cards.

Note: Didn't happen. For some reason the Proxmox VE ISO just refuses to mount the "CD" image on a UNetBootin formatted thumb drive, so I wasn't able to install it. Something about this Intel 2500 Pro SSD I fished out of a spare Dell Latitude also irked me as it didn't want to install grub either. I'll probably need some extended downtime with this t730 in order to get Proxmox going.
 
Last edited:

arglebargle

H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈
Jul 15, 2018
657
244
43
Thanks for trying. A couple of cards have arrived, amusingly 3 of them are Mellanox cards that I've already tested the driver for so they help me not at all. I've got a solarflare sfn6xxx card to test though, so I'll toss that in tonight or tomorrow and see what happens.

I had a similar experience with an Intel 2500 Pro SSD -- were you trying to install proxmox on it and boot in UEFI mode? If you do a ZFS root install Proxmox refuses to make an EFI partition or install UEFI grub. It's annoying, they have a valid reason for slacking on it with raid installs but for single disks they don't bother setting up UEFI grub either. You should be able to boot the Proxmox install in legacy mode but you have to manually set up UEFI booting. I made a thread asking about it in the Linux admin forum with the steps to re-use the solaris reserved partition for EFI.
 

arglebargle

H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈
Jul 15, 2018
657
244
43
Hi, arglebargle
Have you had any update for this topic? I am wondering 10GB card that works for FreeBSD either. Intel card is not so great at this time since the VF driver is troublesome
I moved a few months ago and all of my lab hardware for testing this stuff is in storage.

The only thing I can say with confidence from my prior testing is make sure the host machine has PCIe ACS support, almost all of the NICs drivers I tested refused to even create VFs without it. Mellanox drivers will create VFs without ACS on Linux but I had a lot of trouble passing those to guest VMs and getting them working unless the drivers on the VM were very recent (FreeBSD 12, recent Linux, etc.)

I did end up buying a machine with ACS support to test this on but I didn't have time to really investigate before I moved. One of these days I'll revisit but I'm not sure when that'll be.

Check out DPDK and OpenvSwitch, those might be easier to work with since they don't depend on the hardware vendor supporting NIC virtualization on older FreeBSD.
 
  • Like
Reactions: istqian

istqian

New Member
Jun 9, 2016
14
14
3
Fukuoka, Japan
I moved a few months ago and all of my lab hardware for testing this stuff is in storage.

The only thing I can say with confidence from my prior testing is make sure the host machine has PCIe ACS support, almost all of the NICs drivers I tested refused to even create VFs without it. Mellanox drivers will create VFs without ACS on Linux but I had a lot of trouble passing those to guest VMs and getting them working unless the drivers on the VM were very recent (FreeBSD 12, recent Linux, etc.)

I did end up buying a machine with ACS support to test this on but I didn't have time to really investigate before I moved. One of these days I'll revisit but I'm not sure when that'll be.

Check out DPDK and OpenvSwitch, those might be easier to work with since they don't depend on the hardware vendor supporting NIC virtualization on older FreeBSD.
Finally I got a Chelsio T520-CR (Huawei OEM), and it works great on FreeBSD 12.1R. SR-IOV VF created, and can passthru to bhyve Guest.
the fact is that only a few guests support cxgbevf, FreeBSD/Linux is ok. windows cannot boot with VF passthough(bhyve problem.), other appliance, such as Pfsense, mikrotik routeros, xpenology, vyos etc, does not build with cxgbevf support.