FreeBSD/pfSense guest 10GbE SR-IOV VF successes?

Discussion in 'FreeBSD and FreeNAS' started by arglebargle, Aug 30, 2018.

  1. arglebargle

    arglebargle H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈

    Joined:
    Jul 15, 2018
    Messages:
    244
    Likes Received:
    75
    I've been researching FreeBSD supported SR-IOV capable network cards for a couple of days and I'm not finding much information on support. Has anyone here built a system that passes SR-IOV VFs to a FreeBSD guest and had it work reliably? The big gotcha seems to be driver VF support on the BSD guest, passthrough of the whole nic seems successful in a lot of cases but passthrough of VFs only seems pretty niche (even though that's the entire point of SR-IOV, sheesh.)

    I'd like to host pfSense VMs on Proxmox and use SR-IOV to pass nic VFs to the guests.

    I've tested Mellanox CX2 and CX3 cards fairly extensively and as of FreeBSD 12 VF passthrough does work, but the driver has issues and leaves the VF in an indeterminate state on guest shutdown requiring a host reboot before the VF can be used again.

    My preference would be for a Chelsio card but I'll consider literally anything 10GbE that has a confirmed functional driver for SR-IOV VFs at this point.

    Thanks for your time!
     
    #1
    Last edited: Aug 30, 2018
    gigatexal likes this.
  2. WANg

    WANg Active Member

    Joined:
    Jun 10, 2018
    Messages:
    183
    Likes Received:
    69
    Hm...give Solarflare a try - I know that their Linux drivers are usually pretty solid - passing PCIe VFs to it should not be a major headache - although I am not sure how the VFs will be treated by the Solarflare drivers in FreeBSD. Then maybe give the Intel ixgbe (I350) stuff a try as well. I don't have my hands on a Chelsio Terminator so I can't tell you whether that's doable or not, either.
     
    #2
    Last edited: Aug 31, 2018
  3. arglebargle

    arglebargle H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈

    Joined:
    Jul 15, 2018
    Messages:
    244
    Likes Received:
    75
    Yeah, I was planning to give the I350 a try in the next little bit.

    I kinda splurged on eBay this morning ordering test cards so I've got Chelsio, Solarflare and Qlogic coming in the next week. I didn't realize how comparatively overpriced Chelsio cards are, wow. I can literally pick up 2-3 comparable Mellanox or Solarflare cards for the price of a Chelsio T420-CR.

    I've heard through the grapevine, and by that I mean google searches for SR-IOV and Intel turn up a ton of problem reports, that Intel 10GbE support is possibly not so great right now. I've been told someone broke the driver a couple of months back and it hasn't been fixed yet; I don't have confirmation of that, however.

    If you have a chance to spin up a FreeBSD 11.1 or 11.2 install with VF passthrough from one of your solarflare cards I'd love to hear the results. If not I'll run tests when my cards arrive sometime next week. (Seriously, don't waste your time if it's a bother.)

    I'm going to have quite the stack of spare nics to resell when this is done, oh man.
     
    #3
    Last edited: Aug 31, 2018
  4. WANg

    WANg Active Member

    Joined:
    Jun 10, 2018
    Messages:
    183
    Likes Received:
    69
    Nah, it's no bother - I actually have to bring the ESXi instance down this weekend due to a FreeNAS upgrade anyways. I'll toss up a Promox VE installer ISO, a spare 256GB M2 SSD, and see how far I can go with the SolarFlare cards.

    Note: Didn't happen. For some reason the Proxmox VE ISO just refuses to mount the "CD" image on a UNetBootin formatted thumb drive, so I wasn't able to install it. Something about this Intel 2500 Pro SSD I fished out of a spare Dell Latitude also irked me as it didn't want to install grub either. I'll probably need some extended downtime with this t730 in order to get Proxmox going.
     
    #4
    Last edited: Sep 3, 2018
  5. arglebargle

    arglebargle H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈

    Joined:
    Jul 15, 2018
    Messages:
    244
    Likes Received:
    75
    Thanks for trying. A couple of cards have arrived, amusingly 3 of them are Mellanox cards that I've already tested the driver for so they help me not at all. I've got a solarflare sfn6xxx card to test though, so I'll toss that in tonight or tomorrow and see what happens.

    I had a similar experience with an Intel 2500 Pro SSD -- were you trying to install proxmox on it and boot in UEFI mode? If you do a ZFS root install Proxmox refuses to make an EFI partition or install UEFI grub. It's annoying, they have a valid reason for slacking on it with raid installs but for single disks they don't bother setting up UEFI grub either. You should be able to boot the Proxmox install in legacy mode but you have to manually set up UEFI booting. I made a thread asking about it in the Linux admin forum with the steps to re-use the solaris reserved partition for EFI.
     
    #5
Similar Threads: FreeBSD/pfSense guest
Forum Title Date
FreeBSD and FreeNAS What VM host do you use for FreeBSD guest? Dec 6, 2017

Share This Page