Getting pfsense to utilize sr-iov infiniband virtual adapter

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

solon

Member
Apr 1, 2021
61
5
8
Hey All,

Here's hoping I'm posting this in the right place, but as pfsense is freebsd based, I felt this was the closest category.

As a keep-oneself-occupied through covid project I've been fiddling with my local network. All was originally instigated by annoyance with 1gbE transfer speeds and not having enough cores to run the vm's I wanted on my living room server.

I have MCX354a-qcbt cards, now flashed to official mellanox firmwares, and then one of them is setup for SR-IOV on a Supermicro X10DRH-iT board. That all seems to be working fine. The server runs a few VM's two of which are a small debian install for a pihole, and a pfsense VM. The pfsense VM is intended to load balance between two internet connections, cable and DSL, because with all the working from home internet downtime and having to curb download habits while my better half is in a video meeting is very annoying. The server (ubuntu, though probably soon back to debian) then has an infiniband network over an IS5022 to offer rdma-nfs storage over the network, which all works fine. (except that my workstation now has both an infiniband and and a cat6 going to it to achieve that, which is why I want pfsense to handle it over ipoib)

So what I want is to add the virtual infiniband ipoib adapter to the pfsense VM so that it can manage dhcp for the infiniband part of the network. The SR-IOV setup appears to be working fine, and I can get as far as a result of:

Code:
[2.5.2-RC][root@pfSense.lan]/root: pciconf -lv

mlx4_core0@pci0:10:0:0:    class=0x028000 card=0x61b015b3 chip=0x100415b3 rev=0x00 hdr=0x00

    vendor     = 'Mellanox Technologies'

    device     = 'MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]'

    class      = networ
But then I get stuck. Everything seems to indicate pfsense should be able to do something with the card from here without, say, installing a freebsd image on a seperate VM and building the mellanox server modules and then copying them over, which is the only thing I found that I haven't done yet.

I can't seem to find anything in the packages that I am missing, and I can't seem to get any other modules loaded through the console, mlx4_ib is needed I imagine, but I can't seem to find a way to get/load it other than building it myself on a different freebsd machine, which seems like it shouldn't be neccesary given that netgate says the card is supported.

Anyway, anyone with thoughts on the subject, I'd love to hear it.
 

solon

Member
Apr 1, 2021
61
5
8
If anyone is interested,

A few hours later I took the steps of installing a sandbox freebsd 12 vm and using it to compile the infiniband modules using the script from here:

Script wouldn't work on the pfsense install, I assume it's missing components to avoid this kind of activity.

I copied all the compiled modules over the /boot/kernel on the pfsense system, told loader.conf to load them, and lo and behold, I now have two infiniband ipoib interfaces in connected mode.

However, pfsense still doesn't see fit to regcognize them as interfaces. Not entirely sure what to do with that from here on out, but if I come up with anything, I'll post it.
 

solon

Member
Apr 1, 2021
61
5
8
Right, scratch that last, forgot to reassign adapters.

Next problem has turned up howerver. Now I have and ib0 and ib1 available in infinisense. When I configure one as DHCP server, that locks up everything - all subnets, so that doesn't appear to work. I'm putting that down to an infiniband related thing for now.

With manual configuration of all nodes on the infiniband network, I have connectivity between the nodes. Even though I can reach the pfsense VM directly from the first cable connected machine, I can't get an internet connection over that line for some reason.

Guess I'm off to read more and more infiniband stuff.

Again, all thoughts would be appreciated.