Hey All,
Here's hoping I'm posting this in the right place, but as pfsense is freebsd based, I felt this was the closest category.
As a keep-oneself-occupied through covid project I've been fiddling with my local network. All was originally instigated by annoyance with 1gbE transfer speeds and not having enough cores to run the vm's I wanted on my living room server.
I have MCX354a-qcbt cards, now flashed to official mellanox firmwares, and then one of them is setup for SR-IOV on a Supermicro X10DRH-iT board. That all seems to be working fine. The server runs a few VM's two of which are a small debian install for a pihole, and a pfsense VM. The pfsense VM is intended to load balance between two internet connections, cable and DSL, because with all the working from home internet downtime and having to curb download habits while my better half is in a video meeting is very annoying. The server (ubuntu, though probably soon back to debian) then has an infiniband network over an IS5022 to offer rdma-nfs storage over the network, which all works fine. (except that my workstation now has both an infiniband and and a cat6 going to it to achieve that, which is why I want pfsense to handle it over ipoib)
So what I want is to add the virtual infiniband ipoib adapter to the pfsense VM so that it can manage dhcp for the infiniband part of the network. The SR-IOV setup appears to be working fine, and I can get as far as a result of:
But then I get stuck. Everything seems to indicate pfsense should be able to do something with the card from here without, say, installing a freebsd image on a seperate VM and building the mellanox server modules and then copying them over, which is the only thing I found that I haven't done yet.
I can't seem to find anything in the packages that I am missing, and I can't seem to get any other modules loaded through the console, mlx4_ib is needed I imagine, but I can't seem to find a way to get/load it other than building it myself on a different freebsd machine, which seems like it shouldn't be neccesary given that netgate says the card is supported.
Anyway, anyone with thoughts on the subject, I'd love to hear it.
Here's hoping I'm posting this in the right place, but as pfsense is freebsd based, I felt this was the closest category.
As a keep-oneself-occupied through covid project I've been fiddling with my local network. All was originally instigated by annoyance with 1gbE transfer speeds and not having enough cores to run the vm's I wanted on my living room server.
I have MCX354a-qcbt cards, now flashed to official mellanox firmwares, and then one of them is setup for SR-IOV on a Supermicro X10DRH-iT board. That all seems to be working fine. The server runs a few VM's two of which are a small debian install for a pihole, and a pfsense VM. The pfsense VM is intended to load balance between two internet connections, cable and DSL, because with all the working from home internet downtime and having to curb download habits while my better half is in a video meeting is very annoying. The server (ubuntu, though probably soon back to debian) then has an infiniband network over an IS5022 to offer rdma-nfs storage over the network, which all works fine. (except that my workstation now has both an infiniband and and a cat6 going to it to achieve that, which is why I want pfsense to handle it over ipoib)
So what I want is to add the virtual infiniband ipoib adapter to the pfsense VM so that it can manage dhcp for the infiniband part of the network. The SR-IOV setup appears to be working fine, and I can get as far as a result of:
Code:
[2.5.2-RC][root@pfSense.lan]/root: pciconf -lv
mlx4_core0@pci0:10:0:0: class=0x028000 card=0x61b015b3 chip=0x100415b3 rev=0x00 hdr=0x00
vendor = 'Mellanox Technologies'
device = 'MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]'
class = networ
I can't seem to find anything in the packages that I am missing, and I can't seem to get any other modules loaded through the console, mlx4_ib is needed I imagine, but I can't seem to find a way to get/load it other than building it myself on a different freebsd machine, which seems like it shouldn't be neccesary given that netgate says the card is supported.
Anyway, anyone with thoughts on the subject, I'd love to hear it.