Hey all,
I've been slowly working on a sort of covid-keep-oneself occupoed setup for my home network for a while, of which the scope eventually crept to a 40gbit infiniband network, which is working, in any case for what I originally intended, which is faster storage, both smb and nfs-rdma. Going to do some srp things too, but as that's all 1 hop, that seems unlikely to be a problem.
One thing which is a nice to have, is having the internet connection working over ipoib too, which it currently refuses to do, so I still have a gbit network also running to each machine.
The server runs two VM's one for pfsense, and one for pihole, which is de network's dns server. Getting the connectX-3 to work on pfsense was a bit of a hassle eventually requiring me to build the mellanox kernel modules on a sandbox freebsd 12 install, then copying them over to the pfsense box. Pfsense now recognizes the SR-IOV split connectX-3 cards on the server, and it's happy to talk to the server and my remote workstation - sort of.
Despite pfsense having the ib interface listed and working, it doesn't route internet traffic over it, for instance. Firewall settings for all LANs on the pfsense guest are identical.
Is there something really basic I'm missing why pfsense won't route internet traffic over the ib adapter like it does for the other LANs? Opensm is enabled, and obviously the adapter does have some connectivity.
I've spent a number of hours reading about infiniband configuraion, which usually gives me some direction, but here I still feel like I have no idea why it isn't working.
As an example of something that mistifies me is that can ssh from my workstation or server (that runs the pfsense guest) to pfsense LAN over the gbe network, but not over the ipoib one. I can ssh the other way fine, from pfsense -> workstation or server. Is this something wrong with the SR-IOV setup that only allows partial connectivity?
Anyone who has a suggestion on some reading material that might help, or perhaps just knows what the problem is, I'd really like to hear about it.
Everything is running in connected mode.
Oh the switch is an unmanaged IS5022. It seemed fine for my faster storage goal, but it would be nice to have if it was also capable of moving internet traffic, or at least, that I understand why it can't.
I've been slowly working on a sort of covid-keep-oneself occupoed setup for my home network for a while, of which the scope eventually crept to a 40gbit infiniband network, which is working, in any case for what I originally intended, which is faster storage, both smb and nfs-rdma. Going to do some srp things too, but as that's all 1 hop, that seems unlikely to be a problem.
One thing which is a nice to have, is having the internet connection working over ipoib too, which it currently refuses to do, so I still have a gbit network also running to each machine.
The server runs two VM's one for pfsense, and one for pihole, which is de network's dns server. Getting the connectX-3 to work on pfsense was a bit of a hassle eventually requiring me to build the mellanox kernel modules on a sandbox freebsd 12 install, then copying them over to the pfsense box. Pfsense now recognizes the SR-IOV split connectX-3 cards on the server, and it's happy to talk to the server and my remote workstation - sort of.
Despite pfsense having the ib interface listed and working, it doesn't route internet traffic over it, for instance. Firewall settings for all LANs on the pfsense guest are identical.
Is there something really basic I'm missing why pfsense won't route internet traffic over the ib adapter like it does for the other LANs? Opensm is enabled, and obviously the adapter does have some connectivity.
I've spent a number of hours reading about infiniband configuraion, which usually gives me some direction, but here I still feel like I have no idea why it isn't working.
As an example of something that mistifies me is that can ssh from my workstation or server (that runs the pfsense guest) to pfsense LAN over the gbe network, but not over the ipoib one. I can ssh the other way fine, from pfsense -> workstation or server. Is this something wrong with the SR-IOV setup that only allows partial connectivity?
Anyone who has a suggestion on some reading material that might help, or perhaps just knows what the problem is, I'd really like to hear about it.
Everything is running in connected mode.
Oh the switch is an unmanaged IS5022. It seemed fine for my faster storage goal, but it would be nice to have if it was also capable of moving internet traffic, or at least, that I understand why it can't.