How to bridge Infiniband and Ethernet utilizing a Homebrew router

[hindutool]

Hello,
I have been scouring through the forums but cannot find the answer to several questions for my particular setup. In fact, I am unsure if this is even possible but here goes...

I have a home brew router with a Mellanox MCX456A-ECAT NIC (Dual port 100GB can do either ETH or IB) AND a MELLANOX MCX-416A-CCAT (Dual port 100GB ETH only).
I also have 6-8 supermicro servers that will be functioning as part of an Openstack environment (Controller node, compute node, block storage node, object storage node, etc.) - EACH server has an MCX-455A-ECAT (Single port 100GB IB/ETH - currently in IB mode) with the option to install a MCX-415A-CCAT (Eth only NIC) alongside the IB adapter.

So I have two options: either run a pure IB network connected to Ethernet or run a pure ETH network. I would ideally like to connect all machines to an IB switch, and then have that switch connect to the router and out to the internet so that I use the speed and isolation of the IB protocol for internal data transfer.

I have several questions:

1) with respect to the MCX-456A-ECAT (dual port) - can I operate one port as IB and one port as ETH and "route" between the two ports, or should I use 2 separate NICs and route from the MCX-456A-ECAT to the MCX-416A-CCAT which are both located on the router?
- I ask this because I have read posts and seen videos that suggest that the MCX-456A-ECAT can have one port as IB and another port as ETH simultaneously but FiberStore who I am purchasing transceivers from have informed me that this is not possible. I reached out to NVIDA, but they informed me that I would need to have a service contract with them so no luck there.
This leads to question number 2

2) I know that I will require an IB switch to connect multiple IB devices, but my question is this: How do I connect my IB devices to the Ethernet network?
Can I use my router - which has both IB AND Ethernet NICs as an IB gateway? if this is not a good idea can I have another machine with both IB and ETH NICs to serve this purpose?
If so, how do I go about configuring a machine to route between IB and ETH to serve as an IB gateway?
I know there are standalone IB appliances (IB gateway switches) that function as a an IB gateway, and that some of them require licenses, etc. I was trying to do this myself
I would like for all nodes in my setup to communicate with the speed and isolation of Infinband but also have each node to have access to the internet via a high speed 100GB Ethernet network.

Finally, from a topology perspective - should I
A) connect all IB devices to IB switch and then connect the IB switch to my router directly or should I use a standalone machine to act as an IB bridge?
OR
B) Should I run parallel networks of ETH AND IB (each node has both types of NICs) connected to their respective switches (ETH and IB) with only the ETH portion connected to the router to provide outside network

I am not an IT or network engineer by profession, just someone who is very interested in trying to get my own network setup. I have some knowledge of networks (Layer2/3), bonding and other aspects of networking, as well as DNS/BIND, firewalls/routers, but IB is a completely new area for me. My apologies if I have asked too many disparate questions in one post, or have not phrased my questions correctly. I am grateful for any input or insights you may have. Thank you

 

[necr]

I'd recommend InfiniBand if you have primarily RDMA workloads (NVMeoF, SRP, SMB 3.0, AI/GPU Direct). Connecting your nodes to a switch makes sense, then you connect the switch to the "gateway" Linux box. If you have no RDMA, just go Ethernet.

Sure, you can have a Linux machine as a gateway router between Infiniband IPoIB interface and Ethernet interface on the other port - standard Linux ip_forward will work. Do a few tests with Infiniband, start slowly.

 

[hindutool]

Thank you for your reply. I will be using a fair amount of Samba, so I thought learning about IB would be worthwhile. I guess I have three questions:

1) When you say using ip_forward, is it in the sense that a linux router would forward packets, ie sysctl net.ipv4.ip_forward? and would I then have to enable iptables -t nat -A POSTROUTING -j MASQUERADE in iptables?
2) If the above is indeed true, could I use my current linux router to accomplish this? It is merely a server and has extra PCIe slots where I can put an IB card in it in addition to the ethernet it already has.
3) Do you know offhand of any tutorials that might be out there, or perhaps point me in the general direction of where I should look? I have looked on the internet, but perhaps I am not using the correct search terms. Everything I have read mentions using a commercial IB gateway appliance, but I want to do this myself to learn and have control over the network.

I am grateful for any help you might be able to provide.

 

[nexox]

would I then have to enable iptables -t nat -A POSTROUTING -j MASQUERADE in iptables?

You don't have to use NAT to forward packets, as long as the router has an entry in its routing table to reach the destination network and there's no iptables/nftables filter rule to drop those packets the router can just send them along. The other hosts also need to have a route that points at the router's IP to reach the other subnet, and if you use your existing router then that can just be your default gateway, no more routes requried.

Two possible reasons you're not finding much information: 1) IPoIB kind of sucks, at least it did for a long time, I believe there's some hardware offload available now that might make it less crappy, but ethernet largely won the network battle by the time that was available. 2) Forwarding packets between interfaces is pretty basic and not at all specific to IB, so if your search terms include 'Infiniband' you might just exclude anything useful and only get results about those appliances.

As far as I can tell Samba doesn't really support RDMA yet, only the Windows SMB implementations do, so it sounds like you should just use ethernet and avoid the complication.

 

[hindutool]

If I were to proceed however, I would only need to forward packets from the IB interface to the ETH interface on the router using netplan routing feature specifying the appropriate interfaces - especially given the fact that the router already has net.ipv4.ip_forward flag enabled, correct? Meanwhile ensuring no packets are being blocked by iptables between the two interfaces - am I correct in this assumption?

I will look up packet forwarding between two interfaces, perhaps this is the phrasing I was looking for in my search terms.

The IB hosts will be connected to an IB switch, this IB switch would in turn be connected to the IB interface on the router, and I would then route this interface to the ETH interface on the router. I am assuming this is the scenario I would need.

I am trying to learn more about infiniband, just as I did with FC (hardware was cheap). I hope to add one more notch in my knowledge of networking infrastructure because I find it fascinating.

One final thought has occurred to me. If each machine in my openstack setup has both an IB AND eth interface in it, could I just connect the IB interface of each machine to an IB switch and just use that to transfer information internally for large amounts of information, and have the ethernet interfaces of each machine connected to an ethernet switch which in turn connects to a router for access to the internet? Would this be a smarter topology? I have not purchased an IB switch yet, but I have my eye on some old 100GB Mellanox IB switches, as the IB interfaces in all my machines are 100GB.

Thank you again for all the input, and I am grateful for any insights anyone may have to help me along this journey.

 

[nexox]

If you have ip_forward=1 and no filters then any packet the host receives (addressed by MAC) with a destination IP that isn't local to the host, it will try to forward based on its routing table(s). For simple networks that means you just have to assign addresses on interfaces on each subnet for the router and you'll automatically get a local route that will get packets to the right place.

If you set up a second ethernet network that's fine, but if you're not going to use applications with native RDMA/IB support, you should not use Infiniband. It won't be faster and the amount you'll learn will be pretty limited. Even if you do need RDMA it would be a lot more instructive and useful to run it over ethernet, that's pretty much where the future is headed.

 

[Rand__]

Is there a reason not to get a MLX switch thats intended to be a gateway between IB and ETH?
A SX6036G, or a SX6036/SX6018/SX6012 with... liberated Gateway license, or a converted EMC one? Lots of threads for these here

I mean I have no clue what the cost is for these atm but they used to be dirt cheap...

 

[hindutool]

For me, I was thinking about getting a 100GB IB switch since all my machines have 100GB IB interfaces. The problem I ran into was 1) Cost for these gateway switches that support 100GB and 2) licensing costs - alot of the switches I found required one to purchase (let alone find) a license that would activate gateway functionality, much of the old equipment that was cheap did not have this functionality and I would have to find someone who could sell me a license to open this functionality since support no longer exists for EOL products.

In any case, I want to try and solve this particular problem myself with off the shelf hardware, just like I did with my router. I built my own router with server hardware incorporating WiFi 7 modules (6GHz, 5 Ghz, and 2.4 GHz) with dual 100GB ethernet ports and 25 GB WAN ports to be used with a 100GB ethernet switch (Celestica Seastone DX010).

I also did this with an 8GB FC switch and FC HBAs which introduced me to the world of fiberoptic transceivers and cabling, along with learning about FC.

I am an ED physician by trade but technology is something I am really passionate about, ever since I built my first 80386 machine back in the 80's.

My whole Openstack setup is built from the ground up using servers and hardware I cobbled together. My current project is learning about IB, hence my reason for asking about IB gateway to ethernet.

Are IB gateway appliances out there? YES. Am I running mission critical RDMA applications? NO. Could this be a waste of time? YES - but I would like to learn about this technology and would like to construct my own IB to eth gateway.

I have been able to lookup various tutorials on how to build an IB network with the hardware I have (Still need IB switch), BUT I have been UNABLE to find information on building a home-brew IB gateway - ie linking IB to ethernet.

Every post I have already read asks what Rand_ has asked - why not buy a commercial appliance that already does this? The answer is this - I would like to build and configure this type of appliance myself if this is possible - so that I may learn how this appliance works at a fundamental network and configuration level. Plus I do not want to have to pay licensing fees for something I want to learn how to do myself.

Based on the discussion so far, it seems building my own IB gateway appliance is theoretically possible so I am trying to figure out how. I was looking for input from the plethora of people on this forum who work in IT, and figured many of you have more exposure to IB than I do.

If I may, I would like to crystallize my questions more succinctly based on what I have learned from the responses:

1) Given that building an IB gateway is possible from off the shelf hardware, should I:

a) use my current router to double as an IB gateway or

b)use a separate machine with ETH and IB interfaces to run as a IB gateway?

The router is an DELL 730xd with Dual E5-26xx V4 CPU and 128 GB ram with an extra PCIe 3.0 x16 slot so it is more than capable. Is there a danger using my router to do this from a security standpoint?

2) Given that I will be using a machine with BOTH IB and ETH interfaces, and given that ipv4_forward=1 would I only need to add routing to the netplan configuration of the linux router specifying the subnets to be routed and interfaces to be used? If so, what is the best way to accomplish this? Will I need to alter iptables in any way for this?

3) I appreciate all of the input and help so far, and I know many of you are busy. Do you know of any tutorials or areas to look for further information on this subject? I now realize that I was limiting my results by including IB in my search terms and will instead search for packet forwarding in a more generalized context, but would love to know of more informational resources out there.

I truly appreciate everyone's time and consideration of this matter.

 

[necr]

2) Given that I will be using a machine with BOTH IB and ETH interfaces, and given that ipv4_forward=1 would I only need to add routing to the netplan configuration of the linux router specifying the subnets to be routed and interfaces to be used? If so, what is the best way to accomplish this? Will I need to alter iptables in any way for this?

Yes, it's easier than you think - just do it! Install opensm, configure IPoIB interfaces, your Linux will route packets between interfaces as it should.

 

[hindutool]

Thank you for the information. I now have a better idea of how to approach this. Hopefully my next post will be one of success. I am going to do some reading on opensm. If I am using a dedicated machine to act at the IB gateway, is there any other software you would recommend? And just to clarify, would I do the forwarding of packets from one interface to the other using netplan? Or am I correct in assuming opensm handles this task?

Thank you again for your assistance.

 

[nexox]

OpenSM handles the managed switch kind of tasks for unmanaged IB switches, it doesn't forward packets, it mostly just breaks in weird ways to keep you on your toes. The Linux kernel forwards packets just fine without any additional software, I'm not familiar with netplan, but you can do it without that.

 

[i386]

1) Cost for these gateway switches that support 100GB

Do you have a link? I never saw a switch with >40GBit/s and gateway functionallity, only a 2u appliance from mellanox/nvidia with 8x dual port cx-7*...

* NVIDIA Mellanox Skyway InfiniBand to Ethernet Gateway
Edit:added link to appliance

 

[hindutool]

Mellanox SB7890 36-Port EDR 100Gb/s Infiniband Switch System - Silver for sale online | eBay

Find many great new & used options and get the best deals for Mellanox SB7890 36-Port EDR 100Gb/s Infiniband Switch System - Silver at the best online prices at eBay! Free shipping for many products!

www.ebay.com

Mellanox SB7890 36-Port 100Gb/s Switch-IB 2 Based EDR InfiniBand 1U Switch 2x AC | eBay

Find many great new & used options and get the best deals for Mellanox SB7890 36-Port 100Gb/s Switch-IB 2 Based EDR InfiniBand 1U Switch 2x AC at the best online prices at eBay! Free shipping for many products!

www.ebay.com

Mellanox SB7800 Infiniband EDR Switch 36 x 100Gb QSFP28 Ports 2x AC PSU *No SW* | eBay

Find many great new & used options and get the best deals for Mellanox SB7800 Infiniband EDR Switch 36 x 100Gb QSFP28 Ports 2x AC PSU *No SW* at the best online prices at eBay! Free shipping for many products!

www.ebay.com

These switches are 100GB infiniband but do not have gateway capabilities.... The one you listed does but they are VERY VERY expensive. This the reason I want to build my own infiniband gateway. The 100GB IB gateways are out of reach for guys like me.

one of the above switches would then connect to my IB gateway giving it access to ethernet.

Beware, some of the switches have an I2C management port only (managed externally) while some have both I2C and OOB/console ports (managed internally).