Mellanox switch IB <-> ETH switching

[Kuci]

Hi,
I have a Proxmox cluster with CEPH. It contains 5 servers with ConnectX-2 adapters running CEPH using IPoIB. It runs at 40Gbit with not very good performance (large MTU 64k ?). Servers are 1U, so only 1 PCIe slot
Now I want to replace ConnectX-2 (IPoIB) with ConnectX-3 adapters running in ETH mode at 40 Gbit (MTU 1500).
I plan to do this on the fly - without stopping the whole cluster - one server at a time...
BUT, I have a problem that the old adapters run in IPoIB mode, but the new ones will run in ETH.
Is it possible to run one network on IBoIP + ETH on one SX6036 switch?
Is it possible to connect IPoIB and ETH adapters with one SX6036?
Now I have 1 machine connected to switch using ETH, the others are IPoIB and can't see ETH <-> IPoIB.
My SX6036 has the following licenses:
- GW ports number 1 (ok)
- IB enabled true (ok)
- IB port SW HDR speed limitation (ok)
- Eth enabled true (ok)
- Eth port SW speed limit 100Gb (ok)
- Eth L2 enabled true (ok)
- Eth L3 enabled true (ok)
- FCF enabled true (ok)

Thanks for your advice

 

[nexox]

Not that I have tried this particular thing (never had IB switches that supported Ethernet,) but I believe you would need a host to route between the Ethernet network and the IPoIB network, I don't think a switch would do that for you. That means you'd want different subnets for the two and the router host would be a bottleneck until you could fully transition to Ethernet.

 

[necr]

Is it possible to run one network on IBoIP + ETH on one SX6036 switch?

Yes, you would need a VPI license (various ways to get it, you can search the forum). However, you would need to run OpenSM on one of the Proxmox nodes (maybe you are already)

Is it possible to connect IPoIB and ETH adapters with one SX6036?

Yes. See the VPI configuration examples here on the forum.

40 Gbit (MTU 1500).

You're going to be running TCP/IP presumably, I'd recommend going to 9K MTU whenever you can.

- GW ports number 1 (ok) - IB enabled true (ok)
- Eth port SW speed limit 100Gb (ok)

Doesn't make sense to me, the SX6036 has both IB and Eth port support, and it's limited by FDR/56Gbit/s Ethernet, doesn't support 100Gb/s.

I plan to do this on the fly - without stopping the whole cluster - one server at a time...

Finally, I'd recommend an offline migration - there are too many unknowns for you it seems, you can do this faster with offline migration.

 

[NablaSquaredG]

Doesn't make sense to me, the SX6036 has both IB and Eth port support, and it's limited by FDR/56Gbit/s Ethernet, doesn't support 100Gb/s.

That's just the license capability. I also typically generate licenses with just everything maxed out.

- GW ports number 1 (ok)

Only one GW ports looks weird. Should be more if I remember correctly

 

[jabuzzard]

You're going to be running TCP/IP presumably, I'd recommend going to 9K MTU whenever you can.

Be careful. If you are in datagram mode (the default) then the maximum MTU with IPoIB is 2044. If you go to connected mode then the MTU can go to 65520, but the latency goes up and performance is a lot worse on the default MTU of 2044.

 

[necr]

If you are in datagram mode (the default) then the maximum MTU with IPoIB is 2044

You can set the max MTU to 4092 in partitions.conf

I actually meant Ethernet MTU, of course 9K is not available in datagram mode. Connected mode is perhaps only worth it on old CX2/CX3 cards, it's not great.