IB switch connect to ethernet network?

sbonikowske

New Member
Aug 1, 2022
6
0
1
Good morning,

Sorry if this is a really dumb question, but I was wondering if it's possible to connect a mellanox 6036 switch to an ethernet switch? I have been trying to read through all the posts about the switches, and there is conflicting info.

I read that the 6036s have a VPI system which allows them to talk ethernet, but is that just local on the switch?

Ultimately my goal is to setup 2 switches:
1 6036 for IB (trying to get RDMA working) and maybe some high speed ethernet traffic. I want to get some cheap dual port connectX 3 cards (any recommendations?)
2 A switch that can maybe to 40g/10g/1g like an arista for the rest of the network, like my internet gateway, and PCs/devices that don't have access to extra connectivity and have to run at 1g.

I also want to setup several VLANs and other stuff to play with and learn L3 connectivity, which is why I was thinking those two switches if I can get them to talk to each other, as a lot of the VLANs would be for an ESXi cluster, and my NAS.

Any help/suggestions/warnings would awesome!

Thanks!
 

Rand__

Well-Known Member
Mar 6, 2014
6,332
1,599
113
SX6036 can run IB and ETH both on any port if you have the correct license (ETH) and it also can act as bridge between the two (not sure if it is a separate license). The license key can be... computed, check the various sx6036 threads for a how to (all other licenses are available as well).

I am not sure if you need to run IPoIB on the IB side to bridge the two networks, but I guess so.
Unless you need more than the 36 40/56G ports there should be no need to run a second switch except for 1g/10g RJ45, as you also split some of the 6036's ports into 4 SFP+ ports (with the correct cable, not all ports).

Regarding cards - most of the CX3 QSFP+ ones can be converted into each other by flashing the appropriate firmware to it - for this there are a lot of threads here as well.

Lastly, the 6036 can do VLANs and most of what you would need.

Good luck
 

sbonikowske

New Member
Aug 1, 2022
6
0
1
Thank you so much for the info!! That helps a ton.

Unfortunately I do need some 1g rj45 connections so I am kind of stuck with another switch. I am looking for something that would convert the sfp+ male connection to a copper rj45, but haven't found that yet. If I can do that, then there's no need for another switch.
 

Rand__

Well-Known Member
Mar 6, 2014
6,332
1,599
113
Unfortunately I do need some 1g rj45 connections so I am kind of stuck with another switch. I am looking for something that would convert the sfp+ male connection to a copper rj45, but haven't found that yet. If I can do that, then there's no need for another switch.
If it's not many you can do qsfp->sfp->rj45, but if its more than 2 or 3 that's not worth the effort and cost involved.
I run the qsfp switches with icx7450's so I have a 40g interconnect but of course plenty of options for that and totally dependent on requirements.
Previously ran a 4:1 split and also qsfp->sfp dac to a 10g fibre capable switch, was fine too
 
  • Like
Reactions: sbonikowske

sbonikowske

New Member
Aug 1, 2022
6
0
1
Thank you for the article, It will be helpful in the long term. In the short term though, if I connect a breakout cable to the 6036 so far all I see are breakout cables that are qsfp+ to sfp+ male to male. I need something on the sfp+ side that I can convert the male to a female rj45 connection so I can plug normal 1g copper into the ends. Does that make sense?

Thanks!
 

Rand__

Well-Known Member
Mar 6, 2014
6,332
1,599
113
There is no such thing. Only option is to spend a whole port to use a qsfp->sfp adapter (MAM1Q00A-QSA) and then use a male sfp+ to female rj45 adapter
 

sbonikowske

New Member
Aug 1, 2022
6
0
1
that explains why I am having a hard time finding them!

Again, thank you all for the quick answers!
 

necr

Active Member
Dec 27, 2017
136
40
28
122
Beware of 100Mbit/s appliances (some IPMIs and older media devices), those won't link at 1Gbit/s, you'll need to connect them to some different switch.
 
  • Like
Reactions: sbonikowske