ICX7750 vs 7050QX vs SX6036

koifish59

Member
Sep 30, 2020
66
19
8
I currently have a 4 node vSAN cluster using all 4 QSFP ports on my ICX 6650 switch and need to expand. Below are the NICs I already have, and the switch I’m deciding between. How do these 3 switches compare, and which is best suited for my upgrade?

(4x) MCX416A-BCAT (dual 56/50/40 gbe ports)
(4x) MCX4131A-GCAT (single 50/40 gbe port)
(4x) MCX4121C-ACAT (dual 25/10 gbe port)
  • Brocade ICX 7750-26Q
  • Arista 7050QX-32S
  • Mellanox SX6036

I’m leaning towards the 7050QX-32S since I will need a SFP port for the fiber WAN. Although I think the SX6036 may be the better choice since I already have all Mellanox NICs and can leverage the 56/50 gbe InfiniBand speeds. Additional questions I have:
  1. Can a QSFP switch port be used with an adapter to use a SFP optic for the 1gbps WAN fiber connection?
  2. Can the SX6036 run 56gbe on some ports and 50gbe on other ports at the same time?
  3. Based on datasheet, all 3 switches use about the same power consumption at idle or with passive cable, correct?
  4. I may need to connect a few 1gbps and 10gbps ethernet cables from various devices. Will an QSFP to SFP adapter with SFP to 10Gbase-T adapter work to have the RJ-45 ethernet plugged into the QSFP port?
 

i386

Well-Known Member
Mar 18, 2016
3,510
1,197
113
33
Germany
I had a qsfp+ to sfp+ adapter with one 1GBE base-t in use with an arista, no problems.

Personally I would prefer the arista because of the command to set the fans to 30% :D
If noise is not a concern get the cheapest one, they are all enterprise and high quality switches
Can the SX6036 run 56gbe on some ports and 50gbe on other ports at the same time?
50GBE is too new
 
  • Like
Reactions: koifish59

NablaSquaredG

Well-Known Member
Aug 17, 2020
728
341
63
If you absolutely need 56GBit, go for the Mellanox. It is a proprietary standard, so no other vendor supports it.

If you can live with 40GBe, go for the Arista or Brocade (whatever is cheaper or whatever CLI you prefer)

The PowerPC control plane of the SX6036 isn't great and I'd avoid it if you can
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,616
2,810
113
32
fohdeesha.com
worth noting the icx7750 is also PPC control plane, I would probably go with the arista just for x86 (and I favor EOS a bit, that said, if you already have one brocade model and are familiar with the CLI, there's something to be said about keeping your setups and all the CLIs you have to deal with homogenous)
 
  • Like
Reactions: koifish59 and gb00s

AndreiL

New Member
Jun 30, 2019
23
9
3
None of those 3 support 50GbE, 56GbE is Mellanox proprietary, non-standard as already mentioned.

SX6036 supports breaking out some 40G ports in 4x10G using special QSFP modules, then you can connect it to SR or LR 10G transceivers on the other end. I've done this, works as advertised. Only 4 ports support breakout into 4x10G LR (high-power ports)
The control plane on SX6036 is PPC and is not great, the SX1710 is exact same switch ASIC but with x86 control plane, it's currently much more expensive then SX6036.

Looks like you have ConnectX-4 EN NiCs, if you spring for Mellanox SN2700 you can put your vSAN on RoCEv2 transport (vmware reference implementation uses SN2700 + CNX4 EN NiCs)
 
  • Like
Reactions: koifish59

koifish59

Member
Sep 30, 2020
66
19
8
I may buy both 7050QX and SX6036 just to play around with. I'm accustomed to brocade's setup, but I'm ok with learning a new switch. Additional Q's:

1. My math may or may not be correct, but pcie 3.0 for x16 lanes should be able to support 16 GB/s (128gbps), so the dual 56 GbE ports from MCX416A-BCAT will run full speed if connected to a SX6036 switch?

2. If ethernet licensing is applied to the SX6036, it's per port so I can run both 56g and 40g on the same switch and same subnet?

3. It's also my understanding that Arista is pretty forgiving for DACs and optics from different vendors as long as the vendor unlock is applied to the config, but mellanox switch must use official mellanox FDR cables? (This might make most of my DACs/modules useless :()
 

NablaSquaredG

Well-Known Member
Aug 17, 2020
728
341
63
1. My math may or may not be correct, but pcie 3.0 for x16 lanes should be able to support 16 GB/s (128gbps), so the dual 56 GbE ports from MCX416A-BCAT will run full speed if connected to a SX6036 switch?
Yes, the available bandwidth is enough to saturate the 56G links. But bear in mind that you will have to do some serious performance tuning to actually achieve the 56G/s

2. If ethernet licensing is applied to the SX6036, it's per port so I can run both 56g and 40g on the same switch and same subnet?
Yes, that is absolutely no problem.

3. It's also my understanding that Arista is pretty forgiving for DACs and optics from different vendors as long as the vendor unlock is applied to the config, but mellanox switch must use official mellanox FDR cables? (This might make most of my DACs/modules useless :()
Yep, you will only see 56G with cables that are Mellanox compatible (i.e. has the Mellanox OUI programmed)
Depending on what DACs you have, they might already have the Mellanox OUI (fs.com generic cables have the mellanox oui)

Also depending on the quantity of DACs you have, it might be worthwile to buy a QSFP programmer and reprogram than (YMMV, not possible with every DAC...)
 

LodeRunner

Active Member
Apr 27, 2019
489
210
43
worth noting the icx7750 is also PPC control plane, I would probably go with the arista just for x86 (and I favor EOS a bit, that said, if you already have one brocade model and are familiar with the CLI, there's something to be said about keeping your setups and all the CLIs you have to deal with homogenous)
Adding to this, parts for the Arista switches are often easier to find at reasonable prices than replacements for the ICX7750 (where it'll be $150 per fan tray and $500 for a PSU sort of thing, last time I looked before getting a Arisa 7050QX-32S and being done with it). Even just buying the Arista fully equipped will be less than the 7750's I see listed.

Despite free availability of software for the 7750, I'd give it a hard pass due to availability, pricing (unit and component), and that new versions of the ICX software drop support for it, so pretty sure Ruckus wants it dead.