Drag to reposition cover

Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

nezach

Active Member
Oct 14, 2012
210
119
43
Thanks for sharing guys, this gives me an idea on what a good deal is. I will try to score one for <$180, hopefully in the next couple weeks
 

arglebargle

H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈
Jul 15, 2018
657
243
43
yeah, this thread really impacted ebay prices. Before I created this thread I bought 3 6450's, 2 non-poe and 1 PoE. Didn't pay more than 90 dollars for any of them.

In the future, if I ever found another neat switch that fits STH needs, I would be sure to buy up all of them before mentioning it, cough https://i.imgur.com/RkqI0UR.jpg
To be fair you did find the unicorn switch (two really) in a sea of ludicrously expensive enterprise licensed kit. We'd have a lot more options if the other $200-500 switches didn't also require 5-10x their purchase cost in fees to actually use them.

And...God, I still want to buy a 6610 even though I don't need it yet and I'm seriously tempted to pick up another 6450 just to have more 10GbE ports. Look at what you've done :-/

Question: can a pair of 6450s do everything I'd need to use RoCE v1 over a relatively simple (not routed, purely switched vlan) topography? I'm setting up a proxmox cluster and I'm thinking about replicated storage (hopefully over RDMA to keep overhead down, thin clients aren't beef kings.) I can't really afford a dedicated storage fabric (nor do I actually need anything like 40Gb/s bandwidth for it with 1-2 SSDs per node) so I was thinking I might fake it with RoCE and vlans if I can.
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,587
2,776
113
31
fohdeesha.com
RoCE "technically" requires switches to support PFC (priority flow control) to run over them, and the ICX line does not support pfc as they are the "campus" product line. For RoCE feature support you'd need the data center line, which is the VDX series (which I really don't like).

PFC support on the switch ensures the Ethernet transport layer is lossless - eg no dropped frames, which is how RoCE guaruntees it's speed/very low latency. That said, RoCE has its own error correction and retransmission faculties, it will run just fine over a "lossy" network (eg a switch without PFC support), it won't corrupt your data or anything malicious like that. It will just impact performance.

On a non-oversubbed switch with only a few clients, how often will you get dropped frames and therefore how much will it kill RoCE performance? I haven't the slightest clue, I've been wondering that myself. I've seen tests showing plain RoCE (eg the type intended for lossless networks only) working plenty fast without PFC in smaller networks, but I've also heard from engineers "don't ever do that, it'll be slower than non-rdma protocols" so I really have no clue.

Maybe someone here with an existing pfc-enabled switch and RoCE clients and some spare time can run a quick benchmark, then disable PFC on the switch and run it again. I honestly can't even begin to guess the performance impact, I would assume the biggest impact would be to latency, and you might get dropped frames that need to be recovered (costly time wise) just from one nic outpacing the switch or vice versa. What I do know is many people will tell you "just don't do it without PFC support"
 
Last edited:

starheaven

New Member
Aug 21, 2018
26
14
3
Guys, what's your take on Brocade BR-VDX6710 switches? I know they are not FastIron, any ideas on firmware access etc?
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,587
2,776
113
31
fohdeesha.com
not good. They draw more power than the ICX6610 with WAY less 10gbe ports, they run NOS which is nothing like fastiron (I do not like it), they're now owned by Extreme which has no public firmware updates, the VDX series has almost no L3 features, they are extremely vendor locked (non-brocade optics will not even link up).

The VDX product philosophy has always been the total opposite of the fastiron stuff. Fastiron heritage is a lot of engineers from Foundry etc, while VDX has always been a totally separate group that shares more heritage with Brocade's fibrechannel product lines than it does with Foundry's ethernet stuff

any reason why you don't just pick up an icx6610?
 

starheaven

New Member
Aug 21, 2018
26
14
3
not good. They draw more power than the ICX6610 with WAY less 10gbe ports, they run NOS which is nothing like fastiron (I do not like it), they're now owned by Extreme which has no public firmware updates, the VDX series has almost no L3 features, they are extremely vendor locked (non-brocade optics will not even link up).

The VDX product philosophy has always been the total opposite of the fastiron stuff. Fastiron heritage is a lot of engineers from Foundry etc, while VDX has always been a totally separate group that shares more heritage with Brocade's fibrechannel product lines than it does with Foundry's ethernet stuff

any reason why you don't just pick up an icx6610?
Thanks, good to know. I was looking for something with smaller power footprint, and per specs VDX uses <130W vs 185W on icx6610-48 for example. But the rest of those issues like access to fw make it a non-starter.
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,587
2,776
113
31
fohdeesha.com
as per my original post in this thread, the ICX6610 draws 100w - 110w depending on 24/48 port version. It will never get close to 185 watts
 

sean

Member
Sep 26, 2013
66
33
18
CT
I'm using the VDX switches for work and so far most of what fohdeesha says I would second. So far, I like NOS better than FastIron, but it's a huge PITA to get firmware updates, even within the same release. I was trying to set up a VCS fabric (the VDX's fancy stack) and got an extremely cryptic error because the versions didn't exactly match. You have to request every single firmware version and it's a manual process to be approved. Their support engineer worked around that to get me the firmware version I needed.

That said, the VDX line targets a different market since it's 48 10 GbE ports. And you need licenses to unlock all of those ports, plus more licenses for the 40 GbE ports. It doesn't make sense to get that for a homelab.
 
  • Like
Reactions: fohdeesha

arglebargle

H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈
Jul 15, 2018
657
243
43
RoCE "technically" requires switches to support PFC (priority flow control) to run over them, and the ICX line does not support pfc as they are the "campus" product line. For RoCE feature support you'd need the data center line, which is the VDX series (which I really don't like).

PFC support on the switch ensures the Ethernet transport layer is lossless - eg no dropped frames, which is how RoCE guaruntees it's speed/very low latency. That said, RoCE has its own error correction and retransmission faculties, it will run just fine over a "lossy" network (eg a switch without PFC support), it won't corrupt your data or anything malicious like that. It will just impact performance.

On a non-oversubbed switch with only a few clients, how often will you get dropped frames and therefore how much will it kill RoCE performance? I haven't the slightest clue, I've been wondering that myself. I've seen tests showing plain RoCE (eg the type intended for lossless networks only) working plenty fast without PFC in smaller networks, but I've also heard from engineers "don't ever do that, it'll be slower than non-rdma protocols" so I really have no clue.

Maybe someone here with an existing pfc-enabled switch and RoCE clients and some spare time can run a quick benchmark, then disable PFC on the switch and run it again. I honestly can't even begin to guess the performance impact, I would assume the biggest impact would be to latency, and you might get dropped frames that need to be recovered (costly time wise) just from one nic outpacing the switch or vice versa. What I do know is many people will tell you "just don't do it without PFC support"
Huh, I suppose I'll have to try it and see how it performs then. It would be nice to have some numbers from someone else with a supported setup to compare to though, that's for sure.

Ironically my cheapest option for supported RDMA at this point is to just run an Infiniband network alongside my existing Ethernet network. The cost of CX-3 Pro cards alone would set me back more than a couple of QDR IB switches and 30m of MTP/MPO.
 
  • Like
Reactions: fohdeesha

BackupProphet

Well-Known Member
Jul 2, 2014
881
356
63
Stavanger, Norway
kingmakers.no
I have the VDX 6720-60, I works on every optic transceiver that I have tested. Twinax cables are a different story though. The thing is a beast, no need for homelab. I ordered one on Ebay two years ago for a client that pulled his order the last minute, been struggling to sell it afterwards. Most homelabs here locally in Norway is still into 1Gbps Base-T :(
 

Thomas H

Member
Dec 2, 2017
63
25
8
51
I wanted to organize a list of items to buy for the ICX6450/6610. There's a lot of good information in this thread but they are scattered through out. So I am compiling this list for convenience and before the next eBay promo comes up. Most are copy and paste from previous posts. Let me know if I get something wrong or want something added.

Switch Deals:
  • Brocade ICX6450: 4x 10gbE SFP+, low power 22W,
  • Brocade ICX6610: 16x 10gbE SFP+, 2x 40gbE, 100w power draw
Both switches offer 24/48 ports and PoE available.

Adapter Cards:
HP 649281-B21 are OEM rebadged MCX354A-FCBT. You can save a lot of money (~$100) by buying the (HP 649281-B21) QCBT and flashing it with the FCBT firmware. Mellanox Connectx3 series is much cheaper and has better compatibility (in regards to optics/transceivers etc) and is much cheaper versus the intels.

The ConnectX-3 is a newer card revision and supported under current versions of Mellanox's WinOF drivers. It's also a PCIe 3.0 x4 card, versus the ConnectX-2's PCIe 2.0 x8. Officially the CX3s support SR-IOV and RDMA/RoCE v1 while the CX2 supports neither.

Miscellaneous:

Breakout Port Options by fohdeesha:
To break out those 2x breakout ports, you have many options:

** Avoid** You can use a DAC breakout cable: 2M Cisco QSFP-4SFP10G-CU2M Compatible QSFP+ to 4SFP+ Breakout Cable 889028000663 | eBay

Or you can use a 40gbE optic to give you an MPO fiber connector: Brocade XBR-000232 QSFP 57-1000267-01 4*16Gb SWL 100m for DCX8510 EMC | eBay

And then connect an MPO to LC breakout cable to that optic: MPO Fan-out Patch Cable, Multimode Fiber, MPO to LC Uniboot, 3-meter 10FT 603149398665 | eBay

Or you can get fancy, and using an optic like above, use MPO fiber like this: 5m 50/125 Standard MTP Multimode Fiber Optic Patch Cable Key-up to Key-down | eBay

To go to an MPO breakout panel like this: Systimax SCS InstaPATCH 24-Port Fiber Patch Panel Module 642337970740 | eBay
Members are saying to avoid the:
2M Cisco QSFP-4SFP10G-CU2M Compatible QSFP+ to 4SFP+ Breakout Cable (Monoprice cable)

Alternatively (or something similiar):
Dell 40G QSFP+ to 4x10G SFP+ 3M DAC Splitter Cable 27GG5

QSFP+ to SFP+ adapters so you can use SFP+:
GENUINE 655902-001 655874-B21 HP MELLANOX QSFP/SFP ADAPTER

brocade sfp+ 10g SR transceivers:
Brocade 10GB 57-0000075-01 10G-SFPP-SR 10Gbase-sr/sw 850nm SFP+ Transceiver >500

Cheap Cabling:
NetApp X6558-R6 External SAS Cable QSFP-QSFP 112-00177 2 Meter
NetApp X6559-R6 SAS expansion Cable 5m 5 meter QSFP-QSFP 112-00178 15ft

Inexpensive cable to connectx-2:
MCP2104-X001B Mellanox 1M 10G SFP+ Passive Direct Attach Copper Twinax Cable

USB Console Cable (any standard cisco rj45 serial pinout will work):
Gopala 6 FT(1.8M) Cisco Console Cable with FTDI Chip, Usb to Rj45 cable

Tall Bracket for Mellanox MCX354A/MCX314A:
Mellanox MTM000574 FRU Tall Bracket for 2-Port QSFP Adapter with Gasket

Rackmount Kit:
Brocade Ruckus ICX6450-RMK 2 Post Rackmount Kit + Full Accessory Kit (listing ended)
 
Last edited:

billc.cn

Member
Oct 6, 2017
47
9
8
Could anyone with a 6450-48 measure the actual power consumption with ~10 ports connected?

I currently use a T1700G-28TQ which uses <20W with that many connections. I want to try the routing features on the 6450, but I can't find a reasonable deal on the 24 port version here in the UK.

I will get the -48 if it uses less power than the 30-51W in the datasheet with fewer ports connected.
 

tommybackeast

Active Member
Jun 10, 2018
290
111
43
Could anyone with a 6450-48 measure the actual power consumption with ~10 ports connected?

I currently use a T1700G-28TQ which uses <20W with that many connections. I want to try the routing features on the 6450, but I can't find a reasonable deal on the 24 port version here in the UK.

I will get the -48 if it uses less power than the 30-51W in the datasheet with fewer ports connected.
Mine is 6450-24P simply as FYI: 2 10GB ports being used, 12 1GB ports used, no PoE; Kill-a-watt was reporting 30-32 watts.
 

Gamble

New Member
Aug 20, 2018
11
5
3
This thread has been amazing to read through. Convinced me to upgrade my homelab TP-Link switches to something more meaningful and really learn some new tricks. Two ICX6610-24p switches are coming my way. Can't wait to get them setup!
 
  • Like
Reactions: ncarlson42

tommybackeast

Active Member
Jun 10, 2018
290
111
43
the P version power draw with no PoE devices is identical to the non-P version, the 54v rail doesn't even turn on. With PoE devices, just add 2 or 3 watts per PoE device
Off Topic: the 19" Rack PDU's I see used on ebay for about $100; which offer watt metering via a LAN browser, do they only meter the wattage going through the entire PDU - or are there any PDU which can report wattage of each outlet independently ?