Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)
- Thread starter fohdeesha
- Start date
Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
yeah, this thread really impacted ebay prices. Before I created this thread I bought 3 6450's, 2 non-poe and 1 PoE. Didn't pay more than 90 dollars for any of them.
In the future, if I ever found another neat switch that fits STH needs, I would be sure to buy up all of them before mentioning it, cough https://i.imgur.com/RkqI0UR.jpg
In the future, if I ever found another neat switch that fits STH needs, I would be sure to buy up all of them before mentioning it, cough https://i.imgur.com/RkqI0UR.jpg
To be fair you did find the unicorn switch (two really) in a sea of ludicrously expensive enterprise licensed kit. We'd have a lot more options if the other $200-500 switches didn't also require 5-10x their purchase cost in fees to actually use them.
And...God, I still want to buy a 6610 even though I don't need it yet and I'm seriously tempted to pick up another 6450 just to have more 10GbE ports. Look at what you've done :-/
Question: can a pair of 6450s do everything I'd need to use RoCE v1 over a relatively simple (not routed, purely switched vlan) topography? I'm setting up a proxmox cluster and I'm thinking about replicated storage (hopefully over RDMA to keep overhead down, thin clients aren't beef kings.) I can't really afford a dedicated storage fabric (nor do I actually need anything like 40Gb/s bandwidth for it with 1-2 SSDs per node) so I was thinking I might fake it with RoCE and vlans if I can.
RoCE "technically" requires switches to support PFC (priority flow control) to run over them, and the ICX line does not support pfc as they are the "campus" product line. For RoCE feature support you'd need the data center line, which is the VDX series (which I really don't like).
PFC support on the switch ensures the Ethernet transport layer is lossless - eg no dropped frames, which is how RoCE guaruntees it's speed/very low latency. That said, RoCE has its own error correction and retransmission faculties, it will run just fine over a "lossy" network (eg a switch without PFC support), it won't corrupt your data or anything malicious like that. It will just impact performance.
On a non-oversubbed switch with only a few clients, how often will you get dropped frames and therefore how much will it kill RoCE performance? I haven't the slightest clue, I've been wondering that myself. I've seen tests showing plain RoCE (eg the type intended for lossless networks only) working plenty fast without PFC in smaller networks, but I've also heard from engineers "don't ever do that, it'll be slower than non-rdma protocols" so I really have no clue.
Maybe someone here with an existing pfc-enabled switch and RoCE clients and some spare time can run a quick benchmark, then disable PFC on the switch and run it again. I honestly can't even begin to guess the performance impact, I would assume the biggest impact would be to latency, and you might get dropped frames that need to be recovered (costly time wise) just from one nic outpacing the switch or vice versa. What I do know is many people will tell you "just don't do it without PFC support"
PFC support on the switch ensures the Ethernet transport layer is lossless - eg no dropped frames, which is how RoCE guaruntees it's speed/very low latency. That said, RoCE has its own error correction and retransmission faculties, it will run just fine over a "lossy" network (eg a switch without PFC support), it won't corrupt your data or anything malicious like that. It will just impact performance.
On a non-oversubbed switch with only a few clients, how often will you get dropped frames and therefore how much will it kill RoCE performance? I haven't the slightest clue, I've been wondering that myself. I've seen tests showing plain RoCE (eg the type intended for lossless networks only) working plenty fast without PFC in smaller networks, but I've also heard from engineers "don't ever do that, it'll be slower than non-rdma protocols" so I really have no clue.
Maybe someone here with an existing pfc-enabled switch and RoCE clients and some spare time can run a quick benchmark, then disable PFC on the switch and run it again. I honestly can't even begin to guess the performance impact, I would assume the biggest impact would be to latency, and you might get dropped frames that need to be recovered (costly time wise) just from one nic outpacing the switch or vice versa. What I do know is many people will tell you "just don't do it without PFC support"
Last edited:
I purchased these Dell 40G QSFP breakout cables and now my 10G ports on the back of my ICX6610 is now working with stacking enable. Finally some success.
Now I can start working on my Sonos issue..
Last edited:
Guys, what's your take on Brocade BR-VDX6710 switches? I know they are not FastIron, any ideas on firmware access etc?
not good. They draw more power than the ICX6610 with WAY less 10gbe ports, they run NOS which is nothing like fastiron (I do not like it), they're now owned by Extreme which has no public firmware updates, the VDX series has almost no L3 features, they are extremely vendor locked (non-brocade optics will not even link up).
The VDX product philosophy has always been the total opposite of the fastiron stuff. Fastiron heritage is a lot of engineers from Foundry etc, while VDX has always been a totally separate group that shares more heritage with Brocade's fibrechannel product lines than it does with Foundry's ethernet stuff
any reason why you don't just pick up an icx6610?
The VDX product philosophy has always been the total opposite of the fastiron stuff. Fastiron heritage is a lot of engineers from Foundry etc, while VDX has always been a totally separate group that shares more heritage with Brocade's fibrechannel product lines than it does with Foundry's ethernet stuff
any reason why you don't just pick up an icx6610?
Thanks, good to know. I was looking for something with smaller power footprint, and per specs VDX uses <130W vs 185W on icx6610-48 for example. But the rest of those issues like access to fw make it a non-starter.
as per my original post in this thread, the ICX6610 draws 100w - 110w depending on 24/48 port version. It will never get close to 185 watts
I'm using the VDX switches for work and so far most of what fohdeesha says I would second. So far, I like NOS better than FastIron, but it's a huge PITA to get firmware updates, even within the same release. I was trying to set up a VCS fabric (the VDX's fancy stack) and got an extremely cryptic error because the versions didn't exactly match. You have to request every single firmware version and it's a manual process to be approved. Their support engineer worked around that to get me the firmware version I needed.
That said, the VDX line targets a different market since it's 48 10 GbE ports. And you need licenses to unlock all of those ports, plus more licenses for the 40 GbE ports. It doesn't make sense to get that for a homelab.
That said, the VDX line targets a different market since it's 48 10 GbE ports. And you need licenses to unlock all of those ports, plus more licenses for the 40 GbE ports. It doesn't make sense to get that for a homelab.
Huh, I suppose I'll have to try it and see how it performs then. It would be nice to have some numbers from someone else with a supported setup to compare to though, that's for sure.
Ironically my cheapest option for supported RDMA at this point is to just run an Infiniband network alongside my existing Ethernet network. The cost of CX-3 Pro cards alone would set me back more than a couple of QDR IB switches and 30m of MTP/MPO.
I have the VDX 6720-60, I works on every optic transceiver that I have tested. Twinax cables are a different story though. The thing is a beast, no need for homelab. I ordered one on Ebay two years ago for a client that pulled his order the last minute, been struggling to sell it afterwards. Most homelabs here locally in Norway is still into 1Gbps Base-T 
I wanted to organize a list of items to buy for the ICX6450/6610. There's a lot of good information in this thread but they are scattered through out. So I am compiling this list for convenience and before the next eBay promo comes up. Most are copy and paste from previous posts. Let me know if I get something wrong or want something added.
Switch Deals:
Adapter Cards:
The ConnectX-3 is a newer card revision and supported under current versions of Mellanox's WinOF drivers. It's also a PCIe 3.0 x4 card, versus the ConnectX-2's PCIe 2.0 x8. Officially the CX3s support SR-IOV and RDMA/RoCE v1 while the CX2 supports neither.
Miscellaneous:
Breakout Port Options by fohdeesha:
2M Cisco QSFP-4SFP10G-CU2M Compatible QSFP+ to 4SFP+ Breakout Cable (Monoprice cable)
Alternatively (or something similiar):
Dell 40G QSFP+ to 4x10G SFP+ 3M DAC Splitter Cable 27GG5
QSFP+ to SFP+ adapters so you can use SFP+:
GENUINE 655902-001 655874-B21 HP MELLANOX QSFP/SFP ADAPTER
brocade sfp+ 10g SR transceivers:
Brocade 10GB 57-0000075-01 10G-SFPP-SR 10Gbase-sr/sw 850nm SFP+ Transceiver >500
Cheap Cabling:
NetApp X6558-R6 External SAS Cable QSFP-QSFP 112-00177 2 Meter
NetApp X6559-R6 SAS expansion Cable 5m 5 meter QSFP-QSFP 112-00178 15ft
Inexpensive cable to connectx-2:
MCP2104-X001B Mellanox 1M 10G SFP+ Passive Direct Attach Copper Twinax Cable
USB Console Cable (any standard cisco rj45 serial pinout will work):
Gopala 6 FT(1.8M) Cisco Console Cable with FTDI Chip, Usb to Rj45 cable
Tall Bracket for Mellanox MCX354A/MCX314A:
Mellanox MTM000574 FRU Tall Bracket for 2-Port QSFP Adapter with Gasket
Rackmount Kit:
Brocade Ruckus ICX6450-RMK 2 Post Rackmount Kit + Full Accessory Kit (listing ended)
Switch Deals:
- Brocade ICX6450: 4x 10gbE SFP+, low power 22W,
- Brocade ICX6610: 16x 10gbE SFP+, 2x 40gbE, 100w power draw
Adapter Cards:
- InfiniBand and VPI Adapter Cards:
- Ethernet Adapter Cards:
- Mellanox MCX311A-XCAT CX311A ConnectX-3 Network 10GbE SFP+ Both Low+Full Profile (single port version)
- CX312A MELLANOX CONNECTX-3 DUAL PORT GIGABIT ETHERNET CARD MCX312A-XCBT (dual port version)
The ConnectX-3 is a newer card revision and supported under current versions of Mellanox's WinOF drivers. It's also a PCIe 3.0 x4 card, versus the ConnectX-2's PCIe 2.0 x8. Officially the CX3s support SR-IOV and RDMA/RoCE v1 while the CX2 supports neither.
Miscellaneous:
Breakout Port Options by fohdeesha:
Members are saying to avoid the:
2M Cisco QSFP-4SFP10G-CU2M Compatible QSFP+ to 4SFP+ Breakout Cable (Monoprice cable)
Alternatively (or something similiar):
Dell 40G QSFP+ to 4x10G SFP+ 3M DAC Splitter Cable 27GG5
QSFP+ to SFP+ adapters so you can use SFP+:
GENUINE 655902-001 655874-B21 HP MELLANOX QSFP/SFP ADAPTER
brocade sfp+ 10g SR transceivers:
Brocade 10GB 57-0000075-01 10G-SFPP-SR 10Gbase-sr/sw 850nm SFP+ Transceiver >500
Cheap Cabling:
NetApp X6558-R6 External SAS Cable QSFP-QSFP 112-00177 2 Meter
NetApp X6559-R6 SAS expansion Cable 5m 5 meter QSFP-QSFP 112-00178 15ft
Inexpensive cable to connectx-2:
MCP2104-X001B Mellanox 1M 10G SFP+ Passive Direct Attach Copper Twinax Cable
USB Console Cable (any standard cisco rj45 serial pinout will work):
Gopala 6 FT(1.8M) Cisco Console Cable with FTDI Chip, Usb to Rj45 cable
Tall Bracket for Mellanox MCX354A/MCX314A:
Mellanox MTM000574 FRU Tall Bracket for 2-Port QSFP Adapter with Gasket
Rackmount Kit:
Brocade Ruckus ICX6450-RMK 2 Post Rackmount Kit + Full Accessory Kit (listing ended)
Last edited:
If anyone wants to play the FDR game and direct link two machines at 56Gb this is the cheapest DAC I could find on eBay:
IBM 1m Mellanox QSFP Passive Copper FDR14 InfiniBand Cable 00W0049 Server ZZ 883436241496 | eBay
IBM 1m Mellanox QSFP Passive Copper FDR14 InfiniBand Cable 00W0049 Server ZZ 883436241496 | eBay
Could anyone with a 6450-48 measure the actual power consumption with ~10 ports connected?
I currently use a T1700G-28TQ which uses <20W with that many connections. I want to try the routing features on the 6450, but I can't find a reasonable deal on the 24 port version here in the UK.
I will get the -48 if it uses less power than the 30-51W in the datasheet with fewer ports connected.
I currently use a T1700G-28TQ which uses <20W with that many connections. I want to try the routing features on the 6450, but I can't find a reasonable deal on the 24 port version here in the UK.
I will get the -48 if it uses less power than the 30-51W in the datasheet with fewer ports connected.
Mine is 6450-24P simply as FYI: 2 10GB ports being used, 12 1GB ports used, no PoE; Kill-a-watt was reporting 30-32 watts.
This thread has been amazing to read through. Convinced me to upgrade my homelab TP-Link switches to something more meaningful and really learn some new tricks. Two ICX6610-24p switches are coming my way. Can't wait to get them setup!
the P version power draw with no PoE devices is identical to the non-P version, the 54v rail doesn't even turn on. With PoE devices, just add 2 or 3 watts per PoE device
Off Topic: the 19" Rack PDU's I see used on ebay for about $100; which offer watt metering via a LAN browser, do they only meter the wattage going through the entire PDU - or are there any PDU which can report wattage of each outlet independently ?