Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

fohdeesha

Kaini Industries
Nov 20, 2016
1,960
1,783
113
29
fohdeesha.com
I need 3-4 24 port switches with 10G uplinks. How does this compare to the Mikrotik 326-24G-2S+RM?

I was considering the Mikrotik because of price and quiet operation, but the ICX6450 are tempting. Could even get rid of some POE injectors by getting a POE version of the switch.
well, it's not particularly fair to compare them, one is an enterprise switch that's $3000 new, and one is a mikrotik device that's $130 new. The Brocade was intended for enterprise and campus environments for the most part and wear items like onboard flash are duplicated for redundancy, the power supply is much higher quality (Delta onboard instead of a wall wart), a lot more ESD input protection on the physical ports/PHY (most of the mikrotik devices I've taken apart have NONE), the software and hardware has been QA'd quite a lot more (I know the ICX engineering & QA team at brocade on its own was larger than all of Mikrotik as a company) - if you want a general idea of Mikrotik's level of QA, read through some of the mikrotik megathreads here on STH

Aside from the obvious things above like hardware, they're also in different leagues software capability wise, the mikrotik is an l2 device only (no routing) while the 6450 is a full L3 router (at millions of packets per second & hundreds of gigabits per sec, not in CPU like some mikrotik models). However even comparing them for just L2 features, the mikrotik is still missing quite a few - no deep ACL's, no sflow, no stacking support, no QinQ, no out of band management, no 802.1X auth, no per vlan spanning tree, dynamic ARP inspection, no RADIUS/TACACS/TACACS+ authentication, etc etc, and that's just the L2 differences off the top of my head

The ICX6450 also has twice the 10gbE ports, which I'd imagine is the most important difference to most people on STH. If they were wildly different prices I would say to weigh your wants and needs to decide which to get, but given they're basically the same price thanks to ebay it's pretty much a no brainer
 
Last edited:

arglebargle

H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈
Jul 15, 2018
656
233
43
I need 3-4 24 port switches with 10G uplinks. How does this compare to the Mikrotik 326-24G-2S+RM?

I was considering the Mikrotik because of price and quiet operation, but the ICX6450 are tempting. Could even get rid of some POE injectors by getting a POE version of the switch.
I researched the cheap Mikrotiks before I found @fohdeesha's thread about the icx 6xxx, the 326 was actually one of my top candidates for low cost low power 10Gb uplink before I learned about the icx 6450.

They aren't the same thing, the <$200 Mikrotik switches are what you buy to replace a $20 unmanaged switch when you want 10Gb uplink. If you need to do anything beyond basic L2 switching the Mikrotik has to offload the traffic to its CPU, which is connected to the switch by a single 1Gb link. I've seen reports of 300Mbps performance across multi gigabit LAGGs and L3 routing performance under 100Mbps.

The icx 6450-24 under my desk is basically silent after replacing the fan, you could probably quiet a -24p a bit if you wanted to but since mine is powering every PoE device on my network I just left it with the stock cooling.

If they were wildly different prices I would say to weigh your wants and needs to decide which to get, but given they're basically the same price thanks to ebay it's pretty much a no brainer
Also, FWIW, I think I trust Brocade's ability to write secure software more than Mikrotik's.
 

fohdeesha

Kaini Industries
Nov 20, 2016
1,960
1,783
113
29
fohdeesha.com
@PGlover I finally got some free time to test your DAC breakout issue - I created a lab setup identical to yours - 2x fully licensed ICX6610-48P's, both running 8030s, stacked with 2x 40gbE ports only.

Stacking config (should match yours):

Code:
stack unit 1
  module 1 icx6610-48p-poe-port-management-module
  module 2 icx6610-qsfp-10-port-160g-module
  module 3 icx6610-8-port-10g-dual-mode-module
  priority 128
  stack-port 1/2/1 1/2/6
stack unit 2
  module 1 icx6610-48p-poe-port-management-module
  module 2 icx6610-qsfp-10-port-160g-module
  module 3 icx6610-8-port-10g-dual-mode-module
  stack-port 2/2/1 2/2/6

Stacking status:
Code:
ICX3>sh stack
T=2h11m27.7: alone: standalone, D: dynamic cfg, S: static
ID   Type          Role    Mac Address    Pri State   Comment
1  S ICX6610-48P   active  cc4e.2484.f218 128 local   Ready
2  S ICX6610-48P   standby cc4e.243e.aa74   0 remote  Ready

    active       standby
     +---+        +---+
 -2/6| 1 |2/1--2/1| 2 |2/6-
 |   +---+        +---+   |
 |                        |
 |------------------------|
Standby u2 - protocols ready, can failover
Current stack management MAC is cc4e.2484.f218

I then plugged the monoprice breakout DAC into 1/2/2 (top right QSFP port). with the 4x SFP's on the other end connecting to a bunch of other switches. You can see it immediately links up all 4 channels with no issue:
Code:
ICX3>sh int br e 1/2/2 to 1/2/6

Port       Link    State   Dupl Speed Trunk Tag Pvid Pri MAC             Name
1/2/2      Up      Forward Full 10G   None  No  1    0   cc4e.2484.f218
1/2/3      Up      Forward Full 10G   None  No  1    0   cc4e.2484.f218
1/2/4      Up      Forward Full 10G   None  No  1    0   cc4e.2484.f218
1/2/5      Up      Forward Full 10G   None  No  1    0   cc4e.2484.f218
I proceeded to try the monoprice breakout DAC in all 4 breakout ports (the 2 on stack member 1, and the 2 on stack member 2). They all linked up without an issue. To make sure it wasn't some weird race condition or timeout issue, I rapidly plugged and unplugged the 40gbE stacking connections while plugging and unplugging the breakout DAC - no matter what I did or what combination/order I plugged the rear ports in, it always worked. This leads me to believe you have a bad breakout cable or something else
 
  • Like
Reactions: tommybackeast

PGlover

Active Member
Nov 8, 2014
470
55
28
54
@PGlover I finally got some free time to test your DAC breakout issue - I created a lab setup identical to yours - 2x fully licensed ICX6610-48P's, both running 8030s, stacked with 2x 40gbE ports only.

Stacking config (should match yours):

Code:
stack unit 1
  module 1 icx6610-48p-poe-port-management-module
  module 2 icx6610-qsfp-10-port-160g-module
  module 3 icx6610-8-port-10g-dual-mode-module
  priority 128
  stack-port 1/2/1 1/2/6
stack unit 2
  module 1 icx6610-48p-poe-port-management-module
  module 2 icx6610-qsfp-10-port-160g-module
  module 3 icx6610-8-port-10g-dual-mode-module
  stack-port 2/2/1 2/2/6

Stacking status:
Code:
ICX3>sh stack
T=2h11m27.7: alone: standalone, D: dynamic cfg, S: static
ID   Type          Role    Mac Address    Pri State   Comment
1  S ICX6610-48P   active  cc4e.2484.f218 128 local   Ready
2  S ICX6610-48P   standby cc4e.243e.aa74   0 remote  Ready

    active       standby
     +---+        +---+
 -2/6| 1 |2/1--2/1| 2 |2/6-
 |   +---+        +---+   |
 |                        |
 |------------------------|
Standby u2 - protocols ready, can failover
Current stack management MAC is cc4e.2484.f218

I then plugged the monoprice breakout DAC into 1/2/2 (top right QSFP port). with the 4x SFP's on the other end connecting to a bunch of other switches. You can see it immediately links up all 4 channels with no issue:
Code:
ICX3>sh int br e 1/2/2 to 1/2/6

Port       Link    State   Dupl Speed Trunk Tag Pvid Pri MAC             Name
1/2/2      Up      Forward Full 10G   None  No  1    0   cc4e.2484.f218
1/2/3      Up      Forward Full 10G   None  No  1    0   cc4e.2484.f218
1/2/4      Up      Forward Full 10G   None  No  1    0   cc4e.2484.f218
1/2/5      Up      Forward Full 10G   None  No  1    0   cc4e.2484.f218
I proceeded to try the monoprice breakout DAC in all 4 breakout ports (the 2 on stack member 1, and the 2 on stack member 2). They all linked up without an issue. To make sure it wasn't some weird race condition or timeout issue, I rapidly plugged and unplugged the 40gbE stacking connections while plugging and unplugging the breakout DAC - no matter what I did or what combination/order I plugged the rear ports in, it always worked. This leads me to believe you have a bad breakout cable or something else
I finally was able to get the stuck breakout cable out of the QSFP slot and returned both cables to the Ebay seller.

I decided to get the following items based on your recommendation. I will test next week to see if I have better results than the Monoprice cable.

Brocade XBR-000232 QSFP 57-1000267-01 4*16Gb SWL 100m for DCX8510 EMC | eBay

MPO Fan-out Patch Cable, Multimode Fiber, MPO to LC Uniboot, 3-meter 10FT 603149398665 | eBay

Brocade 10GB 57-0000075-01 10G-SFPP-SR 10Gbase-sr/sw 850nm SFP+ Transceiver >500 8904725111763 | eBay
 
Last edited:
  • Like
Reactions: fohdeesha

aliasxneo

New Member
Aug 4, 2018
8
1
3
Awesome write up! After reading through the thread, I wasn't able to find any additional details regarding the mention of a needed license on the first page. Are these easy to find already licensed on Ebay? Is there a trick to determine if a unit being sold is properly licensed or not?
 

explosivelobster

New Member
Jul 28, 2018
24
2
3
I was also wondering this, so thank you for asking - it made me go find a spare cable and test.
For the moment I just tried to loop the two ports of the only 40 GbE device I currently have (Mellanox ConnectX-3 card) with NetApp QSFP-QSFP 2m/6ft SAS cable (P/N: 112-00177) and at least the physical link layer seems to be up:
Code:
4: enp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq portid 24be05ffffb24101 state UP group default qlen 1000
    link/ether 24:be:05:b2:41:01 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::26be:5ff:feb2:4101/64 scope link
       valid_lft forever preferred_lft forever
5: enp2s0d1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq portid 24be05ffffb24102 state UP group default qlen 1000
    link/ether 24:be:05:b2:41:02 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::26be:5ff:feb2:4102/64 scope link
       valid_lft forever preferred_lft forever

# ethtool enp2s0
Settings for enp2s0:
    Supported ports: [ FIBRE ]
    Supported link modes:   1000baseKX/Full
                            10000baseKX4/Full
                            10000baseKR/Full
                            40000baseCR4/Full
                            40000baseSR4/Full
                            56000baseCR4/Full
                            56000baseSR4/Full
    Supported pause frame use: Symmetric Receive-only
    Supports auto-negotiation: Yes
    Advertised link modes:  1000baseKX/Full
                            10000baseKX4/Full
                            10000baseKR/Full
                            40000baseCR4/Full
                            40000baseSR4/Full
    Advertised pause frame use: Symmetric
    Advertised auto-negotiation: Yes
    Link partner advertised link modes:  40000baseCR4/Full
    Link partner advertised pause frame use: No
    Link partner advertised auto-negotiation: Yes
    Speed: 40000Mb/s
    Duplex: Full
    Port: Direct Attach Copper
    PHYAD: 0
    Transceiver: internal
    Auto-negotiation: on
    Supports Wake-on: d
    Wake-on: d
    Current message level: 0x00000014 (20)
                   link ifdown
    Link detected: yes
Once I receive my switches will try the full setup (as in passing traffic) and will confirm if it is working.
I'd be interested in hearing how you get on, I'm trying a similar setup (Dual CX353A-FCBT card connected via passive QSFP+ to ICX 6610 40gbps ports) and when I link the cards together they show a link LED, but not when I connect them to the ICX and nor do I see the links up in the ICX command line. I am new to all these bits of hardware though so it's most likely a misconfiguration on something!

Test machine is running Windows Server 2012, I've installed the Mellanox MFT but nothing else. Cards report as IBM1100110019 with F/W 2.36.5000.

Here's what the switch shows:

Code:
SSH@ICX6610-24-Router>show media | include 1/2/[1,6]
Port 1/2/1:  Type : 40GE-Passive Copper
Port 1/2/6:  Type : 40GE QSFP(Copper-Pigtail)
Port 1/2/10:  Type : EMPTY

SSH@ICX6610-24-Router>sh int b e 1/2/1 to 1/2/6
Port       Link    State   Dupl Speed Trunk Tag Pvid Pri MAC             Name
1/2/1      Down    None    None None  None  No  1    0   cc4e.2439.c5c4
1/2/2      Down    None    None None  None  No  1    0   cc4e.2439.c5c4
1/2/3      Down    None    None None  None  No  1    0   cc4e.2439.c5c4
1/2/4      Down    None    None None  None  No  1    0   cc4e.2439.c5c4
1/2/5      Down    None    None None  None  No  1    0   cc4e.2439.c5c4
1/2/6      Down    None    None None  None  No  1    0   cc4e.2439.c5c4
 

arglebargle

H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈
Jul 15, 2018
656
233
43
I'd be interested in hearing how you get on, I'm trying a similar setup (Dual CX353A-FCBT card connected via passive QSFP+ to ICX 6610 40gbps ports) and when I link the cards together they show a link LED, but not when I connect them to the ICX and nor do I see the links up in the ICX command line. I am new to all these bits of hardware though so it's most likely a misconfiguration on something!

Test machine is running Windows Server 2012, I've installed the Mellanox MFT but nothing else. Cards report as IBM1100110019 with F/W 2.36.5000.
What type of device is the C-X3 showing as under network adapters in device manager? My cards defaulted to IPoIB on both ports. I spent about 5 hours trying to figure out how to switch to Ethernet before I learned that the card presents another device under "System Devices" that allows you to change which protocol each port uses. Make sure your port is set to Ethernet there and that should do it.

I still can't get my desktop to POST with a CX-3 installed so I can't test 40GbE speeds on my QSFP DACs. I can tell you that they work just fine for 10GbE and refuse to link when directly connected between two C-X cards in Infiniband mode.

Edit: Have you installed Mellanox OFED?
 
Last edited:

arglebargle

H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈
Jul 15, 2018
656
233
43
You need to set up ip addresses manualy and a subnet manager running for a link in infinibnad mode.
Yeah, I've done that. I'm not even getting physical link lights between the Infiniband ports and running opensm on the Centos side never gets past "Physical state: Polling" reported by ibstat. I'm using $8 NetApp SAS cables though, so I kinda suspected that might be the issue.
 
Last edited:

explosivelobster

New Member
Jul 28, 2018
24
2
3
What type of device is the C-X3 showing as under network adapters in device manager? My cards defaulted to IPoIB on both ports. I spent about 5 hours trying to figure out how to switch to Ethernet before I learned that the card presents another device under "System Devices" that allows you to change which protocol each port uses. Make sure your port is set to Ethernet there and that should do it.
Thanks a lot! - installed OFED and forced the ports to Ethernet and the link came up. Well interestingly, only on one of the 2 types of cable I have to test.

The cable that provided a link was an SGI 018-1422-001.
The cable that didn't seem to link properly was a QLOGIC CBL1-0600230.

Here's what the switch sees now (on the SGI cables):
Code:
SSH@ICX6610-24-Router>sh int b e 1/2/1

Port       Link    State   Dupl Speed Trunk Tag Pvid Pri MAC             Name
1/2/1      Up      Forward Full 40G   None  No  1    0   cc4e.2439.c5c4
SSH@ICX6610-24-Router>sh med e 1/2/1
Port   1/2/1:Type  : 40GBASE-Passive Copper
Vendor Name: Amphenol         Serial Num: APF13270034F1M  Revision: F
SSH@ICX6610-24-Router>
SSH@ICX6610-24-Router>sh int b e 1/2/6

Port       Link    State   Dupl Speed Trunk Tag Pvid Pri MAC             Name
1/2/6      Up      Forward Full 40G   None  No  1    0   cc4e.2439.c5c4
SSH@ICX6610-24-Router>sh med e 1/2/6
Port   1/2/6:Type  : 40GBASE-Passive Copper
Vendor Name: Amphenol         Serial Num: APF13270034F27  Revision: F
 
Last edited:

arglebargle

H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈
Jul 15, 2018
656
233
43
FYI my crossflashing post here - https://forums.servethehome.com/ind...net-dual-port-qsfp-adapter.20525/#post-198015

the second part of it has quick instructions on setting the ports to ethernet using mlxconfig, a utility included in the MFT package so you don't need to install OFED. It also sets these options in the card's NVRAM/EEPROM, so it persists no matter where you install the card
If anyone needs RDMA capable firmware for a ConnectX-2 I've been mirroring copies that I found while searching for it to a shared folder on Google Drive. I upgraded 3 of my C-X2 cards last night to play with Infiniband and RDMA but I'm having Computer Problems™ of some form or another on all of my test machines so I haven't been able to get IB working yet.

@fohdeesha Do you know if the C-X3 and C-X3 Pro ICs are actually different chips? With Mellanox differentiating SKUs in firmware the way they do I wouldn't be surprised if the cards themselves are the same. I haven't been able to find any high res pictures of both boards without heatsinks but the part numbers on the IC pictures make me suspect that they both use the same hardware:

http://www.mellanox.com/uploads/product_families/cat_83/gfx_01819.jpg
http://www.mellanox.com/uploads/product_families/cat_81/gfx_00725.jpg
 
Last edited:

scurrier

New Member
Aug 4, 2018
18
23
3
Illinois, USA
I'm a long-time lurker and wanted to jump out of the bushes to say thanks to @fohdeesha for sharing this with the community!

I do have a question on the ICX 6450 POE models. If I don't have a lot of POE devices, is there any reason to expect a significant power or noise penalty compared to the non-POE versions of the ICX 6450's?

I would think not, but one thing that gives me pause is this datasheet, where on page 13 they show that the POE versions use close to the max PSU power at a mere 5% throughput. Is that assuming all POE ports are filled with POE+ devices? If I only had a few POE devices, would it be much closer to the power/noise of the non-POE version? I have to think it would be, but I want to make sure I'm not making a huge mistake by buying the POE version and under-utilizing that feature.

Also, is there a large likelihood of getting hosed by lack of license or the wrong license level on these eBay units?

Thanks!

 

Attachments

  • Like
Reactions: tommybackeast

fohdeesha

Kaini Industries
Nov 20, 2016
1,960
1,783
113
29
fohdeesha.com
that power figure for the PoE model at 5% is a typo, that's what it would draw if you drew max PoE power on every port at the same time. My PoE and non-PoE models draw the exact same amount of power, then the PoE will start to draw a couple watts more per PoE device you add. the poe model is a TINY bit louder as it has 2 fans instead of 1, but they're quite quiet
 
  • Like
Reactions: scurrier

SocalServerGuy

New Member
Jul 11, 2017
2
0
1
38
Longtime lurker just wanted to say thanks to fohdeesha for lots of great info. I thought I would share my personal experience, I picked up a 6450-24 and a 6450-48 to integrate into my homelab. The 48 port is significantly louder at idle than the 24 port. The 24 port is great as far as noise goes but I was surprised how much louder the 48 port is at idle without any devices plugged in.

Also be careful with ebay listings I got burned by a seller advertising it had 10G-LIC-POD but guess what "license record empty" and now no response.