Drag to reposition cover

Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

AlphaG

Member
Jun 8, 2017
84
16
8
50
Thanks guys for your replies. I purchased 3 of the ICX6450-24P from tams to upgrade my network. Will be hitting up fs.com for fiber soon.

I’m a little afraid... whole new world from dumb switches.
 

arglebargle

H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈
Jul 15, 2018
656
233
43
Last edited:

istamov

New Member
Jul 31, 2015
14
4
3
Well, you already received information from @arglebargle, I can also confirm these NetApp QSFP SAS cables are working great as a DAC.
Just received my switches, flashed and set them up (thank you, @fohdeesha!), then did the tests below.

My test topology is:
Linux Ubuntu 16.04 Server (Mellanox ConnectX-3, 10.10.10.1/30) -----> 5m NetApp 12-00178 SAS cable -----> [1/2/1] ICX6610 [1/2/6] -----> 5m NetApp 12-00178 SAS cable -----> Linux CentOS 7 Server (Mellanox Connectx-3 PRO, 10.10.10.2/30)

Code:
### Server, int enp2s0d1, ip 10.10.10.1/30, MTU 9000
ubuntu# iperf -s -P 8 -B 10.10.10.1
------------------------------------------------------------
Server listening on TCP port 5001
Binding to local address 10.10.10.1
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[  4] local 10.10.10.1 port 5001 connected with 10.10.10.2 port 52348
[  5] local 10.10.10.1 port 5001 connected with 10.10.10.2 port 52350
[  6] local 10.10.10.1 port 5001 connected with 10.10.10.2 port 52349
[  7] local 10.10.10.1 port 5001 connected with 10.10.10.2 port 52351
[  8] local 10.10.10.1 port 5001 connected with 10.10.10.2 port 52352
[  9] local 10.10.10.1 port 5001 connected with 10.10.10.2 port 52353
[ 10] local 10.10.10.1 port 5001 connected with 10.10.10.2 port 52354
[ 11] local 10.10.10.1 port 5001 connected with 10.10.10.2 port 52355
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.0 sec  5.51 GBytes  4.73 Gbits/sec
[  7]  0.0-10.0 sec  7.05 GBytes  6.05 Gbits/sec
[  9]  0.0-10.0 sec  6.58 GBytes  5.65 Gbits/sec
[  5]  0.0-10.0 sec  4.26 GBytes  3.65 Gbits/sec
[  6]  0.0-10.0 sec  4.06 GBytes  3.48 Gbits/sec
[  8]  0.0-10.0 sec  8.01 GBytes  6.87 Gbits/sec
[ 10]  0.0-10.0 sec  5.00 GBytes  4.29 Gbits/sec
[ 11]  0.0-10.0 sec  5.17 GBytes  4.43 Gbits/sec
[SUM]  0.0-10.0 sec  45.6 GBytes  39.1 Gbits/sec


### Client, int ens1d1, ip 10.10.10.2/30, MTU 9000
centos# iperf -c 10.10.10.1 -P 8
------------------------------------------------------------
Client connecting to 10.10.10.1, TCP port 5001
TCP window size:  325 KByte (default)
------------------------------------------------------------
[ 12] local 10.10.10.2 port 52355 connected with 10.10.10.1 port 5001
[  4] local 10.10.10.2 port 52350 connected with 10.10.10.1 port 5001
[  3] local 10.10.10.2 port 52348 connected with 10.10.10.1 port 5001
[  5] local 10.10.10.2 port 52349 connected with 10.10.10.1 port 5001
[  7] local 10.10.10.2 port 52351 connected with 10.10.10.1 port 5001
[ 10] local 10.10.10.2 port 52353 connected with 10.10.10.1 port 5001
[  9] local 10.10.10.2 port 52354 connected with 10.10.10.1 port 5001
[  6] local 10.10.10.2 port 52352 connected with 10.10.10.1 port 5001
[ ID] Interval       Transfer     Bandwidth
[ 12]  0.0-10.0 sec  5.17 GBytes  4.44 Gbits/sec
[  3]  0.0-10.0 sec  5.51 GBytes  4.73 Gbits/sec
[  7]  0.0-10.0 sec  7.05 GBytes  6.05 Gbits/sec
[ 10]  0.0-10.0 sec  6.58 GBytes  5.65 Gbits/sec
[  4]  0.0-10.0 sec  4.26 GBytes  3.66 Gbits/sec
[  5]  0.0-10.0 sec  4.06 GBytes  3.48 Gbits/sec
[  9]  0.0-10.0 sec  5.00 GBytes  4.29 Gbits/sec
[  6]  0.0-10.0 sec  8.01 GBytes  6.88 Gbits/sec
[SUM]  0.0-10.0 sec  45.6 GBytes  39.2 Gbits/sec


SSH@ICX6610-48 Router#show int eth 1/2/1
40GigabitEthernet1/2/1 is up, line protocol is up
  Port up for 12 minute(s) 30 second(s)
  Hardware is 40GigabitEthernet, address is xxxx.xxxx.xxxx (bia xxxx.xxxx.xxxx)
  Interface type is 40Gig Fiber
  Configured speed 40Gbit, actual 40Gbit, configured duplex fdx, actual fdx
  Configured mdi mode AUTO, actual none
  Member of L2 VLAN ID 1, port is untagged, port state is FORWARDING
  BPDU guard is Disabled, ROOT protect is Disabled, Designated protect is Disabled
  Link Error Dampening is Disabled
  STP configured to ON, priority is level0, mac-learning is enabled
  Openflow is Disabled, Openflow Hybrid mode is Disabled,  Flow Control is enabled
  Mirror disabled, Monitor disabled
  Mac-notification is disabled
  Not member of any active trunks
  Not member of any configured trunks
  Port name is Ubuntu
  MTU 10200 bytes, encapsulation ethernet
  300 second input rate: 275970912 bits/sec, 28938 packets/sec, 0.70% utilization
  300 second output rate: 4934602432 bits/sec, 69395 packets/sec, 12.37% utilization
  21352777 packets input, 50929941506 bytes, 0 no buffer
  Received 0 broadcasts, 0 multicasts, 21352777 unicasts
  0 input errors, 0 CRC, 0 frame, 0 ignored
  0 runts, 0 giants
  43970928 packets output, 385579675525 bytes, 0 underruns
  Transmitted 1365 broadcasts, 449 multicasts, 43969114 unicasts
  0 output errors, 0 collisions
  Relay Agent Information option: Disabled

Egress queues:
Queue counters    Queued packets    Dropped Packets
    0            44948217                   0
    1                   0                   0
    2                   0                   0
    3                   0                   0
    4                   0                   0
    5                   0                   0
    6                   0                   0
    7                   0                   0

SSH@ICX6610-48 Router#show int eth 1/2/6
40GigabitEthernet1/2/6 is up, line protocol is up
  Port up for 12 minute(s) 42 second(s)
  Hardware is 40GigabitEthernet, address is xxxx.xxxx.xxxx (bia xxxx.xxxx.xxxx)
  Interface type is 40Gig Fiber
  Configured speed 40Gbit, actual 40Gbit, configured duplex fdx, actual fdx
  Configured mdi mode AUTO, actual none
  Member of L2 VLAN ID 1, port is untagged, port state is FORWARDING
  BPDU guard is Disabled, ROOT protect is Disabled, Designated protect is Disabled
  Link Error Dampening is Disabled
  STP configured to ON, priority is level0, mac-learning is enabled
  Openflow is Disabled, Openflow Hybrid mode is Disabled,  Flow Control is enabled
  Mirror disabled, Monitor disabled
  Mac-notification is disabled
  Not member of any active trunks
  Not member of any configured trunks
  Port name is CentOS
  MTU 10200 bytes, encapsulation ethernet
  300 second input rate: 5325435680 bits/sec, 75985 packets/sec, 13.32% utilization
  300 second output rate: 717185616 bits/sec, 36940 packets/sec, 1.81% utilization
  43969148 packets input, 385581562672 bytes, 0 no buffer
  Received 0 broadcasts, 0 multicasts, 43969148 unicasts
  351 input errors, 0 CRC, 0 frame, 0 ignored
  0 runts, 0 giants
  21354582 packets output, 50929899921 bytes, 0 underruns
  Transmitted 1365 broadcasts, 449 multicasts, 21352768 unicasts
  0 output errors, 0 collisions
  Relay Agent Information option: Disabled

Egress queues:
Queue counters    Queued packets    Dropped Packets
    0            21708255                  21
    1                   0                   0
    2                   0                   0
    3                   0                   0
    4                   0                   0
    5                   0                   0
    6                   0                   0
    7                   0                   0
As you can see the speed is between 39.1 and 39.5 Gbps, which is great! There are no any other optimisations on the operating systems/network stack/network cards, apart from setting MTU to 9000 and activating "jumbo" MTU on the ICX6610. Both servers are running Xeon E3-1230 V2.

The interesting thing is that I see input errors toward the client (CentOS), but when I reverse the tests there are no errors on the port towards Ubuntu server - will have to investigate this further.
 
Last edited:
  • Like
Reactions: leebo_28

tommybackeast

Active Member
Jun 10, 2018
253
83
28
Switch: Brocade 6450-24p

I am getting a Dell Server T420 as soon as i find one;
and will be getting a 10GBe PCI card for it to plug into the Brocade 6450 10GBe SFP+ port.
The distance between Dell Server and Brocade will be 1m (next to each other).

so I need to buy a 10gig card for the Dell Server:

Can anyone educate me as to the difference between:

Mellanox PCIe card with two optical fiber transceivers and 2m of fiber cable
vs.
Intel X520-DA1 10GB PCIe SFP+ with 2m TwinAx cable DAC
 

arglebargle

H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈
Jul 15, 2018
656
233
43
The Intel card will cost you about $35 more than a Mellanox Connect-X2 and shouldn't function any differently in practice. The only reason I'd buy an Intel 10Gb card is for BSD, the drivers on Windows and Linux are flawless (someone chime in about ESXi please ESXi reported fine as well.)

This is the $16 Mellanox card that most people are buying if you just want one port:
671798-001 COMPATIBLE 10GB MELLANOX CONNECTX-2 PCIe 10GBe ETHERNET NIC | eBay

Mellanox is great about compatibility with dacs and optics, you shouldn't have any trouble using most cables or transceivers. I use these:
MCP2104-X001B Mellanox 1M 10G SFP+ Passive Direct Attach Copper Twinax Cable | eBay
 
Last edited:

tommybackeast

Active Member
Jun 10, 2018
253
83
28
The Intel card will cost you about $35 more than a Mellanox Connect-X2 and shouldn't function any differently in practice. The only reason I'd buy an Intel 10Gb card is for BSD, the drivers on Windows and Linux are flawless (someone chime in about ESXi please.)

This is the $16 Mellanox card that most people are buying if you just want one port:
671798-001 COMPATIBLE 10GB MELLANOX CONNECTX-2 PCIe 10GBe ETHERNET NIC | eBay

Mellanox is great about compatibility with dacs and optics, you shouldn't have any trouble using most cables or transceivers. I use these:
MCP2104-X001B Mellanox 1M 10G SFP+ Passive Direct Attach Copper Twinax Cable | eBay

Thank you for mentioning ESXi - duh to me for not considering that.

I'm a network noob; so I do believe i'll spend a bit more money and just get the Intel card; for I know it will work with ESXi; and I won't have to worry about complex matters that might come up pertaining to Mellanox drivers re ESXi.

Plus Ebay has a 15% off sale tonight :)
 

K D

Well-Known Member
Dec 24, 2016
1,431
309
83
30041
I have a mellanox card from the same listing that works without issue with esxi 6.5
 

PGlover

Active Member
Nov 8, 2014
471
58
28
54
I finally was able to get the stuck breakout cable out of the QSFP slot and returned both cables to the Ebay seller.

I decided to get the following items based on your recommendation. I will test next week to see if I have better results than the Monoprice cable.

Brocade XBR-000232 QSFP 57-1000267-01 4*16Gb SWL 100m for DCX8510 EMC | eBay

MPO Fan-out Patch Cable, Multimode Fiber, MPO to LC Uniboot, 3-meter 10FT 603149398665 | eBay

Brocade 10GB 57-0000075-01 10G-SFPP-SR 10Gbase-sr/sw 850nm SFP+ Transceiver >500 8904725111763 | eBay
I purchased the QSFP and Fan-out patch cable below. The QSFP does not seems to fit snug in the ICX 6610 QSFP slot. You can pull it out very easily. Additionally, the fan-out-patch cable does not stay in the QSFP. You can easily pull it out as well. What am I doing wrong?

Brocade XBR-000232 QSFP 57-1000267-01 4*16Gb SWL 100m for DCX8510 EMC | eBay

MPO Fan-out Patch Cable, Multimode Fiber, MPO to LC Uniboot, 3-meter 10FT 603149398665 | eBay


IMG_1382.jpg
 

fohdeesha

Kaini Industries
Nov 20, 2016
1,994
1,813
113
29
fohdeesha.com
the qsfp isn't latching anymore because when you forcefully removed the broken monoprice DAC, it loosened/bent the retaining tabs in the QSFP slot. If you try that optic in another lost, it should latch via the handle

MPO needs to be gently pushed all the way in (push using the black part) and the blue part will snap back and latch. if you just push on the blue part it won't latch. it's keyed (only fits one direction) so make sure that's correct

to remove it, you then pull back on the blue plastic part
 

PGlover

Active Member
Nov 8, 2014
471
58
28
54
the qsfp isn't latching anymore because when you forcefully removed the broken monoprice DAC, it loosened/bent the retaining tabs in the QSFP slot. If you try that optic in another lost, it should latch via the handle

MPO needs to be gently pushed all the way in (push using the black part) and the blue part will snap back and latch. if you just push on the blue part it won't latch. it's keyed (only fits one direction) so make sure that's correct

to remove it, you then pull back on the blue plastic part
You are right.. Removing the broken monoprice DAC damaged the QSFP slot.. Is there anyway to fix this or replace the entire card slot.

I was able to push the MPO cable in and it did latch.
 

fohdeesha

Kaini Industries
Nov 20, 2016
1,994
1,813
113
29
fohdeesha.com
you might be able to fix it with very small screw drivers/tools by opening it up to get access to the outside of the QSFP cage and bending the retaining hooks back in, but it'll be a PITA. It's not a card slot, it's a big board permanently screwed down to the main motherboard in the switch. I wouldn't really worry about it unless you're prone to violent earthquakes or small children running around pulling on cable, no other reason it would come out really once fully seated in. you can also just replace it with one of the other 4 6610's you bought, lol
 

PGlover

Active Member
Nov 8, 2014
471
58
28
54
you might be able to fix it with very small screw drivers/tools by opening it up to get access to the outside of the QSFP cage and bending the retaining hooks back in, but it'll be a PITA. It's not a card slot, it's a big board permanently screwed down to the main motherboard in the switch. I wouldn't really worry about it unless you're prone to violent earthquakes or small children running around pulling on cable, no other reason it would come out really once fully seated in. you can also just replace it with one of the other 4 6610's you bought, lol
That slot cannot be saved.. Check out the photos.. Can I replace it with another board?

IMG_1383.jpg
 
  • Like
Reactions: arglebargle

fohdeesha

Kaini Industries
Nov 20, 2016
1,994
1,813
113
29
fohdeesha.com
HOLY S**T!!! I've never seen a slot damaged like that, it looks like it pulled the plastic pin mold straight off the pcb, that's insane

you can theoretically replace just that board, but good luck - I've never seen one for sale by itself in my life, not even from brocade directly - it was never a "replaceable" part. it also needs to be from a 48 port chassis (the stacking card is different on the 24 port models, as it does not have the management and serial ports)

At the very least if you plan to use that card again, I would straighten out those pins and cut them - so none of the pins are touching each other. Some of those are power and ground, if those short out the whole card will probably not work, depending on how it's fused
 
Last edited:
  • Like
Reactions: tommybackeast

arglebargle

H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈
Jul 15, 2018
656
233
43
I had a hell of a time getting the DAC out of my Connect-X2 when I upgraded to C-X3s and ended up ripping the pull tab off the DAC before prying it out with a small screwdriver. The catch tab in the slot was bent up so that the module couldn't release without something wedged in there to push it down.

Now I check carefully every time I buy *SFP gear, it's one of those things you don't really have to think about with 10-base cards and I think it bites everyone at least once.
 

PGlover

Active Member
Nov 8, 2014
471
58
28
54
HOLY S**T!!! I've never seen a slot damaged like that, it looks like it pulled the plastic pin mold straight off the pcb, that's insane

you can theoretically replace just that board, but good luck - I've never seen one for sale by itself in my life, not even from brocade directly - it was never a "replaceable" part. it also needs to be from a 48 port chassis (the stacking card is different on the 24 port models, as it does not have the management and serial ports)

At the very least if you plan to use that card again, I would straighten out those pins and cut them - so none of the pins are touching each other. Some of those are power and ground, if those short out the whole card will probably not work, depending on how it's fused
I will cut them.. But man I am pissed at that QFSP cable from monoprice...