Drag to reposition cover

Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

tozmo

Active Member
Feb 1, 2017
142
102
43
74
Anyone have any "best practices" for the Brocade ICX6610 48p?
Here is a little about my setup;
Windows 10 machine with a MCX354A-FCBT --> Cisco QSFP-40G-SR-BD Compatible [fs.com brand] --> 15m om5 LC to LC --> Cisco QSFP-40G-SR-BD Compatible --> ICX6610 port 1/2/1.
Server 2019 with a MCX354A-FCBT --> Mellanox MC2207130-002 56gb/s DAC --> ICX6610 port 1/2/6
I get about 30-35 Gbps running 3 iperf3 tests at once.
I wasnt sure about enabling all ports poe, does this increase overall power consumption?
thanks for any helpful tips or tricks
Why the BD at $169 vs SR4 at $39?
 

deified

New Member
Jun 11, 2020
4
0
1
Why the BD at $169 vs SR4 at $39?
@tozmo I got the BD because I had the LC fiber already installed and just wanted a quick switch to 40gb and they were 45(offered) a piece on ebay. Here is the link if anyone wants to check them out, the pair that I got were new in box.
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,728
3,078
113
33
fohdeesha.com
Unfortunately my settings were the same as yours dammit lol

what I am seeing is
When connecting through putty

ICX6610-24 Router#

I use to see

ICX6610-24 Switch#

unfortunately what you suggested was already done :-(

I know there’s two different modes for the switch after upgrading the firmware I now have it in router mode, that I’m looking to change back to switch mode and I can’t find documentation anywhere. unless of course I’m understanding something incorrectly.

The reason why I want to put it back to switch mode is under router mode it doesn’t function the way I need it.
until I did the firmware update it was working perfectly so I guess in fact I most likely should’ve done the firmware update.

for example:

can’t set IP address / Gateway etc,
all I get is invalid input, with almost every command I put in, so the inputs must be different under router mode, I’m assuming.

did you not just follow the bolded update/config guide in my OP? it gets you up to date, the switch configured in a "dumb" l2 only switching mode with an IP to manage it,, but running the router firmware so if you want to play with routing features at a later point you can. It's kind of silly to run the switch only image code, and in fact they're being killed off in the 7 series soon as the router image provides all the same features and more
 

hmw

Active Member
Apr 29, 2019
570
226
43
@fohdeesha - Thank you so much! Finally got my ICX-6610-24PE up and running, noise level thankfully isn't a problem (I have a 18U network cabinet with doors)

Amazing that I have my ESXi server connected to the switch with a $9 NetApp QSFP+ cable @ 40 Gbit !! woohoo !!!

I also used an Ipolex 10GBase-T SFP module and I'm doing 10 GbE over Cat5E :)

Does the switch support 24V Passive POE? Some of my devices are 24V passive, looks like I will be buying some 802.3af/at compatible cameras :)
 
  • Like
Reactions: fohdeesha

Spearfoot

Active Member
Apr 22, 2015
111
51
28
Question for the folks here who know more than I do about breakout cables -- and that would be all of you, because I know very little.:)

I'm shopping for a pair of 40Gb-to-4x10Gb breakout cables to use with my brand, spankin' new (to me) ICX 6610.

With respect to passive DAC versus active optical cables: what, if any, are the advantages/disadvantages of one over the other?

I have my eye on these Finisair optical cables, 'new' at $30 apiece:


And also these used Arista DAC cables at $29 each:


Later: Well, after a little light googling I've answered my own question: the active optical cable will use more power and work at a greater distance, while the passive copper uses less power and is only good for short runs. So, DAC it is!
 
Last edited:

klui

Well-Known Member
Feb 3, 2019
824
453
63
Later: Well, after a little light googling I've answered my own question: the active optical cable will use more power and work at a greater distance, while the passive copper uses less power and is only good for short runs. So, DAC it is!
A benefit of an AOC is they are much slimmer and allows easier cable management.
 
  • Like
Reactions: Spearfoot

infoMatt

Active Member
Apr 16, 2019
222
100
43
A benefit of an AOC is they are much slimmer and allows easier cable management.
Yes, they are essentially a fiber optic cable with the optics "glued together" in a single piece.
Although the cable is slimmer, you have to fiddle around with the entire SFP connector when moving those things around, partially negating the advantages of a slim cable.
 

klui

Well-Known Member
Feb 3, 2019
824
453
63
Although the cable is slimmer, you have to fiddle around with the entire SFP connector when moving those things around, partially negating the advantages of a slim cable.
Our racks in our lab at work have a multitude of mini-SAS HD and thin and thick passive copper DACs which make maintenance a bear. Rerouting an AOC's transceiver is welcomed compared to getting at a component blocked by those aforementioned monstrosities.
 
  • Like
Reactions: infoMatt

i386

Well-Known Member
Mar 18, 2016
4,220
1,540
113
34
Germany
I got a 6450-24p two months ago (thanks @fohdeesha for the licenses and for recommending and "supporting" the ICX switches :cool:).
It's a far better solution than that unmanaged 8port switch that I used for the ipmi/managemange ports in my serverrack.

It's stable, there are free firmware/software updates and I'm happy with it :D
 
  • Like
Reactions: fohdeesha

hmw

Active Member
Apr 29, 2019
570
226
43
Does anyone put dust covers/dust caps on the SFP and QSFP ports on front and back? Is it recommended?
 

infoMatt

Active Member
Apr 16, 2019
222
100
43
Does anyone put dust covers/dust caps on the SFP and QSFP ports on front and back? Is it recommended?
IMHO, if you have those handy then they cause no harm being put in; if you don't have those, unless your equipment lives in a very extreme environment, they don't make a big difference in dust collection inside the switch. The same damages (or worse) will occour also on the metal part of the RJ45 ports.
 
May 1, 2020
39
9
8
I bought an ICX6450 recently and I want to start making use of the SFP+ ports for my server.

Here's the question, can I use a QSFP+ to 4xSFP+ breakout DAC on my server end and connect up only some of the SFP+ connectors to the switch and get them to work at something like 20/30Gbit speeds? It's either this or I'll need to get a QSFP+ to SFP+ adapter and use just an SFP+ DAC.

I need at least 1 port for a 10Gbase-T transceiver for my computer in another room (I have the house wired with Cat6a).

EDIT: After looking a bit more, it seems that NICs don't typically support breakout cables; the one I'm specifically looking at getting is a ConnectX-3 rebrand (the HP 649281-B21) and Mellanox says none of their NICs support it with specific mention of the ConnectX-3 as an example. So it looks like an adapter and SFP+ DAC are the only options (until the switch bug bites again and I upgrade to a model with a QSFP port).
 
Last edited:

safrax

New Member
Jun 21, 2020
8
1
3
I've got an ICX-6610-48P on the way along with some mikrotik S+RJ10 SFPs that I'm planning on using to create a 10G link over CAT6 to a Mikrotik CRS305-1G-4S+IN that will sit on my desk. The final piece of my puzzle is the link from the switch to the desktop. From this thread it seems like the best bet is a ConnectX-3 (is that still the case?). I'm just not sure what I need to buy for the QSFP (??) to Mikrotik. I'd prefer to go fiber. If anyone could point me to the correct parts and cable I need to do 10G from the Mikrotik to the Desktop I'd appreciate it.
 

hmw

Active Member
Apr 29, 2019
570
226
43
I've got an ICX-6610-48P on the way along with some mikrotik S+RJ10 SFPs that I'm planning on using to create a 10G link over CAT6 to a Mikrotik CRS305-1G-4S+IN that will sit on my desk. The final piece of my puzzle is the link from the switch to the desktop. From this thread it seems like the best bet is a ConnectX-3 (is that still the case?). I'm just not sure what I need to buy for the QSFP (??) to Mikrotik. I'd prefer to go fiber. If anyone could point me to the correct parts and cable I need to do 10G from the Mikrotik to the Desktop I'd appreciate it.
QSFP+ and SFP+ are different standards.

SFP/SFP+ is smaller and does ~ 10GbE.
QSFP+ is bigger than SFP/SFP+ and can do 40 ~ 56GbE

Both can support direct attached cables or transceivers. And transceivers can be optical or copper. But your Mikrotik doesnt support QSFP, only SFP

If you wish to buy a ConnectX-3 card (that has two QSFP slots) you would need a QSFP to SFP adapter: Mellanox MAM1Q00A-QSA-SP (Single Pack MAM1Q00A-QSA) Ethernet Cable Adapter 40Gb/s to 10Gb/s QSFP to SFP+ and you can then use a copper transceiver on both ends: https://www.fs.com/products/87588.html or Cisco SFP-10G-T-S Compatible 10GBASE-T SFP+ Transceiver Module - OR you can use an optical transceiver: https://www.fs.com/products/65334.html and cable: https://www.fs.com/products/41730.html (available in different lengths)

Just make sure the optical transceivers you select are compatible with the Mikrotik switch and with the Mellanox card
 

tozmo

Active Member
Feb 1, 2017
142
102
43
74
I bought two Connect X3's (HP) and flashed them. Have QSFP+ -> SFP+ tranceivers from Ebay connected to my switch via DAC, and a QSFP+ Amazon cable as a peer-to-peer DAC connection between the two computers.

The Connect X3's are the way to go, imo. Or something newer that is SFP+ only since 40gbe seems to be a dead tech.
 

klui

Well-Known Member
Feb 3, 2019
824
453
63
QSFP+ and SFP+ are different standards.
It's just a lot of acronyms that are very confusing. Kinda like 802.11g, a, n, ac, ax. They should really rename them.

Because the SFP and QSFP form factors are common place in data centers newer data rates have piggybacked on these two for newer standards when possible.

SFP (1G); SFP+ (10G); SFP28 (25G, 'cuz it maxes out at 28Gbps)
QSFP (4 x 1G); QSFP+ (4 x 10G; 40G [+56G for Mellanox]); QSFP28 (100G; 4 x 28G)
OSFP (400G)
QSFP-DD (400G)

They're backward compatible. For example, QSFP28 to SFP28 breakouts should (ours at work does) support 40G to 4 x 10G. But QSFP+/SFP+ most likely will not support 100G/25G/4 X 25G.
 

Schut

New Member
Apr 13, 2020
9
2
3
I've got an ICX-6610-48P on the way along with some mikrotik S+RJ10 SFPs that I'm planning on using to create a 10G link over CAT6 to a Mikrotik CRS305-1G-4S+IN that will sit on my desk. The final piece of my puzzle is the link from the switch to the desktop. From this thread it seems like the best bet is a ConnectX-3 (is that still the case?). I'm just not sure what I need to buy for the QSFP (??) to Mikrotik. I'd prefer to go fiber. If anyone could point me to the correct parts and cable I need to do 10G from the Mikrotik to the Desktop I'd appreciate it.
Just to be completely clear, the Mikrotik CRS305-1G-4S+IN you say you want to connect to your desktop only has SFP+ ports on it. It does not have QSFP or QSFP+ ports on it, so those aren't relevant.

You have these choices as I see it:

1a) Another copper run to a copper card:
* A copper 10gbe card (I've been using Intel X540-T2s, and even used on EBay where I get them, they are pricey.)
* Another mikrotik S+RJ10 or similar for the SFP+ port on the CRS305-1G-4S+IN.
* A Cat6a copper cable

1b) Another copper run to an SFP+ card:
* A dual or single port SFP+ card. (see 2a below for details)
* A pair of mikrotik S+RJ10 or similar for the SFP+ port on the CRS305-1G-4S+IN and the SFP+ NIC card
* A Cat6a copper cable

2a) A fiber run:
* A dual or single port SFP+ card. I've been using Solarflare SFN7122Fs which go for $30-$40 on Ebay, but there's also the ConnectX-3 which I have no personal experience with, and there are surely others as well.
* Two 10GBASE-SR SFP+ fiber transceivers. I've been using the house branded ones from fs.com (Cisco SFP-10G-SR Compatible 10GBASE-SR SFP+ Transceiver Module), but I am certain you can save here if you shop around and know what you are doing. I'm just getting started on 10gb so I wanted something I knew would work. Probably $6-$20/transceiver, but the experts here should feel free to set me straight.
* An appropriate length of fiber cable. I've bought different lengths of these for short (<50 foot) fiber runs: OM4 50/125 LC-LC Multimode Fiber Optic Cable Duplex 5m (16ft) - Aqua. Again, you can probably save here, but I'm a novice so I went with fs.com to be sure I'd get something I was confident would work.

2b) A DAC run:
* A dual or single port SFP+ card. (same as 2a; see above.)
* A appropriate DAC (Direct Attached Copper) cable. For example, here's a 10ft cable from fs.com: Brocade 10G-SFPP-TWX-0301 10G SFP+ DAC Twinax Cable. This replaces both fiber transceivers and the fiber cable in 2a.

2a & 2b are the best choices in my book. If your switch is a good distance away in your room - say you have a large room - fiber may be the only practical option of the two - DAC cables are length limited. But if your CRS305-1G-4S+IN switch is very close to your desktop, the 2b (DAC) option is likely best since it will be cheaper and is very simple to understand and use. 1b may be a bit cheaper than 1a, but either will likely be more expensive than either 2a or 2b. I would do Fiber (2a) before I would do 1a unless I had to use copper for some reason, like for example, pre-existing Copper wiring.

In my server room I have been using a mix of all of these, and I'll admit to buying some stuff without cost being my overriding concern, just so I can see how it all works.
 
  • Like
Reactions: safrax and hmw

Rand__

Well-Known Member
Mar 6, 2014
6,626
1,767
113
There are only two 'newer' options:
25G in x8 or 100G in x16 slots...

Unless you want 100 or features that only newer cards aupport (RocE v2) the CX3 are fine
 

hmw

Active Member
Apr 29, 2019
570
226
43
I'll admit to buying some stuff without cost being my overriding concern, just so I can see how it all works.
I find myself buying 3x the requisite number of required patch cables just so that I can make the rack look pretty :eek: Or read the Great Deals section and blindly buy stuff off eBay

This forum is VERY addictive for that

:D:D