Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[Reece5646]

Unfortunately my settings were the same as yours dammit lol

what I am seeing is
When connecting through putty

ICX6610-24 Router#

I use to see

ICX6610-24 Switch#

unfortunately what you suggested was already done :-(

I know there’s two different modes for the switch after upgrading the firmware I now have it in router mode, that I’m looking to change back to switch mode and I can’t find documentation anywhere. unless of course I’m understanding something incorrectly.

The reason why I want to put it back to switch mode is under router mode it doesn’t function the way I need it.
until I did the firmware update it was working perfectly so I guess in fact I most likely should’ve done the firmware update.

for example:

can’t set IP address / Gateway etc,
all I get is invalid input, with almost every command I put in, so the inputs must be different under router mode, I’m assuming.

 

[hmw]

I know there’s two different modes for the switch after upgrading the firmware I now have it in router mode, that I’m looking to change back to switch mode and I can’t find documentation anywhere. unless of course I’m understanding something incorrectly.

Hmm - if you download the firmware from Brocade, there's - for example - the FCXS08030u.bin and the FCXR08030u.bin firmwares in 'Images'. The 'S' firmware is the switch firmware that is smaller and drops ALL L3 code. You can switch back and forth between these two images. Is this what you're referring to?

 

[klui]

ICX6610-24 Router#

I use to see

ICX6610-24 Switch#
.
.
.
can’t set IP address / Gateway etc, all I get is invalid input, with almost every command I put in, so the inputs must be different under router mode, I’m assuming.

Router code is more flexible and I would prefer to use that. In your old case there was a global IP address assigned for the switch's management switch through what I presume is ip address ....

What you could do on router code is

interface management 1​

ip address <IP> <mask>​

The above command will set your out-of-band management interface to what you want. You can also have IP addresses associated with your VLANs and you have access to those inband.

 

[Reece5646]

Hmm - if you download the firmware from Brocade, there's - for example - the FCXS08030u.bin and the FCXR08030u.bin firmwares in 'Images'. The 'S' firmware is the switch firmware that is smaller and drops ALL L3 code. You can switch back and forth between these two images. Is this what you're referring to?

 

[tozmo]

Anyone have any "best practices" for the Brocade ICX6610 48p?
Here is a little about my setup;
Windows 10 machine with a MCX354A-FCBT --> Cisco QSFP-40G-SR-BD Compatible [fs.com brand] --> 15m om5 LC to LC --> Cisco QSFP-40G-SR-BD Compatible --> ICX6610 port 1/2/1.
Server 2019 with a MCX354A-FCBT --> Mellanox MC2207130-002 56gb/s DAC --> ICX6610 port 1/2/6
I get about 30-35 Gbps running 3 iperf3 tests at once.
I wasnt sure about enabling all ports poe, does this increase overall power consumption?
thanks for any helpful tips or tricks

Why the BD at $169 vs SR4 at $39?

 

[deified]

Why the BD at $169 vs SR4 at $39?

@tozmo I got the BD because I had the LC fiber already installed and just wanted a quick switch to 40gb and they were 45(offered) a piece on ebay. Here is the link if anyone wants to check them out, the pair that I got were new in box.

 

[fohdeesha]

Unfortunately my settings were the same as yours dammit lol

what I am seeing is
When connecting through putty

ICX6610-24 Router#

I use to see

ICX6610-24 Switch#

unfortunately what you suggested was already done :-(

I know there’s two different modes for the switch after upgrading the firmware I now have it in router mode, that I’m looking to change back to switch mode and I can’t find documentation anywhere. unless of course I’m understanding something incorrectly.

The reason why I want to put it back to switch mode is under router mode it doesn’t function the way I need it.
until I did the firmware update it was working perfectly so I guess in fact I most likely should’ve done the firmware update.

for example:

can’t set IP address / Gateway etc,
all I get is invalid input, with almost every command I put in, so the inputs must be different under router mode, I’m assuming.

did you not just follow the bolded update/config guide in my OP? it gets you up to date, the switch configured in a "dumb" l2 only switching mode with an IP to manage it,, but running the router firmware so if you want to play with routing features at a later point you can. It's kind of silly to run the switch only image code, and in fact they're being killed off in the 7 series soon as the router image provides all the same features and more

 

[hmw]

@fohdeesha - Thank you so much! Finally got my ICX-6610-24PE up and running, noise level thankfully isn't a problem (I have a 18U network cabinet with doors)

Amazing that I have my ESXi server connected to the switch with a $9 NetApp QSFP+ cable @ 40 Gbit !! woohoo !!!

I also used an Ipolex 10GBase-T SFP module and I'm doing 10 GbE over Cat5E

Does the switch support 24V Passive POE? Some of my devices are 24V passive, looks like I will be buying some 802.3af/at compatible cameras

 

[infoMatt]

Does the switch support 24V Passive POE?

No, it doesn't. "Passive" PoE is not a standardized one, and each vendor has its quirks added to it. The only interoperable between vendors and equipments for sure is the 802.3af/at.

 

[Spearfoot]

Question for the folks here who know more than I do about breakout cables -- and that would be all of you, because I know very little.

I'm shopping for a pair of 40Gb-to-4x10Gb breakout cables to use with my brand, spankin' new (to me) ICX 6610.

With respect to passive DAC versus active optical cables: what, if any, are the advantages/disadvantages of one over the other?

I have my eye on these Finisair optical cables, 'new' at $30 apiece:

Finisar Quadwire 40G QSFP 4x10G SFP Fan-Out Cable FCBN 510QE2C02 2M NEW @3 | eBay

510QE2C02 2M 40G QSFP+ to 4 x 10GB SFP+ Active Optical Cable Fanout 2M. This item is New in Box and is in good working condition. Finisar's Quadwire® Fan-Out AOC is a 4x10 Gb/s parallel active optical cable for storage, data, and high-performance computing interconnectivity.

www.ebay.com

And also these used Arista DAC cables at $29 each:

Arista Original 2m DAC QSFP 40gb to 4 SFP 10gb Breakout Cable Cab-q-s-2m for sale online | eBay

Find many great new & used options and get the best deals for Arista Original 2m DAC QSFP 40gb to 4 SFP 10gb Breakout Cable Cab-q-s-2m at the best online prices at eBay! Free shipping for many products!

www.ebay.com

Later: Well, after a little light googling I've answered my own question: the active optical cable will use more power and work at a greater distance, while the passive copper uses less power and is only good for short runs. So, DAC it is!

 

[klui]

Later: Well, after a little light googling I've answered my own question: the active optical cable will use more power and work at a greater distance, while the passive copper uses less power and is only good for short runs. So, DAC it is!

A benefit of an AOC is they are much slimmer and allows easier cable management.

 

[infoMatt]

A benefit of an AOC is they are much slimmer and allows easier cable management.

Yes, they are essentially a fiber optic cable with the optics "glued together" in a single piece.
Although the cable is slimmer, you have to fiddle around with the entire SFP connector when moving those things around, partially negating the advantages of a slim cable.

 

[klui]

Although the cable is slimmer, you have to fiddle around with the entire SFP connector when moving those things around, partially negating the advantages of a slim cable.

Our racks in our lab at work have a multitude of mini-SAS HD and thin and thick passive copper DACs which make maintenance a bear. Rerouting an AOC's transceiver is welcomed compared to getting at a component blocked by those aforementioned monstrosities.

 

[i386]

I got a 6450-24p two months ago (thanks @fohdeesha for the licenses and for recommending and "supporting" the ICX switches ).
It's a far better solution than that unmanaged 8port switch that I used for the ipmi/managemange ports in my serverrack.

It's stable, there are free firmware/software updates and I'm happy with it

 

[hmw]

Does anyone put dust covers/dust caps on the SFP and QSFP ports on front and back? Is it recommended?

 

[infoMatt]

Does anyone put dust covers/dust caps on the SFP and QSFP ports on front and back? Is it recommended?

IMHO, if you have those handy then they cause no harm being put in; if you don't have those, unless your equipment lives in a very extreme environment, they don't make a big difference in dust collection inside the switch. The same damages (or worse) will occour also on the metal part of the RJ45 ports.

 

[BenjaminCalibration]

I bought an ICX6450 recently and I want to start making use of the SFP+ ports for my server.

Here's the question, can I use a QSFP+ to 4xSFP+ breakout DAC on my server end and connect up only some of the SFP+ connectors to the switch and get them to work at something like 20/30Gbit speeds? It's either this or I'll need to get a QSFP+ to SFP+ adapter and use just an SFP+ DAC.

I need at least 1 port for a 10Gbase-T transceiver for my computer in another room (I have the house wired with Cat6a).

EDIT: After looking a bit more, it seems that NICs don't typically support breakout cables; the one I'm specifically looking at getting is a ConnectX-3 rebrand (the HP 649281-B21) and Mellanox says none of their NICs support it with specific mention of the ConnectX-3 as an example. So it looks like an adapter and SFP+ DAC are the only options (until the switch bug bites again and I upgrade to a model with a QSFP port).

 

[safrax]

I've got an ICX-6610-48P on the way along with some mikrotik S+RJ10 SFPs that I'm planning on using to create a 10G link over CAT6 to a Mikrotik CRS305-1G-4S+IN that will sit on my desk. The final piece of my puzzle is the link from the switch to the desktop. From this thread it seems like the best bet is a ConnectX-3 (is that still the case?). I'm just not sure what I need to buy for the QSFP (??) to Mikrotik. I'd prefer to go fiber. If anyone could point me to the correct parts and cable I need to do 10G from the Mikrotik to the Desktop I'd appreciate it.

 

[hmw]

I've got an ICX-6610-48P on the way along with some mikrotik S+RJ10 SFPs that I'm planning on using to create a 10G link over CAT6 to a Mikrotik CRS305-1G-4S+IN that will sit on my desk. The final piece of my puzzle is the link from the switch to the desktop. From this thread it seems like the best bet is a ConnectX-3 (is that still the case?). I'm just not sure what I need to buy for the QSFP (??) to Mikrotik. I'd prefer to go fiber. If anyone could point me to the correct parts and cable I need to do 10G from the Mikrotik to the Desktop I'd appreciate it.

QSFP+ and SFP+ are different standards.

SFP/SFP+ is smaller and does ~ 10GbE.
QSFP+ is bigger than SFP/SFP+ and can do 40 ~ 56GbE

Both can support direct attached cables or transceivers. And transceivers can be optical or copper. But your Mikrotik doesnt support QSFP, only SFP

If you wish to buy a ConnectX-3 card (that has two QSFP slots) you would need a QSFP to SFP adapter: Mellanox MAM1Q00A-QSA-SP (Single Pack MAM1Q00A-QSA) Ethernet Cable Adapter 40Gb/s to 10Gb/s QSFP to SFP+ and you can then use a copper transceiver on both ends: https://www.fs.com/products/87588.html or Cisco SFP-10G-T-S Compatible 10GBASE-T SFP+ Transceiver Module - OR you can use an optical transceiver: https://www.fs.com/products/65334.html and cable: https://www.fs.com/products/41730.html (available in different lengths)

Just make sure the optical transceivers you select are compatible with the Mikrotik switch and with the Mellanox card

 

[tozmo]

I bought two Connect X3's (HP) and flashed them. Have QSFP+ -> SFP+ tranceivers from Ebay connected to my switch via DAC, and a QSFP+ Amazon cable as a peer-to-peer DAC connection between the two computers.

The Connect X3's are the way to go, imo. Or something newer that is SFP+ only since 40gbe seems to be a dead tech.