Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

Zervun

Member
Feb 2, 2019
44
7
8
Oregon
Any of the rear 40g ports (there are 4 of them per switch 2x breakout, 2x stack) can be used for stacking. However the 2x breakout ports cannot be used for 40g, only the 40g only ones. (Fohdeesha just answered my question on this yesterday)

So what I am going to do with mine is stack across the breakout ports, and use the 4 total ports across my 2x 6610's for 40g.

The max ports per 6610 you can get straight 40g is 2x per switch (if you disable stacking and/or use stacking on the breakout ports).

As for POE - I hadn't seen much difference in price on the 6610's POE vs. non-POE so you might as well get a POE one. By default POE is disabled and you can enable it per port when you need it. I believe the switches are built with the same number fans/power supplies.

Make sure you get them with 2x power supplies and 2x fans as they are expensive/hard to come by (especially the fans). Also keep in mind the exhaust direction - the E models are rear exhaust direction

If you are looking for something not as loud the 6450 is the way to go but you have less ports/no 40g no dual power supplies. I would not get the 6610s if they are in the same room as you are.
 
  • Like
Reactions: richrichgreen

Kewjoe

New Member
Feb 3, 2019
6
2
3
Got my ICX6450-24P yesterday. Followed the guide. Everything is working great. Thanks to @fohdeesha for the awesome thread and all the help!. I have A LOT of reading to do :)
 
Last edited:
  • Like
Reactions: fohdeesha

Blue)(Fusion

Member
Mar 1, 2017
86
14
8
I can't believe I didn't see this thread much much sooner.

I currently have a silly Netgear XSM7224S for 24x 10G and another Netgear GSM7328PS for 24x PoE 1G in my rack. This could knock out both into one awesome switch.

I definitely would need the 2x 40G ports to breakout to 10G links. Do I need the premium version for this option? The premium is of course quite a bit more expensive...

Also, does anybody have a list of what SFP+ modules are compatible with this? I have a ton of Netgear and off-brand SFP+ modules and Cisco DACs.
 

dlaffin1012

New Member
Feb 21, 2019
12
0
1
Ok that is good to hear that I can still have 4 usable 40g ports. Sinc ea lot of you here have a lot of good knowledge, I suppose I can ask some advice on my setup.

Currently- I have a dedicated computer running pfsense as the firewall and router. That runs into the LB4M switch and everything from there is connected to the switch. I have a HP Proliant ML350P G8 running a few VM's one being my NAS. that NAS is connected via SFP+. I also have a separate computer running as my backup for the NAS also running SFP+.

Currently Looking to Setup- I would like to get two 6610's and stack them. Then connect the NAS and Backup System via 40G but also have LACP on those 40G so I can have redundancy. By this I mean having one cable in the first switch and one in the second. This would require me to have dual QSFP cards in each system or one dual port card in each system. I also wanted to do the same to the pfsense system with the 10G ports, so once I start doing more VLANs there wouldn't be a bottleneck and I would also have redundancy if a switch goes down. I also wanted to have dual 10G cables for my other VM's for redundancy as well.

My main questions for my new setup- I keep reading that I should do two dual cards rather than dual port cards because if one card goes down the other is there vs if the dual port card goes down, then I have no connection. the issues I run in to is that both the pfsense and backup systems have motherboards with only one PCIE slot because of their form factor. So I would have to replace the entirety of both systems with systems with more PCIE slots. Starts to get pricey. Is it really worth having dual cards vs dual port cards. Is the switch more likely to die before a card or is it the other way around.

My other question being do I turn off routing on pfsense and allow the switch to do it, one so I can dedicate pfsense as a firewall and also because of VLAN bottlenecking? Although I assume 10G would stop it from bottlenecking.

I was thinking of condensing all my systems into the HP server but
1- it is frowned upon to have the backup and NAS in the same system.
2- it is frowned upon to virtualize pfsense into a server.
 

Kewjoe

New Member
Feb 3, 2019
6
2
3
I can't believe I didn't see this thread much much sooner.

I currently have a silly Netgear XSM7224S for 24x 10G and another Netgear GSM7328PS for 24x PoE 1G in my rack. This could knock out both into one awesome switch.

I definitely would need the 2x 40G ports to breakout to 10G links. Do I need the premium version for this option? The premium is of course quite a bit more expensive...

Also, does anybody have a list of what SFP+ modules are compatible with this? I have a ton of Netgear and off-brand SFP+ modules and Cisco DACs.
Coming from someone who found this thread a month ago. I highly suggest you read at least the first post in this thread. I believe both of your questions are answered (at least in part). But there is a lot of great info throughout the thread (search by keyword "DAC" and you'll find several conversations about it for instance).
 
  • Like
Reactions: fohdeesha

Kewjoe

New Member
Feb 3, 2019
6
2
3
My main questions for my new setup- I keep reading that I should do two dual cards rather than dual port cards because if one card goes down the other is there vs if the dual port card goes down, then I have no connection. the issues I run in to is that both the pfsense and backup systems have motherboards with only one PCIE slot because of their form factor. So I would have to replace the entirety of both systems with systems with more PCIE slots. Starts to get pricey. Is it really worth having dual cards vs dual port cards. Is the switch more likely to die before a card or is it the other way around.

My other question being do I turn off routing on pfsense and allow the switch to do it, one so I can dedicate pfsense as a firewall and also because of VLAN bottlenecking? Although I assume 10G would stop it from bottlenecking.

I was thinking of condensing all my systems into the HP server but
1- it is frowned upon to have the backup and NAS in the same system.
2- it is frowned upon to virtualize pfsense into a server.
There are far more knowledgeable people that can answer your questions. But I plan to do the inter-vlan routing on my ICX6450 and leave the firewall duties on my (virtualized :)) pfsense. It's a beefy L3 switch, might as well take advantage of getting full line speeds.
 

kapone

Well-Known Member
May 23, 2015
784
384
63
Ok that is good to hear that I can still have 4 usable 40g ports. Sinc ea lot of you here have a lot of good knowledge, I suppose I can ask some advice on my setup.

Currently- I have a dedicated computer running pfsense as the firewall and router. That runs into the LB4M switch and everything from there is connected to the switch. I have a HP Proliant ML350P G8 running a few VM's one being my NAS. that NAS is connected via SFP+. I also have a separate computer running as my backup for the NAS also running SFP+.

Currently Looking to Setup- I would like to get two 6610's and stack them. Then connect the NAS and Backup System via 40G but also have LACP on those 40G so I can have redundancy. By this I mean having one cable in the first switch and one in the second. This would require me to have dual QSFP cards in each system or one dual port card in each system. I also wanted to do the same to the pfsense system with the 10G ports, so once I start doing more VLANs there wouldn't be a bottleneck and I would also have redundancy if a switch goes down. I also wanted to have dual 10G cables for my other VM's for redundancy as well.

My main questions for my new setup- I keep reading that I should do two dual cards rather than dual port cards because if one card goes down the other is there vs if the dual port card goes down, then I have no connection. the issues I run in to is that both the pfsense and backup systems have motherboards with only one PCIE slot because of their form factor. So I would have to replace the entirety of both systems with systems with more PCIE slots. Starts to get pricey. Is it really worth having dual cards vs dual port cards. Is the switch more likely to die before a card or is it the other way around.

My other question being do I turn off routing on pfsense and allow the switch to do it, one so I can dedicate pfsense as a firewall and also because of VLAN bottlenecking? Although I assume 10G would stop it from bottlenecking.

I was thinking of condensing all my systems into the HP server but
1- it is frowned upon to have the backup and NAS in the same system.
2- it is frowned upon to virtualize pfsense into a server.
If this is for "home" not a "homelab", you're overthinking this. If it is in fact a lab, fire away.
 

trippinnik

New Member
Oct 13, 2018
11
0
1
Just got one of these bad boy 6610s. Looks like it already has the licenses installed so I want to back them up before reseting. What do I need to do that? Apologies if this is answered above I tried searching for the answer and didn't come up with results.
 

Zervun

Member
Feb 2, 2019
44
7
8
Oregon
Currently Looking to Setup- I would like to get two 6610's and stack them. Then connect the NAS and Backup System via 40G but also have LACP on those 40G so I can have redundancy. By this I mean having one cable in the first switch and one in the second. This would require me to have dual QSFP cards in each system or one dual port card in each system. I also wanted to do the same to the pfsense system with the 10G ports, so once I start doing more VLANs there wouldn't be a bottleneck and I would also have redundancy if a switch goes down. I also wanted to have dual 10G cables for my other VM's for redundancy as well.

My main questions for my new setup- I keep reading that I should do two dual cards rather than dual port cards because if one card goes down the other is there vs if the dual port card goes down, then I have no connection. the issues I run in to is that both the pfsense and backup systems have motherboards with only one PCIE slot because of their form factor. So I would have to replace the entirety of both systems with systems with more PCIE slots. Starts to get pricey. Is it really worth having dual cards vs dual port cards. Is the switch more likely to die before a card or is it the other way around.
A bit overkill for a home lab. Since you don't need many 40g ports I'd just simplify it on one switch and keep the other switch as backup saving power/noise. Other stuff is going to die probably before a network card. Can always swap one in easily unless this is mission critical business stuff or you just want to play around with LACP etc. Depending on the motherboard the PCIE speeds might be reduced with multiple cards but probably won't matter. Some of the Mellenox cards run at 8x (will run at 4x fine and doubtful you will hit any cap on that.
 
Last edited:

Zervun

Member
Feb 2, 2019
44
7
8
Oregon
Just got one of these bad boy 6610s. Looks like it already has the licenses installed so I want to back them up before reseting. What do I need to do that? Apologies if this is answered above I tried searching for the answer and didn't come up with results.
For a fully licensed switch there will be 4 licenses under "show license"
 

trippinnik

New Member
Oct 13, 2018
11
0
1
I read if I erase the config I loose the licenses that are installed. do i just need the License ID to reinstall? I have two licenses:
ICX6610-10G-LIC-POD
ICX6610-PREM-LIC-SW
 

ViciousXUSMC

Active Member
Nov 27, 2016
201
98
28
37
Been having fun with the HPE Aruba S2500 switches that were so cheap.
Now kind of thinking about getting one of these.

Normally I associate Brocade with Fiber Channel, and I love the black color vs the dull grey of the Aruba stuff :)

Question however (and I did read about 25 pages of this thread)
The OP made no mention of a requirement for licensing, but several posts in the thread make mention of needing licensing to unlock features (4 different licenses if I am correct?)

So perhaps these are not the cheap switches they appear to be after licensing cost, that is what I need to verify before I look to buy.

Could somebody please break down the licensing for me a bit as far as what each license does and about how much it costs?
Once you do obtain one, I assume it's perpetual and not something you need an annual agreement for?

Thanks!
 

jarekd

New Member
Apr 21, 2018
7
0
1
40
they are EOL devices so you cannot purchase licenses directly from ruckus/brocade anymore, there are some places like terabit systems selling old stock licenses for around $1k each

it's a license xml file generated based on your switch's license ID
Ruckus is still selling licenses for ICX 6xxx series (e.g. ICX6610).

Asked their ordering@ and all licenses are still orderable until November, 2023.
New hardware cannot be ordered any more.
 

Ouraing

New Member
Dec 31, 2018
25
24
3
Factory resetting does remove the licenses, so you want to make sure you have a backup of the license info to reinstall it.

You have Ports on Demand licenses and you have Feature licenses, the details vary slightly between the 6610 and 6450.

For a 6610 (Ruckus ICX 6610 Switch Data Sheet | Technical Documents | Ruckus Wireless Support):
Without a license ports 1/3/1 to 1/3/8 (the front ports) are 1Gb only, a single PoD license enabled 4 ports at a time so to be fully licensed you need 2 PoD licenses to get 8 10Gb ports. The rear ports are fully enabled without a license. Fully licensed you can have up to 16 10Gb ports and 2 40Gb ports.

Basic layer 3 (static routes, ACLs) is included in the base license. Most dynamic routing features for IPV4 and IPV6 (OSPF, VRRP, PBR, RIP) require a premium. Note this includes IPV6 versions of these protocols as well. If you want BGP, GRE, IPV6 Tunneling or VRP you need an advanced license (which includes the premium features).

For a 6450 (Ruckus ICX 6430 and 6450 Switch Data Sheet | Technical Documents | Ruckus Wireless Support):
Without a license 2 of the 4 front SFP+ ports are limited to 1Gb only. A PoD license enables 2 ports at a time, so to be fully licensed you only need a single PoD license.

Basic layer 3 (static routes, ACLs) for IPV4 and IPV6 are included in the base license. More advanced IPV4 features like OSPF, VRRP, and GRE require the premium license to unlock, no IPV6 dynamic features. No BGP or IPV6 tunneling support on this model either.
 

fohdeesha

Kaini Industries
Nov 20, 2016
1,907
1,713
113
29
fohdeesha.com
Just got one of these bad boy 6610s. Looks like it already has the licenses installed so I want to back them up before reseting. What do I need to do that? Apologies if this is answered above I tried searching for the answer and didn't come up with results.
you can't back them up or export them (at least not without a JTAG unit). If you just run "erase startup-config" from the enable level that will wipe the config and keep the licenses