Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[Zervun]

Any of the rear 40g ports (there are 4 of them per switch 2x breakout, 2x stack) can be used for stacking. However the 2x breakout ports cannot be used for 40g, only the 40g only ones. (Fohdeesha just answered my question on this yesterday)

So what I am going to do with mine is stack across the breakout ports, and use the 4 total ports across my 2x 6610's for 40g.

The max ports per 6610 you can get straight 40g is 2x per switch (if you disable stacking and/or use stacking on the breakout ports).

As for POE - I hadn't seen much difference in price on the 6610's POE vs. non-POE so you might as well get a POE one. By default POE is disabled and you can enable it per port when you need it. I believe the switches are built with the same number fans/power supplies.

Make sure you get them with 2x power supplies and 2x fans as they are expensive/hard to come by (especially the fans). Also keep in mind the exhaust direction - the E models are rear exhaust direction

If you are looking for something not as loud the 6450 is the way to go but you have less ports/no 40g no dual power supplies. I would not get the 6610s if they are in the same room as you are.

 

[Kewjoe]

Got my ICX6450-24P yesterday. Followed the guide. Everything is working great. Thanks to @fohdeesha for the awesome thread and all the help!. I have A LOT of reading to do

 

[Freebsd1976]

Got my ICX6450-24P yesterday. Followed @fohdeesha guide. Everything is working great. Thanks for the awesome thread. I have A LOT of reading to do

Would you mind post item link?

 

[Kewjoe]

Would you mind post item link?

You're asking for the ebay listing?

edit: in case yes, it's here: Brocade ICX (ICX6450-24P) 24-Ports Rack-mountable Switch - PoE | eBay

Shipped from California to Ontario Canada. He packed it very well. Unit is very mint condition. no marks on it at all.

 

[Freebsd1976]

You're asking for the ebay listing?

edit: in case yes, it's here: Brocade ICX (ICX6450-24P) 24-Ports Rack-mountable Switch - PoE | eBay

Shipped from California to Ontario Canada. He packed it very well. Unit is very mint condition. no marks on it at all.

thanks

 

[Blue)(Fusion]

I can't believe I didn't see this thread much much sooner.

I currently have a silly Netgear XSM7224S for 24x 10G and another Netgear GSM7328PS for 24x PoE 1G in my rack. This could knock out both into one awesome switch.

I definitely would need the 2x 40G ports to breakout to 10G links. Do I need the premium version for this option? The premium is of course quite a bit more expensive...

Also, does anybody have a list of what SFP+ modules are compatible with this? I have a ton of Netgear and off-brand SFP+ modules and Cisco DACs.

 

[dlaffin1012]

Ok that is good to hear that I can still have 4 usable 40g ports. Sinc ea lot of you here have a lot of good knowledge, I suppose I can ask some advice on my setup.

Currently- I have a dedicated computer running pfsense as the firewall and router. That runs into the LB4M switch and everything from there is connected to the switch. I have a HP Proliant ML350P G8 running a few VM's one being my NAS. that NAS is connected via SFP+. I also have a separate computer running as my backup for the NAS also running SFP+.

Currently Looking to Setup- I would like to get two 6610's and stack them. Then connect the NAS and Backup System via 40G but also have LACP on those 40G so I can have redundancy. By this I mean having one cable in the first switch and one in the second. This would require me to have dual QSFP cards in each system or one dual port card in each system. I also wanted to do the same to the pfsense system with the 10G ports, so once I start doing more VLANs there wouldn't be a bottleneck and I would also have redundancy if a switch goes down. I also wanted to have dual 10G cables for my other VM's for redundancy as well.

My main questions for my new setup- I keep reading that I should do two dual cards rather than dual port cards because if one card goes down the other is there vs if the dual port card goes down, then I have no connection. the issues I run in to is that both the pfsense and backup systems have motherboards with only one PCIE slot because of their form factor. So I would have to replace the entirety of both systems with systems with more PCIE slots. Starts to get pricey. Is it really worth having dual cards vs dual port cards. Is the switch more likely to die before a card or is it the other way around.

My other question being do I turn off routing on pfsense and allow the switch to do it, one so I can dedicate pfsense as a firewall and also because of VLAN bottlenecking? Although I assume 10G would stop it from bottlenecking.

I was thinking of condensing all my systems into the HP server but
1- it is frowned upon to have the backup and NAS in the same system.
2- it is frowned upon to virtualize pfsense into a server.

 

[Kewjoe]

I can't believe I didn't see this thread much much sooner.

I currently have a silly Netgear XSM7224S for 24x 10G and another Netgear GSM7328PS for 24x PoE 1G in my rack. This could knock out both into one awesome switch.

I definitely would need the 2x 40G ports to breakout to 10G links. Do I need the premium version for this option? The premium is of course quite a bit more expensive...

Also, does anybody have a list of what SFP+ modules are compatible with this? I have a ton of Netgear and off-brand SFP+ modules and Cisco DACs.

Coming from someone who found this thread a month ago. I highly suggest you read at least the first post in this thread. I believe both of your questions are answered (at least in part). But there is a lot of great info throughout the thread (search by keyword "DAC" and you'll find several conversations about it for instance).

 

[Kewjoe]

My main questions for my new setup- I keep reading that I should do two dual cards rather than dual port cards because if one card goes down the other is there vs if the dual port card goes down, then I have no connection. the issues I run in to is that both the pfsense and backup systems have motherboards with only one PCIE slot because of their form factor. So I would have to replace the entirety of both systems with systems with more PCIE slots. Starts to get pricey. Is it really worth having dual cards vs dual port cards. Is the switch more likely to die before a card or is it the other way around.

My other question being do I turn off routing on pfsense and allow the switch to do it, one so I can dedicate pfsense as a firewall and also because of VLAN bottlenecking? Although I assume 10G would stop it from bottlenecking.

I was thinking of condensing all my systems into the HP server but
1- it is frowned upon to have the backup and NAS in the same system.
2- it is frowned upon to virtualize pfsense into a server.

There are far more knowledgeable people that can answer your questions. But I plan to do the inter-vlan routing on my ICX6450 and leave the firewall duties on my (virtualized ) pfsense. It's a beefy L3 switch, might as well take advantage of getting full line speeds.

 

[kapone]

Ok that is good to hear that I can still have 4 usable 40g ports. Sinc ea lot of you here have a lot of good knowledge, I suppose I can ask some advice on my setup.

Currently- I have a dedicated computer running pfsense as the firewall and router. That runs into the LB4M switch and everything from there is connected to the switch. I have a HP Proliant ML350P G8 running a few VM's one being my NAS. that NAS is connected via SFP+. I also have a separate computer running as my backup for the NAS also running SFP+.

Currently Looking to Setup- I would like to get two 6610's and stack them. Then connect the NAS and Backup System via 40G but also have LACP on those 40G so I can have redundancy. By this I mean having one cable in the first switch and one in the second. This would require me to have dual QSFP cards in each system or one dual port card in each system. I also wanted to do the same to the pfsense system with the 10G ports, so once I start doing more VLANs there wouldn't be a bottleneck and I would also have redundancy if a switch goes down. I also wanted to have dual 10G cables for my other VM's for redundancy as well.

My main questions for my new setup- I keep reading that I should do two dual cards rather than dual port cards because if one card goes down the other is there vs if the dual port card goes down, then I have no connection. the issues I run in to is that both the pfsense and backup systems have motherboards with only one PCIE slot because of their form factor. So I would have to replace the entirety of both systems with systems with more PCIE slots. Starts to get pricey. Is it really worth having dual cards vs dual port cards. Is the switch more likely to die before a card or is it the other way around.

My other question being do I turn off routing on pfsense and allow the switch to do it, one so I can dedicate pfsense as a firewall and also because of VLAN bottlenecking? Although I assume 10G would stop it from bottlenecking.

I was thinking of condensing all my systems into the HP server but
1- it is frowned upon to have the backup and NAS in the same system.
2- it is frowned upon to virtualize pfsense into a server.

If this is for "home" not a "homelab", you're overthinking this. If it is in fact a lab, fire away.

 

[trippinnik]

Just got one of these bad boy 6610s. Looks like it already has the licenses installed so I want to back them up before reseting. What do I need to do that? Apologies if this is answered above I tried searching for the answer and didn't come up with results.

 

[Zervun]

Currently Looking to Setup- I would like to get two 6610's and stack them. Then connect the NAS and Backup System via 40G but also have LACP on those 40G so I can have redundancy. By this I mean having one cable in the first switch and one in the second. This would require me to have dual QSFP cards in each system or one dual port card in each system. I also wanted to do the same to the pfsense system with the 10G ports, so once I start doing more VLANs there wouldn't be a bottleneck and I would also have redundancy if a switch goes down. I also wanted to have dual 10G cables for my other VM's for redundancy as well.

My main questions for my new setup- I keep reading that I should do two dual cards rather than dual port cards because if one card goes down the other is there vs if the dual port card goes down, then I have no connection. the issues I run in to is that both the pfsense and backup systems have motherboards with only one PCIE slot because of their form factor. So I would have to replace the entirety of both systems with systems with more PCIE slots. Starts to get pricey. Is it really worth having dual cards vs dual port cards. Is the switch more likely to die before a card or is it the other way around.

A bit overkill for a home lab. Since you don't need many 40g ports I'd just simplify it on one switch and keep the other switch as backup saving power/noise. Other stuff is going to die probably before a network card. Can always swap one in easily unless this is mission critical business stuff or you just want to play around with LACP etc. Depending on the motherboard the PCIE speeds might be reduced with multiple cards but probably won't matter. Some of the Mellenox cards run at 8x (will run at 4x fine and doubtful you will hit any cap on that.

 

[Zervun]

Just got one of these bad boy 6610s. Looks like it already has the licenses installed so I want to back them up before reseting. What do I need to do that? Apologies if this is answered above I tried searching for the answer and didn't come up with results.

For a fully licensed switch there will be 4 licenses under "show license"

 

[trippinnik]

I read if I erase the config I loose the licenses that are installed. do i just need the License ID to reinstall? I have two licenses:
ICX6610-10G-LIC-POD
ICX6610-PREM-LIC-SW

 

[ViciousXUSMC]

Been having fun with the HPE Aruba S2500 switches that were so cheap.
Now kind of thinking about getting one of these.

Normally I associate Brocade with Fiber Channel, and I love the black color vs the dull grey of the Aruba stuff

Question however (and I did read about 25 pages of this thread)
The OP made no mention of a requirement for licensing, but several posts in the thread make mention of needing licensing to unlock features (4 different licenses if I am correct?)

So perhaps these are not the cheap switches they appear to be after licensing cost, that is what I need to verify before I look to buy.

Could somebody please break down the licensing for me a bit as far as what each license does and about how much it costs?
Once you do obtain one, I assume it's perpetual and not something you need an annual agreement for?

Thanks!

 

[jarekd]

they are EOL devices so you cannot purchase licenses directly from ruckus/brocade anymore, there are some places like terabit systems selling old stock licenses for around $1k each

it's a license xml file generated based on your switch's license ID

Ruckus is still selling licenses for ICX 6xxx series (e.g. ICX6610).

Asked their ordering@ and all licenses are still orderable until November, 2023.
New hardware cannot be ordered any more.

 

[Zervun]

Could somebody please break down the licensing for me a bit as far as what each license does and about how much it costs?
Once you do obtain one, I assume it's perpetual and not something you need an annual agreement for?

Thanks!

See post #6

 

[Ouraing]

Factory resetting does remove the licenses, so you want to make sure you have a backup of the license info to reinstall it.

You have Ports on Demand licenses and you have Feature licenses, the details vary slightly between the 6610 and 6450.

For a 6610 (Ruckus ICX 6610 Switch Data Sheet | Technical Documents | Ruckus Wireless Support):
Without a license ports 1/3/1 to 1/3/8 (the front ports) are 1Gb only, a single PoD license enabled 4 ports at a time so to be fully licensed you need 2 PoD licenses to get 8 10Gb ports. The rear ports are fully enabled without a license. Fully licensed you can have up to 16 10Gb ports and 2 40Gb ports.

Basic layer 3 (static routes, ACLs) is included in the base license. Most dynamic routing features for IPV4 and IPV6 (OSPF, VRRP, PBR, RIP) require a premium. Note this includes IPV6 versions of these protocols as well. If you want BGP, GRE, IPV6 Tunneling or VRP you need an advanced license (which includes the premium features).

For a 6450 (Ruckus ICX 6430 and 6450 Switch Data Sheet | Technical Documents | Ruckus Wireless Support):
Without a license 2 of the 4 front SFP+ ports are limited to 1Gb only. A PoD license enables 2 ports at a time, so to be fully licensed you only need a single PoD license.

Basic layer 3 (static routes, ACLs) for IPV4 and IPV6 are included in the base license. More advanced IPV4 features like OSPF, VRRP, and GRE require the premium license to unlock, no IPV6 dynamic features. No BGP or IPV6 tunneling support on this model either.

 

[fohdeesha]

as stated 9.2E-5 times, licenses are free for sth members

 

[fohdeesha]

Just got one of these bad boy 6610s. Looks like it already has the licenses installed so I want to back them up before reseting. What do I need to do that? Apologies if this is answered above I tried searching for the answer and didn't come up with results.

you can't back them up or export them (at least not without a JTAG unit). If you just run "erase startup-config" from the enable level that will wipe the config and keep the licenses