Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[neonclash]

Does anyone near Indianapolis want to buy any switches? I have so many I need to offload but I don't have the time or energy to deal with shipping. ICX6610s, ICX7250s, ICX6450s, Dell S4810P, Brocade ADX, brocade CES, probably more I'm forgetting

A relative of mine returns to the Indianapolis area this evening, so this is perfect timing. I just can't settle on what hardware would fit my use case the best, so recommendations would be welcome.

The eventual goal for my home setup is reliability and mandatory access control on edge devices to assigned VLANs using 802.1x. Most of the 10gbit (three servers and growing) stays in the rack, so I need a stack there with PoE for my pi hats and backup AP. I also have an office with a PoE AP, and need 10gbit to a workstation there. The Z-series line would be the best fit because of the 2.5Gbase-T that my APs can take advantage of, but they're impossible to find.

Right now I have a 7150-c12p in the office and a 7150-24P in the closet. It's 10gbit from end to end, and I have it working the way I want to with VLANs and even a guest PVLAN. I just need to scale out soon, preferably while adding the redundancy I am after.

The 7250s are still maintained and use less power, but that MACSEC on the 6610 is nice, and I've never played with OpenFlow before, not to mention they're cheaper. What do I lose out on that I might not expect, and what do you recommend?

 

[fohdeesha]

A relative of mine returns to the Indianapolis area this evening, so this is perfect timing. I just can't settle on what hardware would fit my use case the best, so recommendations would be welcome.

The eventual goal for my home setup is reliability and mandatory access control on edge devices to assigned VLANs using 802.1x. Most of the 10gbit (three servers and growing) stays in the rack, so I need a stack there with PoE for my pi hats and backup AP. I also have an office with a PoE AP, and need 10gbit to a workstation there. The Z-series line would be the best fit because of the 2.5Gbase-T that my APs can take advantage of, but they're impossible to find.

Right now I have a 7150-c12p in the office and a 7150-24P in the closet. It's 10gbit from end to end, and I have it working the way I want to with VLANs and even a guest PVLAN. I just need to scale out soon, preferably while adding the redundancy I am after.

The 7250s are still maintained and use less power, but that MACSEC on the 6610 is nice, and I've never played with OpenFlow before, not to mention they're cheaper. What do I lose out on that I might not expect, and what do you recommend?

I don't think you'd lose out on anything going with the 7250 over the 6610 - MACSEC and openflow don't really have real applications in a setup like that - plus with the 7 series you get newer software/features. Just looked and this is what I have that I'm willing to part with (no PoE 7250s left sadly):

ICX7250-48 X2
ICX7250-24 X1

ICX6610-48P X3
ICX6610-48 X2

ICX6450-24 X2
ICX6450-48P X1

FCX648S X4
FCX648S-HPOE X1

Juniper EX4300 X1
Dell S4810P X1
CES-2024C X1

 

[neonclash]

I don't think you'd lose out on anything going with the 7250 over the 6610 - MACSEC and openflow don't really have real applications in a setup like that - plus with the 7 series you get newer software/features. Just looked and this is what I have that I'm willing to part with (no PoE 7250s left sadly):

ICX7250-48 X2
ICX7250-24 X1

ICX6610-48P X3
ICX6610-48 X2

ICX6450-24 X2
ICX6450-48P X1

FCX648S X4
FCX648S-HPOE X1

Juniper EX4300 X1
Dell S4810P X1
CES-2024C X1

Thanks for the response! I'm gonna poke some ebay listings to see if I can get a couple of 7250-xxPs first. In the meantime, what's your asking price for each 6610-48P?

 

[NablaSquaredG]

Is there any way to *lower* the temperature where the fans spin up on ICX6610?

I'm currently using them in a quite warm environment and the fans spin up every ~30-60s and then spin down again... spin up again... spin down...

I'd like to have them permanently running on level 2...

 

[fohdeesha]

Thanks for the response! I'm gonna poke some ebay listings to see if I can get a couple of 7250-xxPs first. In the meantime, what's your asking price for each 6610-48P?

cheapest I see on ebay for 48Ps with dual PSUs is 190, I could do 160? Obviously no shipping fee with the local pickup, and they've been graced by the brocade loser himself

 

[fohdeesha]

OK as of now all my 7250's are spoken for

Edit: wow, how is this thread about to hit 300 pages? Wild - I suppose now I'll really have to publish the new guide including lil surprise within the next page

 

[i386]

lil surprise within the next page

You found switches for 2.5, 5, 10, 25, 50 & 100 gbit ethernet that are as awesome as the icx in the op?

 

[Rootless]

I can't wait for the surprise. I love this thread. I now have two ICX-6610, one of the Dell Networking Cable QSFP+ 40GBE - 4X SFP+ 10GBE 3M 27GG5 cables. All of the licenses are good and both units are online. The Dell cable works fine between the two ICX6610, but I'm having problems getting it working with the " Cavium QLogic 1010/1020/1007/1741 10Gbps CNA (rev 01)" installed in my Dell R910.

The card is visible (lspci):
05:00.2 Ethernet controller: Cavium QLogic 1010/1020/1007/1741 10Gbps CNA (rev 01)
05:00.3 Ethernet controller: Cavium QLogic 1010/1020/1007/1741 10Gbps CNA (rev 01)

But the links always show as down both on the server:
6: ens5f2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc fq_codel state DOWN mode DEFAULT group default qlen 1000
link/ether 8c:7c:ff:20:c0:32 brd ff:ff:ff:ff:ff:ff
altname enp5s0f2
7: ens5f3: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc fq_codel state DOWN mode DEFAULT group default qlen 1000
link/ether 8c:7c:ff:20:c0:33 brd ff:ff:ff:ff:ff:ff
altname enp5s0f3

and on the switch:
Port Link State Dupl Speed Trunk Tag Pvid Pri MAC Name
1/2/7 Down None None None None No 1 0 cc4e.2484.0568
1/2/8 Down None None None None No 1 0 cc4e.2484.0568
1/2/9 Down None None None None No 1 0 cc4e.2484.0568

(the one connected between the switches at the moment is 1/2/10:
SSH@racksw1>sh int br e 1/2/10
Port Link State Dupl Speed Trunk Tag Pvid Pri MAC Name
1/2/10 Up Forward Full 10G None No 1 0 cc4e.2484.0568
)

The Dell breakout DAC
SSH@racksw1>sh media e 1/2/7
Port 1/2/7:Type : 40GBASE-Passive Copper
Vendor Name: Amphenol Serial Num: CN027GG554Q4JWR Revision: C

All of the QSFP ports are removed from stacking and stacking is disabled:
SSH@racksw1>sh conf
!
Startup-config data location is flash memory
!
Startup configuration:
!
ver 08.0.30tT7f3
!
stack unit 1
module 1 icx6610-48p-poe-port-management-module
module 2 icx6610-qsfp-10-port-160g-module
module 3 icx6610-8-port-10g-dual-mode-module
stack disable
stack mac cc4e.2484.0568
!
!
vlan 1 name DEFAULT-VLAN by port
router-interface ve 1
!
!
aaa authentication web-server default local
aaa authentication enable default local
aaa authentication login default local
hostname racksw1
ip dhcp-client disable
ip dns server-address 192.168.1.1
ip route 0.0.0.0/0 192.168.1.1
!
no telnet server
username root password .....
no snmp-server
!
!
clock timezone gmt GMT-07
!
!
ntp
disable serve
server 216.239.35.0
server 216.239.35.4
!
web-management https
hitless-failover enable
!
!
interface ethernet 1/3/1
speed-duplex 10G-full
!
interface ethernet 1/3/2
speed-duplex 10G-full
!
interface ethernet 1/3/3
speed-duplex 10G-full
!
interface ethernet 1/3/4
speed-duplex 10G-full
!
interface ethernet 1/3/5
speed-duplex 10G-full
!
interface ethernet 1/3/6
speed-duplex 10G-full
!
interface ethernet 1/3/7
speed-duplex 10G-full
!
interface ethernet 1/3/8
speed-duplex 10G-full
!
interface ve 1
ip address 192.168.1.58 255.255.255.0
!
!
end

I picked up the Brocade 1020 NIC because I read in a forum post that they just work. I have since found another post that describes them as finicky. Not sure where to go from here. Do I need to pickup something like a MCX354A-QCBT Mellanox CX354A ConnectX-3 VPI QDR IB 10GbE Dual-Port QSFP NIC? If so, why do some of the MCX345A-QCBT list themselves as 10Gb/40Gb, some as 10Gb only, and some as 40Gb only?

Thank you for all of the incredible documentation and information sharing. It is what got me this far. Thank you.

 

[ccie4526]

Pushing for page 300. Wish I could find a 7250-24P for something less than stupid prices.

 

[LodeRunner]

Pushing for page 300. Wish I could find a 7250-24P for something less than stupid prices.

The 48P is almost always less than the 24P. Idle power usage should be on par; unused ports don't draw much.

 

[neonclash]

cheapest I see on ebay for 48Ps with dual PSUs is 190, I could do 160? Obviously no shipping fee with the local pickup, and they've been graced by the brocade loser himself

The US sellers for 24 and 48 port 7250s with POE were all asking stupidly high prices and didn't take any sane offers, but I did find a Canadian seller willing to send me two 7250-48Ps for $300CAD each, for a total of ~$550 USD shipped.

 

[fohdeesha]

You found switches for 2.5, 5, 10, 25, 50 & 100 gbit ethernet that are as awesome as the icx in the op?

damn now the surprise is going to seem really lame in comparison it's a form of info a lot of people in this thread already have, hint hint. also shit we're on 300 everyone uhh take a vacation real quick while I finish this guide

 

[Jason Antes]

I don't think you'd lose out on anything going with the 7250 over the 6610 - MACSEC and openflow don't really have real applications in a setup like that - plus with the 7 series you get newer software/features. Just looked and this is what I have that I'm willing to part with (no PoE 7250s left sadly):

ICX7250-48 X2
ICX7250-24 X1

ICX6610-48P X3
ICX6610-48 X2

ICX6450-24 X2
ICX6450-48P X1

FCX648S X4
FCX648S-HPOE X1

Juniper EX4300 X1
Dell S4810P X1
CES-2024C X1

Nice. Too bad I live to far away to pick one up. Need another POE switch to put in my pole barn once I get it built (who knows when that will be...depends on lumber prices) that I can uplink to my 6610 via the 40GbE optic SFP+'s I have. I know I can with the VDX, just doesn't have PoE. I'm sure these will sell out quick!

 

[blinkenlights]

I have been testing the 09000 release on my personal 7450s. Like @fohdeesha said, the new web GUI is the most visible change. Did not run into any noticeable bugs but I am planning to return my gear to the 0809x codetrain for now.

Earlier today, I stumbled across this notice regarding 08090m:

2 September 2021: RUCKUS has found a defect # FI-247944 due to which we have removed the software image for 08.0.90m from the Support Portal. For more information, please read TSB-2021-005.

I do not have access to the detailed TSB, but it is unusual for a vendor to completely pull a GA release before the patch is available. I would consider downgrading if you deployed the 'm' firmware.

 

[DavidB]

damn now the surprise is going to seem really lame in comparison it's a form of info a lot of people in this thread already have, hint hint. also shit we're on 300 everyone uhh take a vacation real quick while I finish this guide

I hope it's that now the ICX 7250 and ICX 7450 are the best value as I just picked them up for cheap NEW IN BOX on Ebay, after you change your OP to recommend those it will be months until they're cheap again (if ever). Currently flashing them with the help of your guides, keep up the awesome work

 

[nickf1227]

damn now the surprise is going to seem really lame in comparison it's a form of info a lot of people in this thread already have, hint hint. also shit we're on 300 everyone uhh take a vacation real quick while I finish this guide

So what you're saying is all those recent commits on github were for a reason??

 

[Dave Corder]

Random question about using a QSFP-4xSFP+ breakout cable on a 6610: Are the SFP+ ends SFP+/10Gbps only, or could I run one of them into a SFP-only switch at 1Gbps? (My guess is "no".)

Background: I stupidly put my router and core network switch (an ICX6610-48P) in my main rack when I started running Cat6 in my house after we bought it a couple years ago. As I've added more and more drops, this means there's a bigger and bigger bundle of Cat6 coming from my patch panel on the wall over to the rack to get to the core switch (and did I mention the rack is on wheels for easier maintenance access?). I'm in the process of moving the router and core switch over to a 4U vertical wall-mount bracket right under the patch panel to clean up my cable management mess. I've decided to just go with a long (5M) QSFP-4xSFP breakout DAC (a Dell P8T4W) from the core switch on the wall to the two servers that will end up staying in the rack. But there are also a handful of management interfaces in the rack (a couple iDRACs, a networked PDU, APC SNMP card, etc). So it'd be super convenient if I could run one of the extra SFP+ breakouts into an older SFP/gigabit-only switch for those management connections. But I have a feeling that's not actually possible, but was hoping someone with some first-hand knowledge could chime in. If it turns out I have to run a single Cat6 cable over for a small management network switch (or heck, just get a MikroTik CSS610-8G-2S+IN for a benjamin...), no biggie.

 

[fohdeesha]

I have been testing the 09000 release on my personal 7450s. Like @fohdeesha said, the new web GUI is the most visible change. Did not run into any noticeable bugs but I am planning to return my gear to the 0809x codetrain for now.

Earlier today, I stumbled across this notice regarding 08090m:

2 September 2021: RUCKUS has found a defect # FI-247944 due to which we have removed the software image for 08.0.90m from the Support Portal. For more information, please read TSB-2021-005.

I do not have access to the detailed TSB, but it is unusual for a vendor to completely pull a GA release before the patch is available. I would consider downgrading if you deployed the 'm' firmware.

It's just a management access issue in the layer 2 only images, the layer3 images that everyone here should be running are unaffected:

Random question about using a QSFP-4xSFP+ breakout cable on a 6610: Are the SFP+ ends SFP+/10Gbps only, or could I run one of them into a SFP-only switch at 1Gbps? (My guess is "no".)

Background: I stupidly put my router and core network switch (an ICX6610-48P) in my main rack when I started running Cat6 in my house after we bought it a couple years ago. As I've added more and more drops, this means there's a bigger and bigger bundle of Cat6 coming from my patch panel on the wall over to the rack to get to the core switch (and did I mention the rack is on wheels for easier maintenance access?). I'm in the process of moving the router and core switch over to a 4U vertical wall-mount bracket right under the patch panel to clean up my cable management mess. I've decided to just go with a long (5M) QSFP-4xSFP breakout DAC (a Dell P8T4W) from the core switch on the wall to the two servers that will end up staying in the rack. But there are also a handful of management interfaces in the rack (a couple iDRACs, a networked PDU, APC SNMP card, etc). So it'd be super convenient if I could run one of the extra SFP+ breakouts into an older SFP/gigabit-only switch for those management connections. But I have a feeling that's not actually possible, but was hoping someone with some first-hand knowledge could chime in. If it turns out I have to run a single Cat6 cable over for a small management network switch (or heck, just get a MikroTik CSS610-8G-2S+IN for a benjamin...), no biggie.

nope, sadly 10gbe lanes only

 

[atb]

So searching old posts brought me a lot of references for old posts about the 6xxx series, but what are my fan swap/removal options for an icx7250? It's considerably louder than I expected after spindown, I was hoping for it to be comparable to say a Cisco 3750X, or would like to get it as close to that as possible

 

[fohdeesha]

So searching old posts brought me a lot of references for old posts about the 6xxx series, but what are my fan swap/removal options for an icx7250? It's considerably louder than I expected after spindown, I was hoping for it to be comparable to say a Cisco 3750X, or would like to get it as close to that as possible

on the one I sold you, you can remove/unplug 2 of the 3 fans to start with, since it's no longer a PoE model (atleast, I think I pulled the PoE board)