Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

rootwyrm

New Member
Mar 25, 2017
11
1
3
www.rootwyrm.com
So, I have a conundrum... stuck looking at the 7450 and similar because I rather want BGP (I'd prefer ISIS but Brocade doesn't even do that on the VDX.)

But the noise level is nowhere near acceptable for something that has to be 10 feet from my head, and can't go in the rack. The 6450 with fan modification is right about the limit of what I would consider acceptable (with some additional baffle.) Anyone had any luck getting the BGP capable models down to about that noise level? Failing that, any suggestions for alternatives when I don't mind spending more if I have to?
 
  • Like
Reactions: dashpuppy

dashpuppy

Member
Dec 16, 2018
31
4
8
So, I have a conundrum... stuck looking at the 7450 and similar because I rather want BGP (I'd prefer ISIS but Brocade doesn't even do that on the VDX.)

But the noise level is nowhere near acceptable for something that has to be 10 feet from my head, and can't go in the rack. The 6450 with fan modification is right about the limit of what I would consider acceptable (with some additional baffle.) Anyone had any luck getting the BGP capable models down to about that noise level? Failing that, any suggestions for alternatives when I don't mind spending more if I have to?
Why not change the fans ? Or put resistors on the stock fans to slow them down ?
 

rootwyrm

New Member
Mar 25, 2017
11
1
3
www.rootwyrm.com
Why not change the fans ? Or put resistors on the stock fans to slow them down ?
Because there isn't thermal margin on the 7450 and the like to just swap the fans, and the primary issue is the PSU fans. Someone is welcome to do that on their ICX or VDX, so long as they post the literal melted PCBs and PSUs that result. There is no thermal margin on stuff like the 6610 and 7450 and above.

The only way you're doing it is one of two ways:
1. Chuck the 1U form factor in the bin (which is fine since this is on a workbench, but now has you up against the firmware expected RPM issue)
2. Software mod a lower model to do BGP (which is not a small task since RFC2835/5925 needs kernel support)
 

dashpuppy

Member
Dec 16, 2018
31
4
8
Because there isn't thermal margin on the 7450 and the like to just swap the fans, and the primary issue is the PSU fans. Someone is welcome to do that on their ICX or VDX, so long as they post the literal melted PCBs and PSUs that result. There is no thermal margin on stuff like the 6610 and 7450 and above.
Pretty sure 100% NOTHING is going to melt if you change the fans and lower their speed. The switch doesn't pull KW's of power.
 

vangoose

Active Member
May 21, 2019
223
52
28
Canada
So, I have a conundrum... stuck looking at the 7450 and similar because I rather want BGP (I'd prefer ISIS but Brocade doesn't even do that on the VDX.)

But the noise level is nowhere near acceptable for something that has to be 10 feet from my head, and can't go in the rack. The 6450 with fan modification is right about the limit of what I would consider acceptable (with some additional baffle.) Anyone had any luck getting the BGP capable models down to about that noise level? Failing that, any suggestions for alternatives when I don't mind spending more if I have to?
I got a 6650 because I wanted bgp as well as 10Gb ports counts.

I think I gave up on it, will use 7250 stack for 10Gb ports, and use SRX as bgp router for kubernetes.
 

juey

Member
Oct 1, 2018
56
14
8
Germany
Because there isn't thermal margin on the 7450 and the like to just swap the fans, and the primary issue is the PSU fans. Someone is welcome to do that on their ICX or VDX, so long as they post the literal melted PCBs and PSUs that result. There is no thermal margin on stuff like the 6610 and 7450 and above.

The only way you're doing it is one of two ways:
1. Chuck the 1U form factor in the bin (which is fine since this is on a workbench, but now has you up against the firmware expected RPM issue)
2. Software mod a lower model to do BGP (which is not a small task since RFC2835/5925 needs kernel support)
Go for a nasty Linux Box with FRRouting.
 

juey

Member
Oct 1, 2018
56
14
8
Germany
Pretty sure 100% NOTHING is going to melt if you change the fans and lower their speed. The switch doesn't pull KW's of power.
The 6610 ASIC will melt...ive watched transceiver temperatures go up to 55-60 degree celsius and cpu up to 65-70, with stock cooling.
 

frogtech

Well-Known Member
Jan 4, 2016
1,324
206
63
32
@fohdeesha

while my switch is booting, this message comes up:

Starting kernel ...
bcmiproc-gmac 18023000.ethernet: can't find MAC address

Is that normal? The switch boots otherwise.

Also, when I run the command to license the 8x10g ports, I get a message re: an evaluation of 45 days. Does the latest firmware still utilize honor based licensing and will the ports be 10 gig after the 45 days? This is used in a lab so it's not in production or used to make money.
 
Last edited:

fohdeesha

Kaini Industries
Nov 20, 2016
1,857
1,660
113
29
fohdeesha.com
@fohdeesha

while my switch is booting, this message comes up:

Starting kernel ...
bcmiproc-gmac 18023000.ethernet: can't find MAC address

Is that normal? The switch boots otherwise.

Also, when I run the command to license the 8x10g ports, I get a message re: an evaluation of 45 days. Does the latest firmware still utilize honor based licensing and will the ports be 10 gig after the 45 days? This is used in a lab so it's not in production or used to make money.


its normal and yes theyre honor based you'll be fine
 

Wolfstar

Active Member
Nov 28, 2015
155
79
28
45
Pretty sure 100% NOTHING is going to melt if you change the fans and lower their speed. The switch doesn't pull KW's of power.
And I'm 100% sure you're wrong. I've actually seen photos my guys took of Brocade FCX648S-HPOE switches after they caught fire, and saw one that had burned out myself. The 6610s and the FCX line use extremely similar hardware. I can't say I've seen the same of the 6610s, but that's because we don't use them at work - we jumped straight to ICX7450s.

Now, having seen burned out Brocade switches before, you might wonder why I'm still a fan of these things. The reason is simple - They STILL WORKED. Yes, I had a total of four incidents, one of which involved two switches in the same stack, that resulted in literally scorched and charred circuit boards, but the switches were still operating. In these cases, they were PoE boards, but even so.

They've had their problems over the years, but being built fragile is not really one of them.
 

frogtech

Well-Known Member
Jan 4, 2016
1,324
206
63
32
Any idea what a typical idle temp is for the ICX7250? The air coming from the rear of the switch is cool-ish, not very warm, but the heatsink for the main ASIC is rather warm, not hot but not comfortable either if you set your hand on it for a couple of minutes. I am just wondering I guess what the operating temperature specification range is for it and if it's going to be naturally higher than your traditional desktop or server CPU.
 

eduncan911

Active Member
Jul 27, 2015
100
44
28
65
eduncan911.com
I've tried searching this thread, and browsing a dozen or so pages...

Can someone tell me if "PoE" is PoE+? E.g. 48V not for 24V like for Ubiquiti's newer AC Pro and AP AC HD?

These sound awesome... I just picked up my Mikrotik 24 port PoE+ + 4x 10Gbps cages, and was looking at their newer 10 Gbps 10GbaseT switch for $500.

But after finding these, I'm debating selling the NIB Mikrotik for a couple of these.

My only issue is my desktop, and two laptops, are already have 10GBaseT RJ45 10Gbps - so I need an inexpensive RJ45 10Gbps solution at my switch with VLANs.
 

Aestr

Well-Known Member
Oct 22, 2014
901
296
63
Los Angeles
I've tried searching this thread, and browsing a dozen or so pages...

Can someone tell me if "PoE" is PoE+? E.g. 48V not for 24V like for Ubiquiti's newer AC Pro and AP AC HD?

These sound awesome... I just picked up my Mikrotik 24 port PoE+ + 4x 10Gbps cages, and was looking at their newer 10 Gbps 10GbaseT switch for $500.

But after finding these, I'm debating selling the NIB Mikrotik for a couple of these.

My only issue is my desktop, and two laptops, are already have 10GBaseT RJ45 10Gbps - so I need an inexpensive RJ45 10Gbps solution at my switch with VLANs.
The PoE variants of both the 6450 and 6610 supply 802.3at Power over Ethernet Plus (PoE+). You're better off looking up data sheets for this type of info.

I am currently using Mikrotik S+RJ10 modules both a 6450 and 6610 without issues to convert SFP+ to 10Gbase-T. They are pretty cheap, and as long as you don't need too many you'll come out ahead.
 
  • Like
Reactions: eduncan911

CED6688

New Member
Dec 4, 2019
15
9
3
So, I have a conundrum... stuck looking at the 7450 and similar because I rather want BGP (I'd prefer ISIS but Brocade doesn't even do that on the VDX.)

But the noise level is nowhere near acceptable for something that has to be 10 feet from my head, and can't go in the rack. The 6450 with fan modification is right about the limit of what I would consider acceptable (with some additional baffle.) Anyone had any luck getting the BGP capable models down to about that noise level? Failing that, any suggestions for alternatives when I don't mind spending more if I have to?
I wanted BGP to pair my physical network with a Kubernetes cluster running Calico networking (no IPIP, VXLAN, or NAT required). On my current 6610, not only did I get this working in about 15 minutes, but I was able to redistribute the routes to other parts of my network via OSPF. Basically all of your cluster IPs are part of your normal infrastructure, allowing you to set policy both within the cluster and at the network layer. It's pretty cool.

When I drop this to a 6450, I've decided to just do router-on-a-stick with a Mikrotik RB4011, which on its single SFP+ port can route 10gb/s on that one port (I've tested it) and supports BGP. I'll then have it redistribute the routes via OSPF to the rest of the network, which the 6450 will manage. All of my nodes have a separate interface to my storage VLAN, so that won't hit the Mikrotik, so the Mikrotik will really only be routing traffic that enters/leaves the cluster...and 10gb/s should be plenty enough for this for me.

That said, BGP support in the 7250 or 6450 would have eliminated a device and some complexity. The 6610 is incredible and is less than 40db from the next room, but it draws too much attention from the wife when she enters that room (~55db), so this is my current plan... though there is a part of me thinking about trying to swap in a rev b power supply. It would be good if someone can post noise measurement of a 6610 operating with one of those... I feel like I'm just at the borderline of acceptability. :)

Clayton
 

vangoose

Active Member
May 21, 2019
223
52
28
Canada
Any idea what a typical idle temp is for the ICX7250? The air coming from the rear of the switch is cool-ish, not very warm, but the heatsink for the main ASIC is rather warm, not hot but not comfortable either if you set your hand on it for a couple of minutes. I am just wondering I guess what the operating temperature specification range is for it and if it's going to be naturally higher than your traditional desktop or server CPU.
This is mine at idle.

The stack unit 1 chassis info:

Power supply 1 (AC - Regular) present, status ok
Power supply 2 not present

Fan ok, speed (auto): [[1]]<->2

Fan controlled temperature: 75.6 deg-C

Fan speed switching temperature thresholds:
Speed 1: NM<----->98 deg-C
Speed 2: 67<----->105 deg-C (shutdown)

Fan 1 Air Flow Direction: Front to Back
Slot 1 Current Temperature: 75.6 deg-C (Sensor 1)
Slot 2 Current Temperature: NA
Warning level.......: 100.0 deg-C
Shutdown level......: 105.0 deg-C
 
  • Like
Reactions: frogtech

vangoose

Active Member
May 21, 2019
223
52
28
Canada
I wanted BGP to pair my physical network with a Kubernetes cluster running Calico networking (no IPIP, VXLAN, or NAT required). On my current 6610, not only did I get this working in about 15 minutes, but I was able to redistribute the routes to other parts of my network via OSPF. Basically all of your cluster IPs are part of your normal infrastructure, allowing you to set policy both within the cluster and at the network layer. It's pretty cool.

When I drop this to a 6450, I've decided to just do router-on-a-stick with a Mikrotik RB4011, which on its single SFP+ port can route 10gb/s on that one port (I've tested it) and supports BGP. I'll then have it redistribute the routes via OSPF to the rest of the network, which the 6450 will manage. All of my nodes have a separate interface to my storage VLAN, so that won't hit the Mikrotik, so the Mikrotik will really only be routing traffic that enters/leaves the cluster...and 10gb/s should be plenty enough for this for me.

That said, BGP support in the 7250 or 6450 would have eliminated a device and some complexity. The 6610 is incredible and is less than 40db from the next room, but it draws too much attention from the wife when she enters that room (~55db), so this is my current plan... though there is a part of me thinking about trying to swap in a rev b power supply. It would be good if someone can post noise measurement of a 6610 operating with one of those... I feel like I'm just at the borderline of acceptability. :)

Clayton
Exact same reason I got a 6650. My 6650 comes with redundant rev B PSU, they are loud even in a separate room.

I gave up now until I can put a AC unit int the room and close the door.

I'll configure my SRX300 or F5 to be BGP router for kubernetes.
 

juey

Member
Oct 1, 2018
56
14
8
Germany
I've tried searching this thread, and browsing a dozen or so pages...

Can someone tell me if "PoE" is PoE+? E.g. 48V not for 24V like for Ubiquiti's newer AC Pro and AP AC HD?

These sound awesome... I just picked up my Mikrotik 24 port PoE+ + 4x 10Gbps cages, and was looking at their newer 10 Gbps 10GbaseT switch for $500.

But after finding these, I'm debating selling the NIB Mikrotik for a couple of these.

My only issue is my desktop, and two laptops, are already have 10GBaseT RJ45 10Gbps - so I need an inexpensive RJ45 10Gbps solution at my switch with VLANs.
As said, it supports both standards. Ive been running a Unifi AP-HD attached to a 6450-24P, so it works with Unifi Gear.
 
  • Like
Reactions: eduncan911