Drag to reposition cover

Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

vangoose

Active Member
May 21, 2019
326
104
43
Canada
So, I have a conundrum... stuck looking at the 7450 and similar because I rather want BGP (I'd prefer ISIS but Brocade doesn't even do that on the VDX.)

But the noise level is nowhere near acceptable for something that has to be 10 feet from my head, and can't go in the rack. The 6450 with fan modification is right about the limit of what I would consider acceptable (with some additional baffle.) Anyone had any luck getting the BGP capable models down to about that noise level? Failing that, any suggestions for alternatives when I don't mind spending more if I have to?
I got a 6650 because I wanted bgp as well as 10Gb ports counts.

I think I gave up on it, will use 7250 stack for 10Gb ports, and use SRX as bgp router for kubernetes.
 

juey

Member
Oct 1, 2018
56
16
8
Germany
Because there isn't thermal margin on the 7450 and the like to just swap the fans, and the primary issue is the PSU fans. Someone is welcome to do that on their ICX or VDX, so long as they post the literal melted PCBs and PSUs that result. There is no thermal margin on stuff like the 6610 and 7450 and above.

The only way you're doing it is one of two ways:
1. Chuck the 1U form factor in the bin (which is fine since this is on a workbench, but now has you up against the firmware expected RPM issue)
2. Software mod a lower model to do BGP (which is not a small task since RFC2835/5925 needs kernel support)
Go for a nasty Linux Box with FRRouting.
 

juey

Member
Oct 1, 2018
56
16
8
Germany
Pretty sure 100% NOTHING is going to melt if you change the fans and lower their speed. The switch doesn't pull KW's of power.
The 6610 ASIC will melt...ive watched transceiver temperatures go up to 55-60 degree celsius and cpu up to 65-70, with stock cooling.
 

frogtech

Well-Known Member
Jan 4, 2016
1,482
272
83
35
@fohdeesha

while my switch is booting, this message comes up:

Starting kernel ...
bcmiproc-gmac 18023000.ethernet: can't find MAC address

Is that normal? The switch boots otherwise.

Also, when I run the command to license the 8x10g ports, I get a message re: an evaluation of 45 days. Does the latest firmware still utilize honor based licensing and will the ports be 10 gig after the 45 days? This is used in a lab so it's not in production or used to make money.
 
Last edited:

fohdeesha

Kaini Industries
Nov 20, 2016
2,780
3,148
113
33
fohdeesha.com
@fohdeesha

while my switch is booting, this message comes up:

Starting kernel ...
bcmiproc-gmac 18023000.ethernet: can't find MAC address

Is that normal? The switch boots otherwise.

Also, when I run the command to license the 8x10g ports, I get a message re: an evaluation of 45 days. Does the latest firmware still utilize honor based licensing and will the ports be 10 gig after the 45 days? This is used in a lab so it's not in production or used to make money.


its normal and yes theyre honor based you'll be fine
 

Wolfstar

Active Member
Nov 28, 2015
159
84
28
49
Pretty sure 100% NOTHING is going to melt if you change the fans and lower their speed. The switch doesn't pull KW's of power.
And I'm 100% sure you're wrong. I've actually seen photos my guys took of Brocade FCX648S-HPOE switches after they caught fire, and saw one that had burned out myself. The 6610s and the FCX line use extremely similar hardware. I can't say I've seen the same of the 6610s, but that's because we don't use them at work - we jumped straight to ICX7450s.

Now, having seen burned out Brocade switches before, you might wonder why I'm still a fan of these things. The reason is simple - They STILL WORKED. Yes, I had a total of four incidents, one of which involved two switches in the same stack, that resulted in literally scorched and charred circuit boards, but the switches were still operating. In these cases, they were PoE boards, but even so.

They've had their problems over the years, but being built fragile is not really one of them.
 

frogtech

Well-Known Member
Jan 4, 2016
1,482
272
83
35
Any idea what a typical idle temp is for the ICX7250? The air coming from the rear of the switch is cool-ish, not very warm, but the heatsink for the main ASIC is rather warm, not hot but not comfortable either if you set your hand on it for a couple of minutes. I am just wondering I guess what the operating temperature specification range is for it and if it's going to be naturally higher than your traditional desktop or server CPU.
 

eduncan911

The New James Dean
Jul 27, 2015
648
506
93
eduncan911.com
I've tried searching this thread, and browsing a dozen or so pages...

Can someone tell me if "PoE" is PoE+? E.g. 48V not for 24V like for Ubiquiti's newer AC Pro and AP AC HD?

These sound awesome... I just picked up my Mikrotik 24 port PoE+ + 4x 10Gbps cages, and was looking at their newer 10 Gbps 10GbaseT switch for $500.

But after finding these, I'm debating selling the NIB Mikrotik for a couple of these.

My only issue is my desktop, and two laptops, are already have 10GBaseT RJ45 10Gbps - so I need an inexpensive RJ45 10Gbps solution at my switch with VLANs.
 

Aestr

Well-Known Member
Oct 22, 2014
967
388
63
Seattle
I've tried searching this thread, and browsing a dozen or so pages...

Can someone tell me if "PoE" is PoE+? E.g. 48V not for 24V like for Ubiquiti's newer AC Pro and AP AC HD?

These sound awesome... I just picked up my Mikrotik 24 port PoE+ + 4x 10Gbps cages, and was looking at their newer 10 Gbps 10GbaseT switch for $500.

But after finding these, I'm debating selling the NIB Mikrotik for a couple of these.

My only issue is my desktop, and two laptops, are already have 10GBaseT RJ45 10Gbps - so I need an inexpensive RJ45 10Gbps solution at my switch with VLANs.
The PoE variants of both the 6450 and 6610 supply 802.3at Power over Ethernet Plus (PoE+). You're better off looking up data sheets for this type of info.

I am currently using Mikrotik S+RJ10 modules both a 6450 and 6610 without issues to convert SFP+ to 10Gbase-T. They are pretty cheap, and as long as you don't need too many you'll come out ahead.
 
  • Like
Reactions: eduncan911

CED6688

New Member
Dec 4, 2019
15
10
3
So, I have a conundrum... stuck looking at the 7450 and similar because I rather want BGP (I'd prefer ISIS but Brocade doesn't even do that on the VDX.)

But the noise level is nowhere near acceptable for something that has to be 10 feet from my head, and can't go in the rack. The 6450 with fan modification is right about the limit of what I would consider acceptable (with some additional baffle.) Anyone had any luck getting the BGP capable models down to about that noise level? Failing that, any suggestions for alternatives when I don't mind spending more if I have to?
I wanted BGP to pair my physical network with a Kubernetes cluster running Calico networking (no IPIP, VXLAN, or NAT required). On my current 6610, not only did I get this working in about 15 minutes, but I was able to redistribute the routes to other parts of my network via OSPF. Basically all of your cluster IPs are part of your normal infrastructure, allowing you to set policy both within the cluster and at the network layer. It's pretty cool.

When I drop this to a 6450, I've decided to just do router-on-a-stick with a Mikrotik RB4011, which on its single SFP+ port can route 10gb/s on that one port (I've tested it) and supports BGP. I'll then have it redistribute the routes via OSPF to the rest of the network, which the 6450 will manage. All of my nodes have a separate interface to my storage VLAN, so that won't hit the Mikrotik, so the Mikrotik will really only be routing traffic that enters/leaves the cluster...and 10gb/s should be plenty enough for this for me.

That said, BGP support in the 7250 or 6450 would have eliminated a device and some complexity. The 6610 is incredible and is less than 40db from the next room, but it draws too much attention from the wife when she enters that room (~55db), so this is my current plan... though there is a part of me thinking about trying to swap in a rev b power supply. It would be good if someone can post noise measurement of a 6610 operating with one of those... I feel like I'm just at the borderline of acceptability. :)

Clayton
 

vangoose

Active Member
May 21, 2019
326
104
43
Canada
Any idea what a typical idle temp is for the ICX7250? The air coming from the rear of the switch is cool-ish, not very warm, but the heatsink for the main ASIC is rather warm, not hot but not comfortable either if you set your hand on it for a couple of minutes. I am just wondering I guess what the operating temperature specification range is for it and if it's going to be naturally higher than your traditional desktop or server CPU.
This is mine at idle.

The stack unit 1 chassis info:

Power supply 1 (AC - Regular) present, status ok
Power supply 2 not present

Fan ok, speed (auto): [[1]]<->2

Fan controlled temperature: 75.6 deg-C

Fan speed switching temperature thresholds:
Speed 1: NM<----->98 deg-C
Speed 2: 67<----->105 deg-C (shutdown)

Fan 1 Air Flow Direction: Front to Back
Slot 1 Current Temperature: 75.6 deg-C (Sensor 1)
Slot 2 Current Temperature: NA
Warning level.......: 100.0 deg-C
Shutdown level......: 105.0 deg-C
 
  • Like
Reactions: frogtech

vangoose

Active Member
May 21, 2019
326
104
43
Canada
I wanted BGP to pair my physical network with a Kubernetes cluster running Calico networking (no IPIP, VXLAN, or NAT required). On my current 6610, not only did I get this working in about 15 minutes, but I was able to redistribute the routes to other parts of my network via OSPF. Basically all of your cluster IPs are part of your normal infrastructure, allowing you to set policy both within the cluster and at the network layer. It's pretty cool.

When I drop this to a 6450, I've decided to just do router-on-a-stick with a Mikrotik RB4011, which on its single SFP+ port can route 10gb/s on that one port (I've tested it) and supports BGP. I'll then have it redistribute the routes via OSPF to the rest of the network, which the 6450 will manage. All of my nodes have a separate interface to my storage VLAN, so that won't hit the Mikrotik, so the Mikrotik will really only be routing traffic that enters/leaves the cluster...and 10gb/s should be plenty enough for this for me.

That said, BGP support in the 7250 or 6450 would have eliminated a device and some complexity. The 6610 is incredible and is less than 40db from the next room, but it draws too much attention from the wife when she enters that room (~55db), so this is my current plan... though there is a part of me thinking about trying to swap in a rev b power supply. It would be good if someone can post noise measurement of a 6610 operating with one of those... I feel like I'm just at the borderline of acceptability. :)

Clayton
Exact same reason I got a 6650. My 6650 comes with redundant rev B PSU, they are loud even in a separate room.

I gave up now until I can put a AC unit int the room and close the door.

I'll configure my SRX300 or F5 to be BGP router for kubernetes.
 

juey

Member
Oct 1, 2018
56
16
8
Germany
I've tried searching this thread, and browsing a dozen or so pages...

Can someone tell me if "PoE" is PoE+? E.g. 48V not for 24V like for Ubiquiti's newer AC Pro and AP AC HD?

These sound awesome... I just picked up my Mikrotik 24 port PoE+ + 4x 10Gbps cages, and was looking at their newer 10 Gbps 10GbaseT switch for $500.

But after finding these, I'm debating selling the NIB Mikrotik for a couple of these.

My only issue is my desktop, and two laptops, are already have 10GBaseT RJ45 10Gbps - so I need an inexpensive RJ45 10Gbps solution at my switch with VLANs.
As said, it supports both standards. Ive been running a Unifi AP-HD attached to a 6450-24P, so it works with Unifi Gear.
 
  • Like
Reactions: eduncan911

Wolfstar

Active Member
Nov 28, 2015
159
84
28
49
The PoE variants of both the 6450 and 6610 supply 802.3at Power over Ethernet Plus (PoE+). You're better off looking up data sheets for this type of info.

I am currently using Mikrotik S+RJ10 modules both a 6450 and 6610 without issues to convert SFP+ to 10Gbase-T. They are pretty cheap, and as long as you don't need too many you'll come out ahead.
Just to clarify this. It supports standardized IEEE 802.3af and 802.3at Power over Ethernet specifications. It does NOT support the passive, non-standardized 24V PoE that Ubiquiti used early on for the UAP-AC-Lite and UAP-AC-LR (and I believe some of their cameras too). If you have an old UAP-AC-LITE or -LR, you may not be able to power off of these switches.

However, there are converter modules you can get and you can always use a power injector. If you got one of those any time in the last year or so, you're probably okay. Check around on the Ubiquiti forums and they may be able to get you a date code or serial number checker that will tell you what it supports.
 
  • Like
Reactions: eduncan911

csementuh

Member
Oct 7, 2019
36
10
8
Pittsburgh, PA
Awesome thread, many thanks to fohdeesha! Outstanding docs and info here and his website.

Bought a 6450-48P from eBay a couple of months back and stuck it top-of-my-home-rack. Was working great until I discovered that the POE was only intermittently working (and I was out of return policy). I'll be posting this 6450 for sale as it works 100% without the POE, essentially a 6450-48.

Today I bought a 7250-48P and looking forward to getting it in! The ease of licensing and the overkill on 10G ports will be nice. I have about 36 drops in my house now, and will run fiber 10G to my detached garage at some point. At these prices and features, why not!? I moved 8 POE IP cams over from the NVR directly to the switch to help admin and VLAN them and this switch will work well for it all.
 
  • Like
Reactions: fohdeesha