Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[itronin]

What are the temps though?

I had heat issues and went to 2 fans, 2 psu, and heat issues went away. They're REv A.
In one I have a REv B but can't tell the difference... debating if I should swap that B to A for more air flow.
I have a 8-11*C difference between the 6610 on top vs on bottom too, unsure if it's location, A vs B, or due to the rack-ear not having the middle\vent hole.

My original switch purchase had 2 PSU and 2 FANS in each unit. I've only been running 1 switch due to noise and playing with the 40Gbe. If I can source 1 more rev B I will try and figure out if it is really possible to stack with a single 40Gbe and use the remaining 2 for a LAG.

I've marked a seller on the bay that has qty 4 rev B power supplies (confirmed via message), asking a lot for 'em but none of have moved since it was posted. May take a lower number though it will likely be about what I paid for these 2 combined. But my ears, oh my. huge huge difference. my CSE-417 is now the loudest thing in the rack and I removed some fans from it to quiet it down a bit (all flash, 2/3 full, temps are fine there).

I figured since I did not hear the fans ramp up (except at power on) that all must be okay and I was really just floored by the sound difference so I didn’t even check temps.

Original configs:
#1, ICX6610-48P-E, 1xRev A PSU-E , 1x Rev B PSU-E, 2x FAN-E
#2, ICX6610-48P-E, 2xRev A PSU-E, 2xFAN-E

New config ICX
#1 ICX6610-48P-E, 2x Rev B PSU-E, 2x FAN-E,
#2, ICX6610-48P-E, 1xRev A PSU-E , 1x Rev B PSU-E, 2x FAN-E

icx6610-stack#show chassis
The stack unit 1 chassis info:

Power supply 1 (AC - PoE) present, status ok
     Model Number:   23-0000142-02
   Serial Number:   B1S     
   Firmware Ver:     B
Power supply 1 Fan Air Flow Direction:  Front to Back
Power supply 2 (AC - PoE) present, status ok
     Model Number:   23-0000142-02
   Serial Number:   5WP     
   Firmware Ver:     B
Power supply 2 Fan Air Flow Direction:  Front to Back

Fan 1 ok, speed (auto): [[1]]<->2
Fan 2 ok, speed (auto): [[1]]<->2

Fan controlled temperature: 65.5 deg-C

Fan speed switching temperature thresholds:
       Speed 1: NM<----->76       deg-C
       Speed 2:       71<-----> 80 deg-C (shutdown)

Fan 1 Air Flow Direction:  Front to Back
Fan 2 Air Flow Direction:  Front to Back                         
MAC 1 Temperature Readings:
   Current temperature : 45.0 deg-C
MAC 2 Temperature Readings:
   Current temperature : 54.0 deg-C
CPU Temperature Readings:
   Current temperature : 65.5 deg-C
sensor A Temperature Readings:
   Current temperature : 52.0 deg-C
sensor B Temperature Readings:
   Current temperature : 50.5 deg-C
sensor C Temperature Readings:
   Current temperature : 37.0 deg-C
stacking card Temperature Readings:
   Current temperature : 64.0 deg-C
   Warning level.......: 77.0 deg-C
   Shutdown level......: 80.0 deg-C

 

[kapone]

While that switch above is not exactly cooking...it is certainly very very warm. I'm surprised the fans haven't ramped up.

 

[T_Minus]

While that switch above is not exactly cooking...it is certainly very very warm. I'm surprised the fans haven't ramped up.

Which? These just posted?

MAC 1 Temperature Readings:
Current temperature : 45.0 deg-C
MAC 2 Temperature Readings:
Current temperature : 54.0 deg-C
CPU Temperature Readings:
Current temperature : 65.5 deg-C
sensor A Temperature Readings:
Current temperature : 52.0 deg-C
sensor B Temperature Readings:
Current temperature : 50.5 deg-C
sensor C Temperature Readings:
Current temperature : 37.0 deg-C


Mine that wasn't working was running at 79 and 80*C until we fixed the fan, and air flow situation, now both are in the 40s and 50s *C and work great

 

[kapone]

Yeah, that one.

 

[ekke]

Bought a 6450-24p poe .
Get these poe syslog messages.


I also got a Sunon MagLev KDE1204PKV3-MS.AF.GN 3pin 18dB-A Super Silence from ebay, gotta remove that high pitch noise of the current fan.

 

[fohdeesha]

Bought a 6450-24p poe .
Get these poe syslog messages.
View attachment 12569

I also got a Sunon MagLev KDE1204PKV3-MS.AF.GN 3pin 18dB-A Super Silence from ebay, gotta remove that high pitch noise of the current fan.

follow the update and config guide in the first post

 

[fohdeesha]

My original switch purchase had 2 PSU and 2 FANS in each unit. I've only been running 1 switch due to noise and playing with the 40Gbe. If I can source 1 more rev B I will try and figure out if it is really possible to stack with a single 40Gbe and use the remaining 2 for a LAG.

I've marked a seller on the bay that has qty 4 rev B power supplies (confirmed via message), asking a lot for 'em but none of have moved since it was posted. May take a lower number though it will likely be about what I paid for these 2 combined. But my ears, oh my. huge huge difference. my CSE-417 is now the loudest thing in the rack and I removed some fans from it to quiet it down a bit (all flash, 2/3 full, temps are fine there).

I figured since I did not hear the fans ramp up (except at power on) that all must be okay and I was really just floored by the sound difference so I didn’t even check temps.

Original configs:
#1, ICX6610-48P-E, 1xRev A PSU-E , 1x Rev B PSU-E, 2x FAN-E
#2, ICX6610-48P-E, 2xRev A PSU-E, 2xFAN-E

New config ICX
#1 ICX6610-48P-E, 2x Rev B PSU-E, 2x FAN-E,
#2, ICX6610-48P-E, 1xRev A PSU-E , 1x Rev B PSU-E, 2x FAN-E

icx6610-stack#show chassis
The stack unit 1 chassis info:

Power supply 1 (AC - PoE) present, status ok
     Model Number:   23-0000142-02
   Serial Number:   B1S    
   Firmware Ver:     B
Power supply 1 Fan Air Flow Direction:  Front to Back
Power supply 2 (AC - PoE) present, status ok
     Model Number:   23-0000142-02
   Serial Number:   5WP    
   Firmware Ver:     B
Power supply 2 Fan Air Flow Direction:  Front to Back

Fan 1 ok, speed (auto): [[1]]<->2
Fan 2 ok, speed (auto): [[1]]<->2

Fan controlled temperature: 65.5 deg-C

Fan speed switching temperature thresholds:
       Speed 1: NM<----->76       deg-C
       Speed 2:       71<-----> 80 deg-C (shutdown)

Fan 1 Air Flow Direction:  Front to Back
Fan 2 Air Flow Direction:  Front to Back                        
MAC 1 Temperature Readings:
   Current temperature : 45.0 deg-C
MAC 2 Temperature Readings:
   Current temperature : 54.0 deg-C
CPU Temperature Readings:
   Current temperature : 65.5 deg-C
sensor A Temperature Readings:
   Current temperature : 52.0 deg-C
sensor B Temperature Readings:
   Current temperature : 50.5 deg-C
sensor C Temperature Readings:
   Current temperature : 37.0 deg-C
stacking card Temperature Readings:
   Current temperature : 64.0 deg-C
   Warning level.......: 77.0 deg-C
   Shutdown level......: 80.0 deg-C

it doesn't even bother to ramp up fans until 76c, 65 is more than fine. these things run hot (which is why I keep yelling at people to stop wasting their time trying to stick even slower fans in the 6610)

 

[itronin]

it doesn't even bother to ramp up fans until 76c, 65 is more than fine. these things run hot (which is why I keep yelling at people to stop wasting their time trying to stick even slower fans in the 6610)

yes, folks please listen to Uncle Fohdeesha. Just find some deals and invest in rev B psu's . I am still non-sonically stunned by how quiet two 6610 POE rev B's are. Fohdeesha, you posted about the acoustics early on in the thread - just hadn't found a deal until last week and it was SO worth it!

 

[Aaron Britton]

Fohdeesha, anymore licenses available? PM sent.

 

[dashpuppy]

Well I kinda had a decent weekend. First up was installing the new Fans into the 6450,, AHHH!! much silent ! And still passing air and keeping it cool.

Next was installing 10gig nic's into the Hp ML350 G8 server ( Fail ) they don't work

Ordered some PCI-e 10gig HP nic's instead of these Lenovo Emulex ones !!

Oh ! And i don't have an addiction !! So i bought 2 of these Brocade ICX 6430-C12's for 100$ One will be connected to my Brocade 6450 via Fiber then sitting behind my desk The other will be spare ? maybe...

 

[ekke]

forgot to mention: licenses for these models are free for existing STH members - now that they are EOL & discontinued I don't really have an alternative to giving them away

Hi , great guide btw ( FCX / ICX6610 - Fohdeesha Docs ) I couldn't find any info on how to abtain or active licence for 6450 24p for the 10GB ports.

 

[eduncan911]

EDIT: found the link to the published sources, though I haven't extracted it yet (on vacation/mobile). Reading the README for 6450 though indicates that there indeed binary blobs from Brocade/Arris to include. Humm... Makes sense not to include source to this libs as these most likely aren't under GPL. Will have to disassemble them and inspect further.

---

Has anyone successfully pulled down the source code and built the firmware? It was mentioned on the first page that under GPL licensing they are publishing, slowly.

For which device though? And would the process work for both the 6450 48p and 7250 48p ARM CPUs?

I'm on the fence for a 6450 or 7250 for their low noise and source code availability. I have four 10 Gbps 10GbaseT devices right now and technically could have a 5th, which would me swing towards the 7250 series.

However, building the firmware (with little to no proprietary binary blobs) is one thing that really interests me.

Maybe someone could point us to the source code links? I'll go back through the thread again and see if it was brought up...

 

[CED6688]

Ok. Now have 6610-24 (2x RevA PSUs), 6450-24, and 7250-24P that were purchased in that order.

Since there is a lot of discussion on noise:
- The 6450-24 with stock fans was so quiet (38-42db) that I was able to move it to a bedroom closet and nobody knows it is there w/ 2 10gb and a dozen 1gb connections. Everyone has different sound tolerance, but I had multiple people help listen for it and we literally could not hear it -- I suspect that people with louder fans may have bad fans. It nicely replaced both a Mikrotik CRS305 and an older Unifi switch and barely consumes any power. It's incredible for the price and I would highly recommend it for most people that only need a few 10gig ports for a NAS or such.

- The 7250-24P is louder than the 6450 (45-48db). I would say mid-way between my 6450 and 6610. Definitely louder than the 6450, but is at a wife-acceptable volume level with no modifications (stock fans/etc...) from the next room with the door open with 6 10gb and a dozen 1gb connections. Wouldn't put in the bedroom closet or want to sit next to it for extended periods, but would not classify it as loud. Love that the latest OS has more straight-forward handling of ports with both tagged/untagged VLANs, etc... but wouldn't base a buying decision based on that. This replaced a Mikrotik CRS309 and two Unifi switches.

- The 6610-24 with 2xRevA's was obviously louder (~53-58db depending on PSU fan speed) than both of the above with the same load as the 7250. Having just seen more of the RevB stories, I'm watching any keyword related to RPS-15-E/RSP-16-E PSUs on eBay. If the 7250 had BGP, I probably don't really need the QSFP ports and would just stick with the 7250. Would be curious if anyone has done any sound measurements of the actual drop from RevA to RevB's...also it seems that the drop comes with dual B's...would running the 24 with a single B PSU be materially quieter than one or two RevA's? I don't feel like I'm too far from acceptable sound levels and my other option is to start running fiber to the garage/attic or just give up.

 

[juey]

Ok. Now have 6610-24 (2x RevA PSUs), 6450-24, and 7250-24P that were purchased in that order.

Since there is a lot of discussion on noise:
- The 6450-24 with stock fans was so quiet (38-42db) that I was able to move it to a bedroom closet and nobody knows it is there w/ 2 10gb and a dozen 1gb connections. Everyone has different sound tolerance, but I had multiple people help listen for it and we literally could not hear it -- I suspect that people with louder fans may have bad fans. It nicely replaced both a Mikrotik CRS305 and an older Unifi switch and barely consumes any power. It's incredible for the price and I would highly recommend it for most people that only need a few 10gig ports for a NAS or such.

- The 7250-24P is louder than the 6450 (45-48db). I would say mid-way between my 6450 and 6610. Definitely louder than the 6450, but is at a wife-acceptable volume level with no modifications (stock fans/etc...) from the next room with the door open with 6 10gb and a dozen 1gb connections. Wouldn't put in the bedroom closet or want to sit next to it for extended periods, but would not classify it as loud. Love that the latest OS has more straight-forward handling of ports with both tagged/untagged VLANs, etc... but wouldn't base a buying decision based on that. This replaced a Mikrotik CRS309 and two Unifi switches.

- The 6610-24 with 2xRevA's was obviously louder (~53-58db depending on PSU fan speed) than both of the above with the same load as the 7250. Having just seen more of the RevB stories, I'm watching any keyword related to RPS-15-E/RSP-16-E PSUs on eBay. If the 7250 had BGP, I probably don't really need the QSFP ports and would just stick with the 7250. Would be curious if anyone has done any sound measurements of the actual drop from RevA to RevB's...also it seems that the drop comes with dual B's...would running the 24 with a single B PSU be materially quieter than one or two RevA's? I don't feel like I'm too far from acceptable sound levels and my other option is to start running fiber to the garage/attic or just give up.

You could create a mixed stack with the 6610 and the 6450, that further enables more features on the 6450 like openflow and possibly BGP ?

 

[dashpuppy]

Ok. Now have 6610-24 (2x RevA PSUs), 6450-24, and 7250-24P that were purchased in that order.

Since there is a lot of discussion on noise:
- The 6450-24 with stock fans was so quiet (38-42db) that I was able to move it to a bedroom closet and nobody knows it is there w/ 2 10gb and a dozen 1gb connections. Everyone has different sound tolerance, but I had multiple people help listen for it and we literally could not hear it -- I suspect that people with louder fans may have bad fans. It nicely replaced both a Mikrotik CRS305 and an older Unifi switch and barely consumes any power. It's incredible for the price and I would highly recommend it for most people that only need a few 10gig ports for a NAS or such.

- The 7250-24P is louder than the 6450 (45-48db). I would say mid-way between my 6450 and 6610. Definitely louder than the 6450, but is at a wife-acceptable volume level with no modifications (stock fans/etc...) from the next room with the door open with 6 10gb and a dozen 1gb connections. Wouldn't put in the bedroom closet or want to sit next to it for extended periods, but would not classify it as loud. Love that the latest OS has more straight-forward handling of ports with both tagged/untagged VLANs, etc... but wouldn't base a buying decision based on that. This replaced a Mikrotik CRS309 and two Unifi switches.

- The 6610-24 with 2xRevA's was obviously louder (~53-58db depending on PSU fan speed) than both of the above with the same load as the 7250. Having just seen more of the RevB stories, I'm watching any keyword related to RPS-15-E/RSP-16-E PSUs on eBay. If the 7250 had BGP, I probably don't really need the QSFP ports and would just stick with the 7250. Would be curious if anyone has done any sound measurements of the actual drop from RevA to RevB's...also it seems that the drop comes with dual B's...would running the 24 with a single B PSU be materially quieter than one or two RevA's? I don't feel like I'm too far from acceptable sound levels and my other option is to start running fiber to the garage/attic or just give up.

The 6450-24p is not "NOT" quiet ! Maybe the non poe version. but the "P" version isn't. That's why i just changed my switch fans

 

[sth]

I'm struggling to get a ICX7400-4 x 10GC module to connect up, I ran the dd diag command and got this output (trimmed for relevant bits). Is this likely a configuration issue, or busted hardware?

==========================================================================
UNIT 1: SL 2: ICX7400-4X10GF 4-port 40G Module
==========================================================================
UNIT 1: SL 3: ICX7400-4X10GC 4-port 40G Module
==========================================================================
UNIT 1: SL 4: ICX7400-1X40GQ 1-port 40G Module

<snip>

********************************************************
RUNNING PHY DIAGNOSTICS:
********************************************************

<snip>

Port 1/2/1 CPU Packet Diagnostics Running...
Port 1/2/1 CPU Packet Diagnostics - PASSED

Port 1/2/2 CPU Packet Diagnostics Running...
Port 1/2/2 CPU Packet Diagnostics - PASSED

Port 1/2/3 CPU Packet Diagnostics Running...
Port 1/2/3 CPU Packet Diagnostics - PASSED

Port 1/2/4 CPU Packet Diagnostics Running...
Port 1/2/4 CPU Packet Diagnostics - PASSED

Port 1/3/1 CPU Packet Diagnostics Running...
Port 1/3/1 CPU Packet Diagnostics - PASSED

Port 1/3/2 CPU Packet Diagnostics Running...
Port 1/3/2 CPU Packet Diagnostics - PASSED

Port 1/3/3 CPU Packet Diagnostics Running...
Port 1/3/3 CPU Packet Diagnostics - PASSED

Port 1/3/4 CPU Packet Diagnostics Running...
Port 1/3/4 CPU Packet Diagnostics - PASSED

Port 1/4/1 CPU Packet Diagnostics Running...
Port 1/4/1 CPU Packet Diagnostics - PASSED

<snip>
********************************************************
RUNNING PACKET LINE RATE DIAGNOSTICS:
********************************************************

Packet Line rate test is in progress......
port    sent            received        diff
--------------------------------------------------------------

<snip>

1/2/1   437101091       437101091       0               ##PASS
1/2/2   432729626       432729626       0               ##PASS
1/2/3   416756927       416756927       0               ##PASS
1/2/4   412123190       412123190       0               ##PASS
1/3/1   412579691       412579691       0               ##PASS
1/3/2   412056371       412056371       0               ##PASS
1/3/3   408056253 [  474.291120] [BrcdSoftlockup]: Tracker queued..
        408056253       0               ##PASS
1/3/4   402957714       402957714       0               ##PASS
1/4/1   456645136       456645136       0               ##PASS

 

[fohdeesha]

The 6450-24p is not "NOT" quiet ! Maybe the non poe version. but the "P" version isn't. That's why i just changed my switch fans

my stock 6450-24p is dead silent nearly. The main issue with these switches (and why everyone has vastly different noise reports), is due to the age, some fans have started to wear their bearings and dry up and they get much louder

 

[fohdeesha]

EDIT: found the link to the published sources, though I haven't extracted it yet (on vacation/mobile). Reading the README for 6450 though indicates that there indeed binary blobs from Brocade/Arris to include. Humm... Makes sense not to include source to this libs as these most likely aren't under GPL. Will have to disassemble them and inspect further.

---

Has anyone successfully pulled down the source code and built the firmware? It was mentioned on the first page that under GPL licensing they are publishing, slowly.

For which device though? And would the process work for both the 6450 48p and 7250 48p ARM CPUs?

I'm on the fence for a 6450 or 7250 for their low noise and source code availability. I have four 10 Gbps 10GbaseT devices right now and technically could have a 5th, which would me swing towards the 7250 series.

However, building the firmware (with little to no proprietary binary blobs) is one thing that really interests me.

Maybe someone could point us to the source code links? I'll go back through the thread again and see if it was brought up...

The only source code you are going to get for any of these switches is the u-boot bootloader (which is what you linked, I requested u-boot sources for all of their models some time ago and they are *slowly* complying). The actual FastIron software, which is what talks to the ASIC, manages chassis fans, gives you the actual switch management interface etc, is completely proprietary and one giant monolithic binary blob

 

[ronclark]

The 6450-24p is not "NOT" quiet ! Maybe the non poe version. but the "P" version isn't. That's why i just changed my switch fans

I have a 6650-48P Rev A power supplies. I can't hear it over my R710s and my MD1200.
my 6450 came with two power supplies and two fan chartrages. there rear exhaust. I am just running one power supply and one fan chartrage.

 

[vangoose]

I have a 6650-48P Rev A power supplies. I can't hear it over my R710s and my MD1200.
my 6450 came with two power supplies and two fan chartrages. there rear exhaust. I am just running one power supply and one fan chartrage.

6450-48P?

6650 is very very loud even with rev B psus.