Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)
- Thread starter fohdeesha
- Start date
Been quite a while since you posted this, but, could you add these commands to your beautiful docs ?
Noise, ICX 6610 and what an ENORMOUS difference two rev b PSU's make compared to 1 rev b and 1 rev a...
I was able to pick up 2 Rev B POE PSU's < 50 each off the bay (seller had 2 total). I installed them last night and I cannot believe how much of a difference it makes!
I need to find one more reasonably priced rev b for my other 6610.
I was able to pick up 2 Rev B POE PSU's < 50 each off the bay (seller had 2 total). I installed them last night and I cannot believe how much of a difference it makes!
I need to find one more reasonably priced rev b for my other 6610.
What are the temps though?
I had heat issues and went to 2 fans, 2 psu, and heat issues went away. They're REv A.
In one I have a REv B but can't tell the difference... debating if I should swap that B to A for more air flow.
I have a 8-11*C difference between the 6610 on top vs on bottom too, unsure if it's location, A vs B, or due to the rack-ear not having the middle\vent hole.
My original switch purchase had 2 PSU and 2 FANS in each unit. I've only been running 1 switch due to noise and playing with the 40Gbe. If I can source 1 more rev B I will try and figure out if it is really possible to stack with a single 40Gbe and use the remaining 2 for a LAG.
I've marked a seller on the bay that has qty 4 rev B power supplies (confirmed via message), asking a lot for 'em but none of have moved since it was posted. May take a lower number though it will likely be about what I paid for these 2 combined. But my ears, oh my. huge huge difference. my CSE-417 is now the loudest thing in the rack and I removed some fans from it to quiet it down a bit (all flash, 2/3 full, temps are fine there).
I figured since I did not hear the fans ramp up (except at power on) that all must be okay and I was really just floored by the sound difference so I didn’t even check temps.
Original configs:
#1, ICX6610-48P-E, 1xRev A PSU-E , 1x Rev B PSU-E, 2x FAN-E
#2, ICX6610-48P-E, 2xRev A PSU-E, 2xFAN-E
New config ICX
#1 ICX6610-48P-E, 2x Rev B PSU-E, 2x FAN-E,
#2, ICX6610-48P-E, 1xRev A PSU-E , 1x Rev B PSU-E, 2x FAN-E
Code:
icx6610-stack#show chassis
The stack unit 1 chassis info:
Power supply 1 (AC - PoE) present, status ok
Model Number: 23-0000142-02
Serial Number: B1S
Firmware Ver: B
Power supply 1 Fan Air Flow Direction: Front to Back
Power supply 2 (AC - PoE) present, status ok
Model Number: 23-0000142-02
Serial Number: 5WP
Firmware Ver: B
Power supply 2 Fan Air Flow Direction: Front to Back
Fan 1 ok, speed (auto): [[1]]<->2
Fan 2 ok, speed (auto): [[1]]<->2
Fan controlled temperature: 65.5 deg-C
Fan speed switching temperature thresholds:
Speed 1: NM<----->76 deg-C
Speed 2: 71<-----> 80 deg-C (shutdown)
Fan 1 Air Flow Direction: Front to Back
Fan 2 Air Flow Direction: Front to Back
MAC 1 Temperature Readings:
Current temperature : 45.0 deg-C
MAC 2 Temperature Readings:
Current temperature : 54.0 deg-C
CPU Temperature Readings:
Current temperature : 65.5 deg-C
sensor A Temperature Readings:
Current temperature : 52.0 deg-C
sensor B Temperature Readings:
Current temperature : 50.5 deg-C
sensor C Temperature Readings:
Current temperature : 37.0 deg-C
stacking card Temperature Readings:
Current temperature : 64.0 deg-C
Warning level.......: 77.0 deg-C
Shutdown level......: 80.0 deg-CWhich? These just posted?
MAC 1 Temperature Readings:
Current temperature : 45.0 deg-C
MAC 2 Temperature Readings:
Current temperature : 54.0 deg-C
CPU Temperature Readings:
Current temperature : 65.5 deg-C
sensor A Temperature Readings:
Current temperature : 52.0 deg-C
sensor B Temperature Readings:
Current temperature : 50.5 deg-C
sensor C Temperature Readings:
Current temperature : 37.0 deg-C
Mine that wasn't working was running at 79 and 80*C until we fixed the fan, and air flow situation, now both are in the 40s and 50s *C and work great
it doesn't even bother to ramp up fans until 76c, 65 is more than fine. these things run hot (which is why I keep yelling at people to stop wasting their time trying to stick even slower fans in the 6610)
yes, folks please listen to Uncle Fohdeesha. Just find some deals and invest in rev B psu's . I am still non-sonically stunned by how quiet two 6610 POE rev B's are. Fohdeesha, you posted about the acoustics early on in the thread - just hadn't found a deal until last week and it was SO worth it!
Well I kinda had a decent weekend. First up was installing the new Fans into the 6450,, AHHH!! much silent ! And still passing air and keeping it cool.
Next was installing 10gig nic's into the Hp ML350 G8 server ( Fail ) they don't work
Ordered some PCI-e 10gig HP nic's instead of these Lenovo Emulex ones !!
Oh ! And i don't have an addiction !! So i bought 2 of these Brocade ICX 6430-C12's for 100$ One will be connected to my Brocade 6450 via Fiber then sitting behind my desk The other will be spare ? maybe...
Next was installing 10gig nic's into the Hp ML350 G8 server ( Fail ) they don't work
Ordered some PCI-e 10gig HP nic's instead of these Lenovo Emulex ones !!
Oh ! And i don't have an addiction !! So i bought 2 of these Brocade ICX 6430-C12's for 100$ One will be connected to my Brocade 6450 via Fiber then sitting behind my desk
The other will be spare ? maybe...Attachments
Hi , great guide btw ( FCX / ICX6610 - Fohdeesha Docs ) I couldn't find any info on how to abtain or active licence for 6450 24p for the 10GB ports.forgot to mention: licenses for these models are free for existing STH members - now that they are EOL & discontinued I don't really have an alternative to giving them away
EDIT: found the link to the published sources, though I haven't extracted it yet (on vacation/mobile). Reading the README for 6450 though indicates that there indeed binary blobs from Brocade/Arris to include. Humm... Makes sense not to include source to this libs as these most likely aren't under GPL. Will have to disassemble them and inspect further.
---
Has anyone successfully pulled down the source code and built the firmware? It was mentioned on the first page that under GPL licensing they are publishing, slowly.
For which device though? And would the process work for both the 6450 48p and 7250 48p ARM CPUs?
I'm on the fence for a 6450 or 7250 for their low noise and source code availability. I have four 10 Gbps 10GbaseT devices right now and technically could have a 5th, which would me swing towards the 7250 series.
However, building the firmware (with little to no proprietary binary blobs) is one thing that really interests me.
Maybe someone could point us to the source code links? I'll go back through the thread again and see if it was brought up...
---
Has anyone successfully pulled down the source code and built the firmware? It was mentioned on the first page that under GPL licensing they are publishing, slowly.
For which device though? And would the process work for both the 6450 48p and 7250 48p ARM CPUs?
I'm on the fence for a 6450 or 7250 for their low noise and source code availability. I have four 10 Gbps 10GbaseT devices right now and technically could have a 5th, which would me swing towards the 7250 series.
However, building the firmware (with little to no proprietary binary blobs) is one thing that really interests me.
Maybe someone could point us to the source code links? I'll go back through the thread again and see if it was brought up...
Last edited:
Ok. Now have 6610-24 (2x RevA PSUs), 6450-24, and 7250-24P that were purchased in that order.
Since there is a lot of discussion on noise:
- The 6450-24 with stock fans was so quiet (38-42db) that I was able to move it to a bedroom closet and nobody knows it is there w/ 2 10gb and a dozen 1gb connections. Everyone has different sound tolerance, but I had multiple people help listen for it and we literally could not hear it -- I suspect that people with louder fans may have bad fans. It nicely replaced both a Mikrotik CRS305 and an older Unifi switch and barely consumes any power. It's incredible for the price and I would highly recommend it for most people that only need a few 10gig ports for a NAS or such.
- The 7250-24P is louder than the 6450 (45-48db). I would say mid-way between my 6450 and 6610. Definitely louder than the 6450, but is at a wife-acceptable volume level with no modifications (stock fans/etc...) from the next room with the door open with 6 10gb and a dozen 1gb connections. Wouldn't put in the bedroom closet or want to sit next to it for extended periods, but would not classify it as loud. Love that the latest OS has more straight-forward handling of ports with both tagged/untagged VLANs, etc... but wouldn't base a buying decision based on that. This replaced a Mikrotik CRS309 and two Unifi switches.
- The 6610-24 with 2xRevA's was obviously louder (~53-58db depending on PSU fan speed) than both of the above with the same load as the 7250. Having just seen more of the RevB stories, I'm watching any keyword related to RPS-15-E/RSP-16-E PSUs on eBay. If the 7250 had BGP, I probably don't really need the QSFP ports and would just stick with the 7250. Would be curious if anyone has done any sound measurements of the actual drop from RevA to RevB's...also it seems that the drop comes with dual B's...would running the 24 with a single B PSU be materially quieter than one or two RevA's? I don't feel like I'm too far from acceptable sound levels and my other option is to start running fiber to the garage/attic or just give up.
Since there is a lot of discussion on noise:
- The 6450-24 with stock fans was so quiet (38-42db) that I was able to move it to a bedroom closet and nobody knows it is there w/ 2 10gb and a dozen 1gb connections. Everyone has different sound tolerance, but I had multiple people help listen for it and we literally could not hear it -- I suspect that people with louder fans may have bad fans. It nicely replaced both a Mikrotik CRS305 and an older Unifi switch and barely consumes any power. It's incredible for the price and I would highly recommend it for most people that only need a few 10gig ports for a NAS or such.
- The 7250-24P is louder than the 6450 (45-48db). I would say mid-way between my 6450 and 6610. Definitely louder than the 6450, but is at a wife-acceptable volume level with no modifications (stock fans/etc...) from the next room with the door open with 6 10gb and a dozen 1gb connections. Wouldn't put in the bedroom closet or want to sit next to it for extended periods, but would not classify it as loud. Love that the latest OS has more straight-forward handling of ports with both tagged/untagged VLANs, etc... but wouldn't base a buying decision based on that. This replaced a Mikrotik CRS309 and two Unifi switches.
- The 6610-24 with 2xRevA's was obviously louder (~53-58db depending on PSU fan speed) than both of the above with the same load as the 7250. Having just seen more of the RevB stories, I'm watching any keyword related to RPS-15-E/RSP-16-E PSUs on eBay. If the 7250 had BGP, I probably don't really need the QSFP ports and would just stick with the 7250. Would be curious if anyone has done any sound measurements of the actual drop from RevA to RevB's...also it seems that the drop comes with dual B's...would running the 24 with a single B PSU be materially quieter than one or two RevA's? I don't feel like I'm too far from acceptable sound levels and my other option is to start running fiber to the garage/attic or just give up.
You could create a mixed stack with the 6610 and the 6450, that further enables more features on the 6450 like openflow and possibly BGP ?Ok. Now have 6610-24 (2x RevA PSUs), 6450-24, and 7250-24P that were purchased in that order.
Since there is a lot of discussion on noise:
- The 6450-24 with stock fans was so quiet (38-42db) that I was able to move it to a bedroom closet and nobody knows it is there w/ 2 10gb and a dozen 1gb connections. Everyone has different sound tolerance, but I had multiple people help listen for it and we literally could not hear it -- I suspect that people with louder fans may have bad fans. It nicely replaced both a Mikrotik CRS305 and an older Unifi switch and barely consumes any power. It's incredible for the price and I would highly recommend it for most people that only need a few 10gig ports for a NAS or such.
- The 7250-24P is louder than the 6450 (45-48db). I would say mid-way between my 6450 and 6610. Definitely louder than the 6450, but is at a wife-acceptable volume level with no modifications (stock fans/etc...) from the next room with the door open with 6 10gb and a dozen 1gb connections. Wouldn't put in the bedroom closet or want to sit next to it for extended periods, but would not classify it as loud. Love that the latest OS has more straight-forward handling of ports with both tagged/untagged VLANs, etc... but wouldn't base a buying decision based on that. This replaced a Mikrotik CRS309 and two Unifi switches.
- The 6610-24 with 2xRevA's was obviously louder (~53-58db depending on PSU fan speed) than both of the above with the same load as the 7250. Having just seen more of the RevB stories, I'm watching any keyword related to RPS-15-E/RSP-16-E PSUs on eBay. If the 7250 had BGP, I probably don't really need the QSFP ports and would just stick with the 7250. Would be curious if anyone has done any sound measurements of the actual drop from RevA to RevB's...also it seems that the drop comes with dual B's...would running the 24 with a single B PSU be materially quieter than one or two RevA's? I don't feel like I'm too far from acceptable sound levels and my other option is to start running fiber to the garage/attic or just give up.
The 6450-24p is not "NOT" quiet ! Maybe the non poe version. but the "P" version isn't. That's why i just changed my switch fansOk. Now have 6610-24 (2x RevA PSUs), 6450-24, and 7250-24P that were purchased in that order.
Since there is a lot of discussion on noise:
- The 6450-24 with stock fans was so quiet (38-42db) that I was able to move it to a bedroom closet and nobody knows it is there w/ 2 10gb and a dozen 1gb connections. Everyone has different sound tolerance, but I had multiple people help listen for it and we literally could not hear it -- I suspect that people with louder fans may have bad fans. It nicely replaced both a Mikrotik CRS305 and an older Unifi switch and barely consumes any power. It's incredible for the price and I would highly recommend it for most people that only need a few 10gig ports for a NAS or such.
- The 7250-24P is louder than the 6450 (45-48db). I would say mid-way between my 6450 and 6610. Definitely louder than the 6450, but is at a wife-acceptable volume level with no modifications (stock fans/etc...) from the next room with the door open with 6 10gb and a dozen 1gb connections. Wouldn't put in the bedroom closet or want to sit next to it for extended periods, but would not classify it as loud. Love that the latest OS has more straight-forward handling of ports with both tagged/untagged VLANs, etc... but wouldn't base a buying decision based on that. This replaced a Mikrotik CRS309 and two Unifi switches.
- The 6610-24 with 2xRevA's was obviously louder (~53-58db depending on PSU fan speed) than both of the above with the same load as the 7250. Having just seen more of the RevB stories, I'm watching any keyword related to RPS-15-E/RSP-16-E PSUs on eBay. If the 7250 had BGP, I probably don't really need the QSFP ports and would just stick with the 7250. Would be curious if anyone has done any sound measurements of the actual drop from RevA to RevB's...also it seems that the drop comes with dual B's...would running the 24 with a single B PSU be materially quieter than one or two RevA's? I don't feel like I'm too far from acceptable sound levels and my other option is to start running fiber to the garage/attic or just give up.
I'm struggling to get a ICX7400-4 x 10GC module to connect up, I ran the dd diag command and got this output (trimmed for relevant bits). Is this likely a configuration issue, or busted hardware?
Code:
==========================================================================
UNIT 1: SL 2: ICX7400-4X10GF 4-port 40G Module
==========================================================================
UNIT 1: SL 3: ICX7400-4X10GC 4-port 40G Module
==========================================================================
UNIT 1: SL 4: ICX7400-1X40GQ 1-port 40G Module
<snip>
********************************************************
RUNNING PHY DIAGNOSTICS:
********************************************************
<snip>
Port 1/2/1 CPU Packet Diagnostics Running...
Port 1/2/1 CPU Packet Diagnostics - PASSED
Port 1/2/2 CPU Packet Diagnostics Running...
Port 1/2/2 CPU Packet Diagnostics - PASSED
Port 1/2/3 CPU Packet Diagnostics Running...
Port 1/2/3 CPU Packet Diagnostics - PASSED
Port 1/2/4 CPU Packet Diagnostics Running...
Port 1/2/4 CPU Packet Diagnostics - PASSED
Port 1/3/1 CPU Packet Diagnostics Running...
Port 1/3/1 CPU Packet Diagnostics - PASSED
Port 1/3/2 CPU Packet Diagnostics Running...
Port 1/3/2 CPU Packet Diagnostics - PASSED
Port 1/3/3 CPU Packet Diagnostics Running...
Port 1/3/3 CPU Packet Diagnostics - PASSED
Port 1/3/4 CPU Packet Diagnostics Running...
Port 1/3/4 CPU Packet Diagnostics - PASSED
Port 1/4/1 CPU Packet Diagnostics Running...
Port 1/4/1 CPU Packet Diagnostics - PASSED
<snip>
********************************************************
RUNNING PACKET LINE RATE DIAGNOSTICS:
********************************************************
Packet Line rate test is in progress......
port sent received diff
--------------------------------------------------------------
<snip>
1/2/1 437101091 437101091 0 ##PASS
1/2/2 432729626 432729626 0 ##PASS
1/2/3 416756927 416756927 0 ##PASS
1/2/4 412123190 412123190 0 ##PASS
1/3/1 412579691 412579691 0 ##PASS
1/3/2 412056371 412056371 0 ##PASS
1/3/3 408056253 [ 474.291120] [BrcdSoftlockup]: Tracker queued..
408056253 0 ##PASS
1/3/4 402957714 402957714 0 ##PASS
1/4/1 456645136 456645136 0 ##PASS