Drag to reposition cover

Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

CorvetteGS

Member
Jan 20, 2014
35
4
8
Atlanta, GA
I'm wondering if someone can help me with my IPv6 setup with my new ICX 6610. I currently have a Mikrotik RB4011 as my WAN gateway/firewall. It provides DHCP for VLANs (VLAN 10 & 20) behind my Brocade that don't have a presence on the Mikrotik. My ip `helper-address` relays DHCP requests to the firewall. It works exactly as expected. My Brocade is able to do all the switching and forward requests to the firewall via a static route.

I would like something similar to happen with IPv6, but I'm unsure how to get it to happen. I am able to get a /60 prefix from my ISP and I'd like to give one of each of the /64 prefixes to a different VLAN behind the Brocade. I don't know how to do that programmatically, so right now I have them hardcoded in the brocade. I choose a different subnet for each VLAN, then have a static route setup back to the firewall. It works, each of my IPv6 clients configure their own address on their own IPv6 subnet. This is the relevant config section:

ipv6 route ::/0 ve 250 fe80::XXXX:XXff:feXX:XXXX
!
interface ve 10
ip address 172.16.10.1 255.255.255.0
ip helper-address 1 10.250.250.1
ipv6 address 2600:XXXX:XXXX:XXX1::1/64
ipv6 enable
ipv6 nd prefix-advertisement 2600:XXXX:XXXX:XXX1::/64 1000 800 onlink autoconfig
!
interface ve 20
ip address 172.16.20.1 255.255.255.0
ip helper-address 1 10.250.250.1
ipv6 address 2600:XXXX:XXXX:XXX2::1/64
ipv6 enable
ipv6 nd prefix-advertisement 2600:XXXX:XXXX:XXX2::/64 1000 800 onlink autoconfig
!


It works fine, and my prefix doesn't seem to change very often (I've never tracked it very carefully, and I was previously using pfSense with 1 VLAN so I wouldn't have noticed if it did change, but AFAIK, it's never changed) so it isn't that big a deal to hard code them, but I'd love if there was a way to do it programmatically. Could anyone tell me if/how I can get my prefix programmatically? My understanding is that I need to run a DHCPv6 server on my Mikrotik, but I'm not sure what more to do beyond that. The documentation for ipv6 dhcp-relay makes it sound like something used for client DHCPv6 requests, which isn't what I want. I want my clients to continue to auto-config and just send the prefix to the Brocade.
I tried to go down this route as well with my setup. I also get a /60 prefix from my ISP (AT&T) and currently have my edgerouter 4 delegating prefixes to all my VLANS. I posted about his previously in this thread but never got any real direction on how to get the ICX to handle prefix delegation. If you want to read through that conversation you can start at post #3323.
 
  • Like
Reactions: bbqdt

Quasduco

Active Member
Nov 16, 2015
129
46
28
109
Tennessee
I am now the happy owner of an ICX6610-48P thanks to this thread! Already enjoying the PoE life with less power cords to deal with!

Next step - it's finally time to pull the trigger on a rack cabinet...!
 

hmw

Active Member
Apr 29, 2019
265
84
28
Saw an eBay listing for an ICX-6610-48-PE for $150 with "premium s/w license" but only 1GbE on the SFP 10GbE modules. I offered $120, the seller countered with $140 which I accepted. Sadly the seller said he doesn't see many Rev B or C PSUs, most of them are Rev A it seems. It will be here by end of the week, and will be my first foray into VLANs, fabric/aggregate switches and what not.

I've had to read (most) of the thread before making up my mind and kudos to @fohdeesha and everyone else for an amazing guide + all the hard work.
 

ljvb

Member
Nov 8, 2015
87
21
8
44
Issues with 4x breakout cables... I think... Well.. at 187 pages.. I'm pretty sure I missed a ton, but my search turned up nothing.

I have two 40G (4x10G) breakout cables, one is cisco (copper), one is arista (fiber) , but they both exhibit the same problem. The line will drop, and not come back up. It is random it seems.

I picked up a new (to me) DL360pG8 which I dropped in an HP branded intel Flexlom card. I originally had 4 10GB links on one of my servers, all were up and functional. I moved 2 of the links to the new server, and the nic lights come on showing connection.. then it drops and the switch shows the port as down. putting the link back onto the original server does not bring the line up......

This is driving me nuts..... I am temped to dump the 40G 4x10G cables, but they are convenient.. and well, if that is not the problem I would have just thrown away some cash which I would prefer not to do..

Version details below
Code:
  UNIT 1: compiled on Aug 31 2018 at 07:07:11 labeled as FCXR08030sa
                (10539398 bytes) from Primary FCXR08030sa.bin
        SW: Version 08.0.30saT7f3
    UNIT 2: compiled on Aug 31 2018 at 07:07:11 labeled as FCXR08030sa
                (10539398 bytes) from Primary FCXR08030sa.bin
        SW: Version 08.0.30saT7f3
  Boot-Monitor Image size = 370695, Version:10.1.00T7f5 (grz10100)
  HW: Stackable ICX6610-24-HPOE
 

noise850

New Member
Feb 28, 2020
10
0
1
(MB40201VX same static pressure but 10+ CFM instead of 6.someting).

I think that you can solve your problems by choosing one of these.
MB40201VX arrived today. Swapped it out and it's about the same noise level and feels about the same airflow as the stock fan. Overall it is a tad bit quieter since there was some sort of a bad bearing noise on the stock fan. However, the switch still isn't lowering the fan speed on boot. Just to make sure I put the original fan back in, booted it, and it also doesn't drop the fan speed on boot. The only way I can get the fan to slow down is with the dd set_pwrfan_low command. Is this correct, or is there something wrong with this switch?

Overall, I am happier with the MB40201VX. Today the room the switch is in got up to 90 F, which is about as hot as it ever gets (poor insulation, poor ventilation, and the sun sets on that side but I have to work with what I've got)

These were the max temp readings on the switch:

dd set_pwrfan_low

Code:
Fan ok, speed (auto): [[1]]<->2

Fan speed switching temperature thresholds:
        1 -> 2 @ 69 deg-C
        1 <- 2 @ 64 deg-C

Sensor B Temperature Readings:
        Current temperature : 50.0 deg-C
Sensor A Temperature Readings:
        Current temperature : 62.5 deg-C
        Warning level.......: 66.0 deg-C
        Shutdown level......: 76.0 deg-C
dd set_pwrfan_high

Code:
Fan ok, speed (auto): [[1]]<->2

Fan speed switching temperature thresholds:
        1 -> 2 @ 69 deg-C
        1 <- 2 @ 64 deg-C

Sensor B Temperature Readings:
        Current temperature : 47.5 deg-C
Sensor A Temperature Readings:
        Current temperature : 57.5 deg-C
        Warning level.......: 66.0 deg-C
        Shutdown level......: 76.0 deg-C
What do you think? Will the switch be okay with the fan set manually to low? Should I try re-flashing the firmware to see if that fixes the auto fan speed switching? If I use dd set_pwrfan_low will the switch override if it gets too hot or will it shutdown?

Thanks for the help!
 
Last edited:

infoMatt

Active Member
Apr 16, 2019
183
77
28
However, the switch still isn't lowering the fan speed on boot. Just to make sure I put the original fan back in, booted it, and it also doesn't drop the fan speed on boot. The only way I can get the fan to slow down is with the dd set_pwrfan_low command. Is this correct, or is there something wrong with this switch?
No, definitely it should lower the fans. But I'm placing my bet on a software issue, because the fact that it can slow down the fans when forced with the "set_pwrfan_low" clears any suspect of a bad PWM circuitry and what not. Maybe once issued one of the two set_pwrfan commands it won't fall back to an auto speed regulation... I have to call @fohdeesha, I don't have an answer for this, sorry. :(

IMHO, if those are your maximum temps in the hottest days, I don't think that it hurts keeping it on the lower speed, at least until there's an answer to the previous question.. at least, it's easier on your ears ;). You have at least 13°C of headroom until the forced shutdown.
 

noise850

New Member
Feb 28, 2020
10
0
1
There are loud resonant frequencies happening with the MB40201VX that didn't happen with the MB40201V3 when set to low speed mode. Sounds like multiple frequencies mixed together, like a dial tone but pitched up a bit. You can't hear it in the high speed mode since it is masked by the loud noise of the fan but in low speed mode it's very noticeable. I have a very small office space and you can even hear this resonance down the hall with 2 other PCs turned on in the room. You can't hear the resonance with the brocade case lid off, only when it's on.

I'm beginning to think this just isn't going to work as I am now on my third replacement fan. I am not going for pure silence but if I can at least keep it around the same noise as the other desktops that would be great. Are there any other fans I can try that will keep this thing cool, even if manually set to high speed mode? I know a lot of people on here are happy with the 6450 so not sure why I am having such bad luck with this one.
 

infoMatt

Active Member
Apr 16, 2019
183
77
28
if I can at least keep it around the same noise as the other desktops that would be great.
Honestly, a 1U box cannot be as quiet as a desktop PC using 120 or 140mm fans... The wider the fans the lower the noise they make moving the same amount of air.
Maybe you've never listened to a pizza-box server with 14 40x40x40mm (usually 7 groups of 2 stacked for static pressure) fans all revving up to the maximum, screaming at boot
 
Last edited:

noise850

New Member
Feb 28, 2020
10
0
1
I completely understand it is going to be louder than the desktops and we have about 30 pizza boxes at work so I know their sound really well. For a home office it's more of a quality vs quantity of noise issue. The MB40201V3 was pushing about as much air on high speed as the MB40201VX is on low speed, but this resonant frequency wasn't there. The sound of air flow is one thing, a solid tone is another. Even the groaning sound of the failing stock fan was more bearable than whatever weird interaction is happening now.

Also this is a non PoE model, so maybe having the one fan vs multiple on the PoE models is causing me issues?

I guess I can try the Noctua A4x20 and see how it works out. From reading over the thread seems some people have had good luck, others haven't.
 

ljvb

Member
Nov 8, 2015
87
21
8
44
Honestly, a 1U box cannot be as quiet as a desktop PC using 120 or 140mm fans... The wider the fans the lower the noise they make moving the same amount of air.
Maybe you've never listened to a pizza-box server with 14 40x40x40mm (usually 7 groups of 2 stacked for static pressure) fans all revving up to the maximum, screaming at boot
This would be pretty much every HP server... I have 2 DL360pG8's and 2 DL380pG8's on top of 2 icx switches.. and it sounds like a jumbo jet on takeoff when the rack comes up after a power failure (my UPS can only run them for about 15 min)

I keep the rack in the garage, so the noise is less of an issue. The heat (and humidity when the garage door is open) is more of an issue.

Still not as bad as the HP 3000 blade chassis with 8 blades in it.... that was loud.. by loud standards
 

Spearfoot

Member
Apr 22, 2015
68
26
18
62
Many thanks to @fohdeesha and the other posters here who have really made this a great thread.

You've made me lighten my wallet... but it's been well worth it.

Bought a Ruckus ICX 7150-C12P last month to replace the Cisco SG350-10P in my office. It works great!

This morning I negotiated a reasonable price -- less than $200 including shipping and taxes -- from an eBay seller for a beefier ICX 6610-48P-E unit w/ 2 power supplies. That model number specifies a single power supply, so the extra one must have been added on later? Anyway, if anyone's interested, the seller has 4 more identical units for sale:


This bad boy will replace the Aruba S2500-48P out in my shop.

Now I'm getting quotes for pulling fiber from my office out to the shop, and to upgrade the CAT5 run from my office to the living room. The fun never $top$!
 
  • Like
Reactions: fohdeesha

ViciousXUSMC

Active Member
Nov 27, 2016
201
99
28
38
I replaced my S2500's with the Brocades and I am happy I did just a more natural configuration environment than the S2500 that was intended to use with a controller.

The 6450 is perfect for me with the fan mods as its feet away on my rack and I can still do voice over work without any noise, the 6610 is just there waiting for its day to shine as its way too loud to use in the same room.

Id like to re-home the S2500's but I think it would cost nearly as much to ship as what I could sell them for :/ and unfortunately I never did find brackets for my Brocade switches so I had to buy rack shelves and I am not super thrilled about that.

I have not had any heat issues with the fan mod, but the switch is not heavily loaded, maybe 12 or 14 Ethernet connections and only sending PoE to a single AP. Two SFP+ ports running fiber to my ESXi server and my desktop.
 
  • Like
Reactions: fohdeesha

Spearfoot

Member
Apr 22, 2015
68
26
18
62
I replaced my S2500's with the Brocades and I am happy I did just a more natural configuration environment than the S2500 that was intended to use with a controller.

The 6450 is perfect for me with the fan mods as its feet away on my rack and I can still do voice over work without any noise, the 6610 is just there waiting for its day to shine as its way too loud to use in the same room.

Id like to re-home the S2500's but I think it would cost nearly as much to ship as what I could sell them for :/ and unfortunately I never did find brackets for my Brocade switches so I had to buy rack shelves and I am not super thrilled about that.

I have not had any heat issues with the fan mod, but the switch is not heavily loaded, maybe 12 or 14 Ethernet connections and only sending PoE to a single AP. Two SFP+ ports running fiber to my ESXi server and my desktop.
Hello, @ViciousXUSMC -- I remember you well from the Aruba S-series thread; another thread that made me spend money! Ha Ha!

I'll be connecting 3 FreeNAS/ESXi AIO systems to the 6610, plus an AP and other miscellaneous stuff. Just the 3 AIOs use 5 x SFP+ and 13 x Gbe ports, counting IPMI. Truth is -- the Aruba would serve my needs just fine, and this ICX-6610 is vast overkill... but you know the old saying: "There's no kill, like overkill!"
 
  • Like
Reactions: fohdeesha

hmw

Active Member
Apr 29, 2019
265
84
28
Yeah talk about temptation. Read almost all of the thread and then caved in and got a 6610-48P with two PSUs for $140 and a 6610-24P with two Rev B PSUs for $170

Hopefully they should be here next week and I’ll ping @fohdeesha for licenses (you’re awesome man!)

I might just keep the 24 port variant and dispose of the 48 port one. The eBay seller told me he’d try to put REV B PSUs for the 48 port but I don’t have much hopes of that happening
 
  • Like
Reactions: fohdeesha

ViciousXUSMC

Active Member
Nov 27, 2016
201
99
28
38
Hello, @ViciousXUSMC -- I remember you well from the Aruba S-series thread; another thread that made me spend money! Ha Ha!

I'll be connecting 3 FreeNAS/ESXi AIO systems to the 6610, plus an AP and other miscellaneous stuff. Just the 3 AIOs use 5 x SFP+ and 13 x Gbe ports, counting IPMI. Truth is -- the Aruba would serve my needs just fine, and this ICX-6610 is vast overkill... but you know the old saying: "There's no kill, like overkill!"
Yes the S2500 served me well, but the 6450 was more efficient, had more power budget, and was actually quieter after the fan mods. I wanted to explore the CLI more and try to get more videos up so I was doing it just to do it more than out of any real need.

I got so busy IRL and lost a lot of footage I took of everything.

I much prefer configuration of the Brocade to the Aruba though as a Cisco guy it feels pretty natural.
Next thing for me to do is automation via PLINK so I can torture the kids by turning off access when they are bad with a single click of a button.

I was able to fabricate my own brackets for the Aruba switches out of some Cisco brackets that work perfectly, but no luck on the Brocade switches :/ that is probably my only sad thing about the new switches.
 
  • Like
Reactions: fohdeesha

ljvb

Member
Nov 8, 2015
87
21
8
44
Hello, @ViciousXUSMC -- I remember you well from the Aruba S-series thread; another thread that made me spend money! Ha Ha!

I'll be connecting 3 FreeNAS/ESXi AIO systems to the 6610, plus an AP and other miscellaneous stuff. Just the 3 AIOs use 5 x SFP+ and 13 x Gbe ports, counting IPMI. Truth is -- the Aruba would serve my needs just fine, and this ICX-6610 is vast overkill... but you know the old saying: "There's no kill, like overkill!"
Curious, if they are all in one setups, even 3, why are you using so many ports. I have 2 DL360ps with 2 port 10GB intel, and 4x1GB flexlom, and 2 DL380ps with the same. I'm only using 3 ports per machine (2 10GB ports and 1 1GB for backup connection to the switch), plus 2 ports for my ubiquiti setup ( each setup consists of 1 AP and 1 POE powered 8 port switch, AP gets power from the 8 port, the 8 port from the brocade, so only 2 ports there).

Rest of the ports (I have 2 24 port 6610s) are for drops from around the house, and eventually POE IP cameras.
 

rosaage

New Member
Sep 25, 2019
6
2
3
No, definitely it should lower the fans. But I'm placing my bet on a software issue, because the fact that it can slow down the fans when forced with the "set_pwrfan_low" clears any suspect of a bad PWM circuitry and what not. Maybe once issued one of the two set_pwrfan commands it won't fall back to an auto speed regulation... I have to call @fohdeesha, I don't have an answer for this, sorry. :(

IMHO, if those are your maximum temps in the hottest days, I don't think that it hurts keeping it on the lower speed, at least until there's an answer to the previous question.. at least, it's easier on your ears ;). You have at least 13°C of headroom until the forced shutdown.
I'm also having the same problem as noise850 with the switch stuck in high speed fan mode, using the command he used I can lower the fans. I tried a reload with the fans set to low, and at the same time as the "enter b to stop at boot" the fans changed to high and stays like this. I have a 6450-48 non p model just updated using the file from the guide (08030t). There is also some kind of whining noise running in low. It may be present at high too just non audible when the fans are a bit louder.
I haven't tried running it for more than about an hour yet but after that time it sits around 40C for sensor B and 50C for sensor A in low fan mode, so well below the threshold temperatures.

Any specific fans I should be on the lookout for if I want to replace it? one of the suppliers with good shipping (Distrelec) has over 30 different 12v 40mm fans just from sunon. Distrelec Link
 

Spearfoot

Member
Apr 22, 2015
68
26
18
62
Curious, if they are all in one setups, even 3, why are you using so many ports. I have 2 DL360ps with 2 port 10GB intel, and 4x1GB flexlom, and 2 DL380ps with the same. I'm only using 3 ports per machine (2 10GB ports and 1 1GB for backup connection to the switch), plus 2 ports for my ubiquiti setup ( each setup consists of 1 AP and 1 POE powered 8 port switch, AP gets power from the 8 port, the 8 port from the brocade, so only 2 ports there).

Rest of the ports (I have 2 24 port 6610s) are for drops from around the house, and eventually POE IP cameras.
My earlier quote about how there's "No kill like overkill" applies here. I only use so many ports because of that mindset. Hey, I've got 'em -- why not use 'em?

Two of the ESXi systems have 4 x Gbe ports along with a dual-ported 10Gb Intel X520-DA2. So on these I dedicate two Gbe ports to the VMware management network and use two as offline standbys to one of the X520 ports for VM traffic. I use the remaining X520 port for a direct connection between the two FreeNAS instances for rsync and replication. So, adding in an IPMI port apiece, I end up using 10 of the Gbe ports just for these two guys.

The third ESXi box only has 2 x Gbe ports, so I can't set it up with that level of redundancy.
 

infoMatt

Active Member
Apr 16, 2019
183
77
28
There is also some kind of whining noise running in low. It may be present at high too just non audible when the fans are a bit louder.
Mine has it as well, I think it's due to the low-ish frequency of the PWM modulation used to lower the fan speed. Maaayyyybe it's possible to make it go down by adding some capacitor to smooth it, but I don't want to advise anyone to add some random caps... I don't want to risk someone's equipment.
 

rosaage

New Member
Sep 25, 2019
6
2
3
Mine has it as well, I think it's due to the low-ish frequency of the PWM modulation used to lower the fan speed. Maaayyyybe it's possible to make it go down by adding some capacitor to smooth it, but I don't want to advise anyone to add some random caps... I don't want to risk someone's equipment.
Thank you! adding a 330µF cap removed the whining noise.
Here is a video where i power on the switch, test the cap, and measure the pvm frequency (~3kHz).

This video shows a different fan. Since this fan is more quiet and runs slower the whining is even worse. Here you can easily hear the difference with and without the cap. (Ignore the ugly soldering, my gas soldering iron ran out of gas)
The switch had some nice points for soldering, there is a ceramic smd cap together with a set of unpopulated pads on each of the fan headers.