I believe I meet that edge case, but might be able to configure its usage out of the config, but this certainly makes things easier. For the 3 places I am using it, one is to connect another "switch" (a wireless AP) which has both tagged and untagged data going to/from it and I want the untagged to go to a specific VLAN. The next is a server supporting virtual machines where-in I have created bridged interface devices to specific VLANs for them to reside, but the untagged interface is for the host itself (I can probably reconfigure this, but it is much easier to know that all the tagged interfaces are in use by VMs). The last use case will be for a firewall-on-a-stick using a 40gb connection to a pfsense box with all the VLANs configured (routing will only happen for the LAN/WAN, but I need all the VLANs for it to support NAT/DHCP/DNS for all my devices across all VLANs) and using the default untagged VLAN makes it easier for me to also support the other untagged VLAN devices.
Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)
- Thread starter fohdeesha
- Start date
Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
So, I've run into a bit of a headscratcher. Maybe one of you has seen it. I'm standing up a new segment on a 6450 with OSPF. Except... it's not actually working. This isn't the only part of my environment with FRR 7, but it's the only part doing OSPF with an ICX - the rest is BGP. I'm seeing the Hello packets from the ICX just fine, traffic passes just fine, but adjacency is never established in area 0, even with authentication disabled. Seeing the same problem in non-broadcast and point-to-point as well. Any ideas?
edit: derp, I forgot to mention, this configuration is working with a Juniper, which is why I'm suspecting either a quirk or something not in the docs about OSPF on 08.0.30. And full debug logs on both sides show no errors, but the frr7 side also isn't reporting the HELLO packet at all.
edit: derp, I forgot to mention, this configuration is working with a Juniper, which is why I'm suspecting either a quirk or something not in the docs about OSPF on 08.0.30. And full debug logs on both sides show no errors, but the frr7 side also isn't reporting the HELLO packet at all.
Code:
vlan 1 name DEFAULT-VLAN by port
router-interface ve 1
interface ve 1
ip address 10.1.0.5 255.255.254.0
ip ospf area 0
ip ospf active
ip ospf network non-broadcast
router ospf
area 0
area 0 range 10.53.0.0 255.255.255.0 advertise
summary-address 10.1.0.5 255.255.254.0
redistribute connected
redistribute static
log all
neighbor 10.1.0.20
neighbor 10.1.0.82
Code:
router ospf
ospf router-id 10.1.0.20
log-adjacency-changes
redistribute connected
network 10.53.0.20/32 area 0
neighbor 10.1.0.5
Last edited:
Little bit louder than the 7450-32ZP it’s replacing, too loud to work next to for sure though. It’s in a remote part of our home in a APC netshelter rack so it’s acceptable.
I apologize if this is listed somewhere in here, I wasn't able to find a passive DAC max length for the 6610s.
I'm not sure if I'm running to far of a length for a passive DAC on this switch or if I've got something else screwy going on.
I ordered two of these QSFP-4SFP10G-CU5M (40GBASE-CR4 QSFP to 4 10GBASE-CU SFP+ 5m passive) to connect the dual NICs on my 3 vSphere hosts and NAS. The NICs are different brands, so that may be problematic as well. The NAS has an old ConnectX, two hosts have Emulex 00JY823s, and the third host has a Dell branded x520-DA2.
All four NICs got single SFP+ breakout from both 40Gb interfaces. I got link lights on all connections, and initially only 3 interfaces from each breakout would show up and pass traffic. If I started pushing a lot of throughput, they would drop on both sides and only unplugging/replugging them would get them to reconnect.
Using any of the eight SFP+ ports on the front with 3m Cisco 10Gb DACs works fine in any combination of port or NIC, so I don't think it's an OEM compatibility issue?
If at all possible, I'd like to save as many of these front ports as I can for the occasional 10GbBase-t runs in my house.
I'm not sure if I'm running to far of a length for a passive DAC on this switch or if I've got something else screwy going on.
I ordered two of these QSFP-4SFP10G-CU5M (40GBASE-CR4 QSFP to 4 10GBASE-CU SFP+ 5m passive) to connect the dual NICs on my 3 vSphere hosts and NAS. The NICs are different brands, so that may be problematic as well. The NAS has an old ConnectX, two hosts have Emulex 00JY823s, and the third host has a Dell branded x520-DA2.
All four NICs got single SFP+ breakout from both 40Gb interfaces. I got link lights on all connections, and initially only 3 interfaces from each breakout would show up and pass traffic. If I started pushing a lot of throughput, they would drop on both sides and only unplugging/replugging them would get them to reconnect.
Using any of the eight SFP+ ports on the front with 3m Cisco 10Gb DACs works fine in any combination of port or NIC, so I don't think it's an OEM compatibility issue?
If at all possible, I'd like to save as many of these front ports as I can for the occasional 10GbBase-t runs in my house.
Hi all 
Jumping aboard the Brocade party train If everything goes just as planned, I should be getting a used ICX-6450-48 soon. I spotted a used 7250-48P but never got answers from the seller :/
I'm now digging this awesome thread to find :
- the most silent fans suitable for this switch (I've seen Sunon, Noctua, maybe others are good)
- a recommended stable software version
- a way to enable the 4x10G ports, if not already active. I'll see when I get the switch what's already installed inside
Let's go !
Jumping aboard the Brocade party train
If everything goes just as planned, I should be getting a used ICX-6450-48 soon. I spotted a used 7250-48P but never got answers from the seller :/ I'm now digging this awesome thread to find :
- the most silent fans suitable for this switch (I've seen Sunon, Noctua, maybe others are good)
- a recommended stable software version
- a way to enable the 4x10G ports, if not already active. I'll see when I get the switch what's already installed inside
Let's go !
Thank you! It does appear to stay around 56-65°C during the day, with the 65°C being more in the evening.
Only now I'm seeing it on speed 2:
Code:
SSH@10G#show chassis
The stack unit 1 chassis info:
Power supply 1 (NA - AC - PoE) present, status ok
Power supply 2 not present
Power supply 3 not present
Fan 1 ok, speed (auto): 1<->[[2]]
Fan 2 ok, speed (auto): 1<->[[2]]
Fan 3 ok, speed (auto): 1<->[[2]]
Fan controlled temperature: 64.0 deg-C
Fan speed switching temperature thresholds:
Speed 1: NM<----->65 deg-C
Speed 2: 56<-----> 79 deg-C (shutdown)
Sensor B Temperature Readings:
Current temperature : 60.5 deg-C
Sensor A Temperature Readings:
Current temperature : 64.0 deg-C
Warning level.......: 69.0 deg-C
Shutdown level......: 79.0 deg-C- Well, the most silent are always the Noctua that I have (Noctua NF-A4x20 FLX) and judging from my conversation with Scarlet they appear to be as efficient for cooling as the Sunon. The issue that was indicated in the previous posts (the fans not spinning at speed 1) does not appear to happen with these specific Noctua Fans. Don't forget that you have to swap the wires on the connector before installing them!Hi all
Jumping aboard the Brocade party train
I'm now digging this awesome thread to find :
- the most silent fans suitable for this switch (I've seen Sunon, Noctua, maybe others are good)
- a recommended stable software version
- a way to enable the 4x10G ports, if not already active. I'll see when I get the switch what's already installed inside
Let's go !
- The latest. Follow the instructions here: ICX6430 / ICX6450 - Fohdeesha Docs
- Send a PM to @fohdeesha , he will take care of you for the licenses.
BTW, concerning the Noctua, thanks to the Noctua + manual fan management of the 2 R720 via ipmitool my rack is now basically silent!!!!
So in order...
- the best fans for this are unobtanium due to COVID-19, but don't use Noctua. Noctua is overpriced junk - it has the lowest flow rate and lowest static pressure. Anything from Sunon or NMB or Nidec or Delta is fine. They also cost far less.
- 08.0.30t for the 6450 is the maximum possible
- see first post for that, but if the switch was only configuration reset and not factory reset it may have them installed. Do a show licenses first always.
Edit: to expound upon my 'unobtanium' fan choice, bear in mind, I have been doing systems integration for 30 years and have VERY extensive rackmount equipment experience. I know why Brocade chose the fans they did, I have worked with those fans, and they are the same fans I would have chosen. (BTW, at speed 2 the 6450-40P is attempting to move 90.9CFM. Yes, each of those 40mm fans is rated for 30.3CFM free air.)
If you want to quiet down these units, the BIGGEST change you can make is to remove the stamped grill with a Dremel. It's all sharp edges, and significantly blocks the hub as well. Turbulence from things like grills are what generate serious noise. If you still need protection, tack on a wire grill on the OUTSIDE of the chassis. HOWEVER, this will result in slightly reduced cooling to the front right corner. I won't bother going into my unobtanium choice, because unless you're ready to order at least 250 of them, you can't buy them.
So here's some fans you can buy that I'd recommend.
- Delta EFB0412VHD-F00 - $12.40/ea @ Digikey
There's a REASON Delta's a top pick for ODM and OEM. 40x40x20mm, 10.1CFM, 0.416in H2O (which is insane,) 32.5dBA @ 1m, rated for 70k hours at 50C. Make sure it's THAT part number and not the Rev C, which is a vastly inferior part. - Comair Rotron "Gryphon" GDA4028-12BB - discontinued, alas
If you can find these? BUY THESE. 40x40x28mm, but 11CFM, 0.34in H2O, 8800RPM, but just 31.4dBA! I am still mad they discontinued them. No, I am not selling any of my spares. - Mechatronics MR4020E12B1-RSR - $7.50 @ Digikey (when in stock)
READ THE PAGE. These are non-stocked currently! However these offer excellent balance between flow and noise. 40x40x20mm, 15.8CFM, 0.45in H2O, and 39dBA @ 1m at 11000RPM. (Yes, it's about the quietest 11k fan ever made.) You need the E12B1 though - NOT the B2. The B1 has a tach, the B2 is a rotor lock wire. - Mechatronics MR4020H12B1-RSR - about $8
The 'slightly slower' version pushing 13.6CFM at just 35.1dBA. Note that you cannot use the B1+6 that DigiKey stocks. The B1+6 is a 4-wire PWM, and the ICX's 12V control will burn up the motor. - Mechatronics MH4028L12B1-RSR - PLEASE TELL US ALL IF YOU FIND A SOURCE!
These. Are. GLORIOUS. 40x40x28mm, 12.81CFM, 0.37in H2O, 39dBA, I just LOVE these fans. But nobody stocks the low speed version. And the next step up (the M) is already over 45dBA. But they have amazing harmonics due to the 4028L being less than half the speed the frame was designed for (Max 16,500RPM!) - NMB-MAT 1608VL-04W-B69-B00 - $10.47 @ Digikey
These are an excellent 'middle of the middle' choice. 40x40x20mm, 11.3CFM, 0.399in H2O, 34.6dBA @ 1m at 9500RPM, and 40k hour lifetime. They're also generally stocked by multiple vendors. Really great fans for general use, but believe me, that 40k hour lifetime is near spot on - expect to replace every 3 years or so. - Sunon PSD1204PHB1-A(2).Z.F.PWM.GN "Tiny Terror" - also currently unobtainable
These are new design MagLev parts, and really impressive. 40x40x15mm (so the thinnest here,) but 14CFM, 0.63in H2O, and 44.2dBA @ 1m at 12,000RPM. They're also a LOT easier than the Mechatronics; Sunon MOQ is just 30 for a non-custom part, expect around $10-12/ea. These are awesome fans if you can get your hands on them and can stand the noise.
I also don't recommend SanAce because the PSUs you all complain about on the 6610's? Those are SanAce fans. SanAces have a very distinct harmonic that everyone hates. That harmonic is also why SanAces perform the way they do. You can't have one without the other. If you think those 6610's are bad, at least there is much, much worse.
As far as selecting fans? The closer to stock flow, the better, however you achieve it. Installing 10CFM peak fans in an ICX6450-48 or 48P is a net cooling reduction of over 70%! So, you know, don't do that unless you want to run very hot or are not using POE! This is why I make angry noises at people who think they can just swap fans around in equipment like this. For the love of god people, the ICX6610 fan modules? 2x Delta FFB0412UHN's per module, that's over 50CFM per module, over 100CFM excluding the power supplies! Nobody puts that kind of brute force in unless they NEED it. Especially as those FFBs are close to $20/each at quantity.
But as I said: how you get there doesn't matter as long as you get there or accept the trade-off. Ducting a pair of 45CFM 120mm fans on the side of your 6450-48P? That'd work too. No, seriously! (Won't work on the exhaust side because you just don't have enough opening and it's too much pressure loss.) Add more, quieter fans? Also valid. Strap tiny quiet fans to the heatsinks? Absolutely will help. Only care about performance at all costs? Throw in some contra-rotating fans! Want to get stock cooling performance out of a 6450-48? Cut out 2 more fan holes, drop in 3 x 9CFM, dead silent and ready to rock.
Last edited:
Well, Ruckus has released a couple of months ago the 08.0.30u... Not many differences, maybe a couple of minor bugfixes, but still.
I don't doubt your experience in the field, thanks for the information and recommendations that you provided.
But for me the much lower rated KDE1204PKV3 (regarding CFM, static pressure) are good enough. For my home usage "good enough" means they can keep my switches at homelab use at my usual ambient temperatures cool enough, i.e. the switches keep the fans running at speed 1 most of the time. If the fans ramp up to speed 2 they stay there for a few minutes to bring temps down to 59 celsius and are then automatially throttled back to speed 1.
My aim is to keep the switches reasonably cool at a sound level that I can tolerate next to my desk. In addition to the inferior internal fans in the switches I am actively exhausting warm air in the rack using silent 12" fans. *
I picked up a ICX6610-48P, followed the fine instructions from @fohdeesha but when I restarted the switch after resetting to factory defaults, I'm getting the following and the switch just hangs... any suggestions?
ICX Boot Code Version 10.1.00 (grz10100)
Enter 'a' to stop at memory test
Enter 'b' to stop at boot monitor
BOOT INFO: load monitor from boot flash, cksum = 71f1
BOOT INFO: verify flash files.....
BOOT INFO: load image from primary copy...
platform type = 12
PCIE-1 LTSSM status: 22
PCIE Switch status: 0
..............................
Firmware integrity checksum passed
.......
Starting Main Task .Applying factory defaults..
INFO: startup config data is not available, try to read from backup
INFO: startup config data in the backup area is not available
..CPSS DxCh Version: cpss3.4p1 release
ICX Boot Code Version 10.1.00 (grz10100)
Enter 'a' to stop at memory test
Enter 'b' to stop at boot monitor
BOOT INFO: load monitor from boot flash, cksum = 71f1
BOOT INFO: verify flash files.....
BOOT INFO: load image from primary copy...
platform type = 12
PCIE-1 LTSSM status: 22
PCIE Switch status: 0
..............................
Firmware integrity checksum passed
.......
Starting Main Task .Applying factory defaults..
INFO: startup config data is not available, try to read from backup
INFO: startup config data in the backup area is not available
..CPSS DxCh Version: cpss3.4p1 release
wait a few minutes. at that point it takes like another 2 or 3 minutes to boot and it won't have any output
I've been holding off on updating the guide to use 8030u as initially the brocade subreddit and the ICX usergroup on Ruckus's support site kept giving vague reports that it was "buggy" and to just stay on 8030t. However I've been running it at home and in production for about a month now and haven't ran into any issues so I will probably upgrade the guide to it soon.
As for the icx7xxx series guides, still deciding what to do there as I'd like to recommend the latest 8092 train, however it's distributed differently and HAS to be flashed using a UFI image. The fun part: versions before 8080e don't support flashing UFI images. This makes the guide very difficult to write as the process will completely depend on what version your switch came with, I'd basically have to write two guides: one for people on firmware earlier than 8080e, in which case they'll have to jump to 8080e first, then to 8092, and another guide for people already on 8080e or later
I take pride in how simple and straightforward my docu is and this kind of mess/multiple paths breaks pretty much all my rules, so I'm trying to work out something simpler that covers everyone's switch regardless of state. It will probably involve flashing one of the later/latest bootloaders, then flashing the latest UFI 8092 image from within the bootloader - that way it won't rely on the user having a specific version of fastiron with UFI flashing support
Odd. Followed that guidance and left it for the evening. I ultimately rebooted. It came up as before and hung again.
I'm assuming (yeah I know
) that the image file is the only file that needs to be under "primary" directory. Would it make any sense to try and boot it via TFTP?It shouldn't take more than 2-3 minutes to fully boot the switch. Maybe open the chassis and see if it's dirty? Reinsert fans and PSUs. It almost behaves as if some storage is marginal and not bad enough to spew errors in the console. @fohdeesha would be the best person to suggest commands in the boot loader to validate if there are any problems in the flash devices.Odd. Followed that guidance and left it for the evening. I ultimately rebooted. It came up as before and hung again.
I'm assuming (yeah I know
Here is the beginning from when I first got the switch
ICX Boot Code Version 10.1.00 (grz10100)
Enter 'a' to stop at memory test
Enter 'b' to stop at boot monitor
BOOT INFO: load monitor from boot flash, cksum = 71f1
BOOT INFO: verify flash files......
BOOT INFO: load image from primary copy...
platform type = 12
PCIE-1 LTSSM status: 22
PCIE Switch status: 0
..............................
Firmware integrity checksum passed
.......
Starting Main Task .Applying factory defaults..
INFO: startup config data is not available, try to read from backup
INFO: startup config data in the backup area is not available
..CPSS DxCh Version: cpss3.4p1 release
Pre Parsing Config Data ...
INFO: empty config data in the primary area, try to read from backup
INFO: empty config data in the backup area also
Parsing Config Data ...
INFO: empty config data in the primary area, try to read from backup
INFO: empty config data in the backup area also
System initialization completed...console going online.
Copyright (c) 1996-2015 Brocade Communications Systems, Inc. All rights reserved.
UNIT 1: compiled on Jul 31 2015 at 02:52:14 labeled as FCXR08030b
(10436300 bytes) from Primary FCXR08030b.bin
SW: Version 08.0.30bT7f3
Boot-Monitor Image size = 370695, Version:10.1.00T7f5 (grz10100)
HW: Stackable ICX6610-48-HPOE
Here's the beginning after first reboot
ICX Boot Code Version 10.1.00 (grz10100)
Enter 'a' to stop at memory test
Enter 'b' to stop at boot monitor
BOOT INFO: load monitor from boot flash, cksum = 71f1
BOOT INFO: verify flash files.........
BOOT INFO: load image from primary copy...
platform type = 12
PCIE-1 LTSSM status: 22
PCIE Switch status: 0
..............................
Firmware integrity checksum passed
.......
Starting Main Task ...CPSS DxCh Version: cpss3.4p1 release
Pre Parsing Config Data ...
Parsing Config Data ...
System initialization completed...console going online.
Copyright (c) 1996-2015 Brocade Communications Systems, Inc. All rights reserved.
UNIT 1: compiled on Jul 31 2015 at 02:52:14 labeled as FCXR08030b
(10436300 bytes) from Primary FCXR08030b.bin
SW: Version 08.0.30bT7f3
Boot-Monitor Image size = 370695, Version:10.1.00T7f5 (grz10100)
HW: Stackable ICX6610-48-HPOE
Mine seems to halt right after
..CPSS DxCh Version: cpss3.4p1 release
I let it run for several hours, nothing after the line above. Rebooted the switch and it pauses in the same location. It would appear as though something is not recreating the config files and it cannot proceed forward into the console.
Spent the last 3 days going through this monster of a thread (and still only 2/3 way through). I think I will pull the trigger soon on a 6450-24p but have a question:
Currently my main core switch is a Netgear 24 port managed switch with 2 ports trunked to Unifi 8-port POE switches (for locations with limited ethernet ports). I also have 3 VLANs managed by pfsense. If I replace the Netgear switch with the 6450-24p, and enable L3 switching, do I also need to replace the Unifi switches, since they're only L2? In other words, must the network consist of all L3 switches for proper VLAN routing?
Currently my main core switch is a Netgear 24 port managed switch with 2 ports trunked to Unifi 8-port POE switches (for locations with limited ethernet ports). I also have 3 VLANs managed by pfsense. If I replace the Netgear switch with the 6450-24p, and enable L3 switching, do I also need to replace the Unifi switches, since they're only L2? In other words, must the network consist of all L3 switches for proper VLAN routing?
No, it's not necessary, you can route through VLANs on the 6450, and the other switches simply relay traffic on the trunk ports. It's a common architecture on a "collapsed core campus network" (access on L2, trunking on L3 core routing).
Be aware that pfSense cannot lease out DHCP addresses on VLANs not directly connected to itself AFAIK, if that's a thing for you.
Last edited: