Drag to reposition cover

Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

Spearfoot

Active Member
Apr 22, 2015
113
49
28
(Is there no way to search in a specific thread? I could not find!)
There is a way to search a specific thread: click the 'Search' box and select 'This Thread' from the combobox. Invaluable for digging through the "Brocade Mega Thread"!
sthsearch.jpg
Question: I'm trying to use a couple cisco DAC cables. When I plug a cable in to my 6450, even if i plug it into two ports, i get a message "optics inserted, wait for PHY initialization", and nothing else happens.
Should I be able to loop 1/2/2 and 1/2/3 for testing, at least to get a link?
I'm not a network guru, but I think connecting a single cable between two ports on your switch creates a loop, with attendant bad juju. Do you have another switch or computer you can test with instead?
 

gregsachs

Active Member
Aug 14, 2018
426
125
43
There is a way to search a specific thread: click the 'Search' box and select 'This Thread' from the combobox. Invaluable for digging through the "Brocade Mega Thread"!
View attachment 15593
I'm not a network guru, but I think connecting a single cable between two ports on your switch creates a loop, with attendant bad juju. Do you have another switch or computer you can test with instead?
Thank you! That totally eluded me.
Yes, I'm aware I will create a loop, but I was thinking I'd get a link at least. I don't get a link....I do have a powered down Aruba S2500, but would need to reset it to avoid IP conflicts...
 

Vesalius

Active Member
Nov 25, 2019
151
114
43
Those of us with Ruckus AP’s that have unleashed firmware are stuck with vlan 1 for management as well. their zone directors can change the management vlan tag, but not unleashed. I use dual-mode for those. Thinking of changing the default vlan on my 6450 so that I can prpoerly segregate management interfaces to vlan 1 given those constraints.
In regards to changing the default vlan, what can I expect to happen to all the ports currently automatically untagged in vlan 1 once I designate another vlan as default. Will they stay or move to the new default?
assuming the command below, as long as the vlan ID I choose is not already in use, should be all it takes:

Code:
default-vlan-id vlan-id
 

Scarlet

Member
Jul 29, 2019
63
22
8
Will they stay or move to the new default?
The ports will move to the new default. I moved my default-vlan to 4095 because of the Ruckus APs, and yes the command is what is needed to change the default vlan. I don't remember if the switches actually needed a reset after that.
 

kache

New Member
Jun 27, 2020
11
2
3
Replaced the stock fans in the icx6450 with the Noctua NF-A4x20 FLX and the sound is truly amazing now. The rack is nearly silent and there is no annoying alternating sound like with the Foxconn ones.
Problem is that despite there being only 2 PoE devices the temperatures are quite high:
Code:
SSH@10G#show chassis
The stack unit 1 chassis info:

Power supply 1 (NA - AC - PoE) present, status ok
Power supply 2 not present
Power supply 3 not present

Fan 1 ok, speed (auto): [[1]]<->2
Fan 2 ok, speed (auto): [[1]]<->2
Fan 3 ok, speed (auto): [[1]]<->2

Fan controlled temperature: 63.5 deg-C

Fan speed switching temperature thresholds:
                Speed 1: NM<----->65       deg-C
                Speed 2:       56<-----> 79 deg-C (shutdown)

Sensor B Temperature Readings:
        Current temperature : 60.5 deg-C
Sensor A Temperature Readings:
        Current temperature : 63.5 deg-C
        Warning level.......: 69.0 deg-C
        Shutdown level......: 79.0 deg-C
Waiting now for the Sunon Maglev to arrive and will test with those to see if I can have a less annoying sound coupled with decent temperatures.
 
  • Like
Reactions: tommybackeast

Scarlet

Member
Jul 29, 2019
63
22
8
The Temperatures are fine, you probably won't go much lower with Sunon Maglev (I have the sunons in my 6450s and they sometimes go to fan speed 2 on warm days, i.e. > 25 celsius room temperature, the switches are in closed network racks).

This is my top-of-server-rack 6450-24p at 22 celsius room temperature cooled by two sunon maglevs:
Code:
Power supply 1 (NA - AC - PoE) present, status ok
Power supply 2 not present

Fan 1 ok, speed (auto): [[1]]<->2
Fan 2 ok, speed (auto): [[1]]<->2

Fan controlled temperature: 61.5 deg-C

Fan speed switching temperature thresholds:
                Speed 1: NM<----->64       deg-C
                Speed 2:       59<-----> 88 deg-C (shutdown)

Sensor B Temperature Readings:
        Current temperature : 60.5 deg-C
Sensor A Temperature Readings:
        Current temperature : 61.5 deg-C
        Warning level.......: 78.0 deg-C
        Shutdown level......: 88.0 deg-C
 

Fallen Kell

Member
Mar 10, 2020
45
13
8
Having owned these (and other enterprise switches) for years, and running them in both, a business and home environment, I can almost guarantee that "dual mode" has a very edge case, in any configuration.

If you think you need dual mode, think again about your network topology. You're trying to solve a problem with a machete when it really needs a scalpel.
I believe I meet that edge case, but might be able to configure its usage out of the config, but this certainly makes things easier. For the 3 places I am using it, one is to connect another "switch" (a wireless AP) which has both tagged and untagged data going to/from it and I want the untagged to go to a specific VLAN. The next is a server supporting virtual machines where-in I have created bridged interface devices to specific VLANs for them to reside, but the untagged interface is for the host itself (I can probably reconfigure this, but it is much easier to know that all the tagged interfaces are in use by VMs). The last use case will be for a firewall-on-a-stick using a 40gb connection to a pfsense box with all the VLANs configured (routing will only happen for the LAN/WAN, but I need all the VLANs for it to support NAT/DHCP/DNS for all my devices across all VLANs) and using the default untagged VLAN makes it easier for me to also support the other untagged VLAN devices.
 

rootwyrm

Member
Mar 25, 2017
48
59
18
www.rootwyrm.com
So, I've run into a bit of a headscratcher. Maybe one of you has seen it. I'm standing up a new segment on a 6450 with OSPF. Except... it's not actually working. This isn't the only part of my environment with FRR 7, but it's the only part doing OSPF with an ICX - the rest is BGP. I'm seeing the Hello packets from the ICX just fine, traffic passes just fine, but adjacency is never established in area 0, even with authentication disabled. Seeing the same problem in non-broadcast and point-to-point as well. Any ideas?

edit: derp, I forgot to mention, this configuration is working with a Juniper, which is why I'm suspecting either a quirk or something not in the docs about OSPF on 08.0.30. And full debug logs on both sides show no errors, but the frr7 side also isn't reporting the HELLO packet at all.

Code:
vlan 1 name DEFAULT-VLAN by port
router-interface ve 1

interface ve 1
ip address 10.1.0.5 255.255.254.0
ip ospf area 0
ip ospf active
ip ospf network non-broadcast

router ospf
area 0
area 0 range 10.53.0.0 255.255.255.0 advertise
summary-address 10.1.0.5 255.255.254.0
redistribute connected
redistribute static
log all
neighbor 10.1.0.20
neighbor 10.1.0.82
Code:
router ospf
  ospf router-id 10.1.0.20
  log-adjacency-changes
  redistribute connected
  network 10.53.0.20/32 area 0
  neighbor 10.1.0.5
 
Last edited:

ntrlsur

New Member
Sep 3, 2020
1
0
1
Just picked up a 6450-24 to replace a Cisco 3560 and an HP S5800. Was using the S5800 strictly for 10gig ports and the Cisco for everything else. Looking forward to getting it all on one device. Specifically looking forward to inter-vlan routing.
 

sth

Active Member
Oct 29, 2015
340
71
28
How bad is the fan noise?
Little bit louder than the 7450-32ZP it’s replacing, too loud to work next to for sure though. It’s in a remote part of our home in a APC netshelter rack so it’s acceptable.
 

xerxies

New Member
Oct 30, 2018
3
0
1
30
Saint Joseph, Missouri
I apologize if this is listed somewhere in here, I wasn't able to find a passive DAC max length for the 6610s.
I'm not sure if I'm running to far of a length for a passive DAC on this switch or if I've got something else screwy going on.

I ordered two of these QSFP-4SFP10G-CU5M (40GBASE-CR4 QSFP to 4 10GBASE-CU SFP+ 5m passive) to connect the dual NICs on my 3 vSphere hosts and NAS. The NICs are different brands, so that may be problematic as well. The NAS has an old ConnectX, two hosts have Emulex 00JY823s, and the third host has a Dell branded x520-DA2.

All four NICs got single SFP+ breakout from both 40Gb interfaces. I got link lights on all connections, and initially only 3 interfaces from each breakout would show up and pass traffic. If I started pushing a lot of throughput, they would drop on both sides and only unplugging/replugging them would get them to reconnect.

Using any of the eight SFP+ ports on the front with 3m Cisco 10Gb DACs works fine in any combination of port or NIC, so I don't think it's an OEM compatibility issue?

If at all possible, I'd like to save as many of these front ports as I can for the occasional 10GbBase-t runs in my house.
 

kache

New Member
Jun 27, 2020
11
2
3
Little bit louder than the 7450-32ZP it’s replacing, too loud to work next to for sure though. It’s in a remote part of our home in a APC netshelter rack so it’s acceptable.
Thank you for the answer!
So if I understood correctly you're using it as Core switch and the APs are connected directly to it rather than to the edge switches, correct?
 

ptibeur

New Member
Sep 4, 2020
5
0
1
Hi all ;)

Jumping aboard the Brocade party train :D If everything goes just as planned, I should be getting a used ICX-6450-48 soon. I spotted a used 7250-48P but never got answers from the seller :/

I'm now digging this awesome thread to find :
- the most silent fans suitable for this switch (I've seen Sunon, Noctua, maybe others are good)
- a recommended stable software version
- a way to enable the 4x10G ports, if not already active. I'll see when I get the switch what's already installed inside

Let's go ! :p
 

kache

New Member
Jun 27, 2020
11
2
3
The Temperatures are fine, you probably won't go much lower with Sunon Maglev (I have the sunons in my 6450s and they sometimes go to fan speed 2 on warm days, i.e. > 25 celsius room temperature, the switches are in closed network racks).

This is my top-of-server-rack 6450-24p at 22 celsius room temperature cooled by two sunon maglevs:
Code:
Power supply 1 (NA - AC - PoE) present, status ok
Power supply 2 not present

Fan 1 ok, speed (auto): [[1]]<->2
Fan 2 ok, speed (auto): [[1]]<->2

Fan controlled temperature: 61.5 deg-C

Fan speed switching temperature thresholds:
                Speed 1: NM<----->64       deg-C
                Speed 2:       59<-----> 88 deg-C (shutdown)

Sensor B Temperature Readings:
        Current temperature : 60.5 deg-C
Sensor A Temperature Readings:
        Current temperature : 61.5 deg-C
        Warning level.......: 78.0 deg-C
        Shutdown level......: 88.0 deg-C

Thank you! It does appear to stay around 56-65°C during the day, with the 65°C being more in the evening.
Only now I'm seeing it on speed 2:
Code:
SSH@10G#show chassis
The stack unit 1 chassis info:

Power supply 1 (NA - AC - PoE) present, status ok
Power supply 2 not present
Power supply 3 not present

Fan 1 ok, speed (auto): 1<->[[2]]
Fan 2 ok, speed (auto): 1<->[[2]]
Fan 3 ok, speed (auto): 1<->[[2]]

Fan controlled temperature: 64.0 deg-C

Fan speed switching temperature thresholds:
                Speed 1: NM<----->65       deg-C
                Speed 2:       56<-----> 79 deg-C (shutdown)

Sensor B Temperature Readings:
        Current temperature : 60.5 deg-C
Sensor A Temperature Readings:
        Current temperature : 64.0 deg-C
        Warning level.......: 69.0 deg-C
        Shutdown level......: 79.0 deg-C
Though it's interesting that your temperature thresholds are different. Is it because yours is a 24p while mine is a 48p?


Hi all ;)

Jumping aboard the Brocade party train :D If everything goes just as planned, I should be getting a used ICX-6450-48 soon. I spotted a used 7250-48P but never got answers from the seller :/

I'm now digging this awesome thread to find :
- the most silent fans suitable for this switch (I've seen Sunon, Noctua, maybe others are good)
- a recommended stable software version
- a way to enable the 4x10G ports, if not already active. I'll see when I get the switch what's already installed inside

Let's go ! :p
- Well, the most silent are always the Noctua that I have (Noctua NF-A4x20 FLX) and judging from my conversation with Scarlet they appear to be as efficient for cooling as the Sunon. The issue that was indicated in the previous posts (the fans not spinning at speed 1) does not appear to happen with these specific Noctua Fans. Don't forget that you have to swap the wires on the connector before installing them!
- The latest. Follow the instructions here: ICX6430 / ICX6450 - Fohdeesha Docs
- Send a PM to @fohdeesha , he will take care of you for the licenses. ;)




BTW, concerning the Noctua, thanks to the Noctua + manual fan management of the 2 R720 via ipmitool my rack is now basically silent!!!! :D :D :D
 
  • Like
Reactions: ptibeur

rootwyrm

Member
Mar 25, 2017
48
59
18
www.rootwyrm.com
I'm now digging this awesome thread to find :
- the most silent fans suitable for this switch (I've seen Sunon, Noctua, maybe others are good)
- a recommended stable software version
- a way to enable the 4x10G ports, if not already active. I'll see when I get the switch what's already installed inside
So in order...
- the best fans for this are unobtanium due to COVID-19, but don't use Noctua. Noctua is overpriced junk - it has the lowest flow rate and lowest static pressure. Anything from Sunon or NMB or Nidec or Delta is fine. They also cost far less.
- 08.0.30t for the 6450 is the maximum possible
- see first post for that, but if the switch was only configuration reset and not factory reset it may have them installed. Do a show licenses first always.

Edit: to expound upon my 'unobtanium' fan choice, bear in mind, I have been doing systems integration for 30 years and have VERY extensive rackmount equipment experience. I know why Brocade chose the fans they did, I have worked with those fans, and they are the same fans I would have chosen. (BTW, at speed 2 the 6450-40P is attempting to move 90.9CFM. Yes, each of those 40mm fans is rated for 30.3CFM free air.)
If you want to quiet down these units, the BIGGEST change you can make is to remove the stamped grill with a Dremel. It's all sharp edges, and significantly blocks the hub as well. Turbulence from things like grills are what generate serious noise. If you still need protection, tack on a wire grill on the OUTSIDE of the chassis. HOWEVER, this will result in slightly reduced cooling to the front right corner. I won't bother going into my unobtanium choice, because unless you're ready to order at least 250 of them, you can't buy them.

So here's some fans you can buy that I'd recommend.
  • Delta EFB0412VHD-F00 - $12.40/ea @ Digikey
    There's a REASON Delta's a top pick for ODM and OEM. 40x40x20mm, 10.1CFM, 0.416in H2O (which is insane,) 32.5dBA @ 1m, rated for 70k hours at 50C. Make sure it's THAT part number and not the Rev C, which is a vastly inferior part.
  • Comair Rotron "Gryphon" GDA4028-12BB - discontinued, alas
    If you can find these? BUY THESE. 40x40x28mm, but 11CFM, 0.34in H2O, 8800RPM, but just 31.4dBA! I am still mad they discontinued them. No, I am not selling any of my spares.
  • Mechatronics MR4020E12B1-RSR - $7.50 @ Digikey (when in stock)
    READ THE PAGE. These are non-stocked currently! However these offer excellent balance between flow and noise. 40x40x20mm, 15.8CFM, 0.45in H2O, and 39dBA @ 1m at 11000RPM. (Yes, it's about the quietest 11k fan ever made.) You need the E12B1 though - NOT the B2. The B1 has a tach, the B2 is a rotor lock wire.
  • Mechatronics MR4020H12B1-RSR - about $8
    The 'slightly slower' version pushing 13.6CFM at just 35.1dBA. Note that you cannot use the B1+6 that DigiKey stocks. The B1+6 is a 4-wire PWM, and the ICX's 12V control will burn up the motor.
  • Mechatronics MH4028L12B1-RSR - PLEASE TELL US ALL IF YOU FIND A SOURCE!
    These. Are. GLORIOUS. 40x40x28mm, 12.81CFM, 0.37in H2O, 39dBA, I just LOVE these fans. But nobody stocks the low speed version. And the next step up (the M) is already over 45dBA. But they have amazing harmonics due to the 4028L being less than half the speed the frame was designed for (Max 16,500RPM!)
  • NMB-MAT 1608VL-04W-B69-B00 - $10.47 @ Digikey
    These are an excellent 'middle of the middle' choice. 40x40x20mm, 11.3CFM, 0.399in H2O, 34.6dBA @ 1m at 9500RPM, and 40k hour lifetime. They're also generally stocked by multiple vendors. Really great fans for general use, but believe me, that 40k hour lifetime is near spot on - expect to replace every 3 years or so.
  • Sunon PSD1204PHB1-A(2).Z.F.PWM.GN "Tiny Terror" - also currently unobtainable
    These are new design MagLev parts, and really impressive. 40x40x15mm (so the thinnest here,) but 14CFM, 0.63in H2O, and 44.2dBA @ 1m at 12,000RPM. They're also a LOT easier than the Mechatronics; Sunon MOQ is just 30 for a non-custom part, expect around $10-12/ea. These are awesome fans if you can get your hands on them and can stand the noise.
And while we're here, let's talk about the Sunon KDE MagLevs. Which will be a very short talk: KDE MagLevs are terrible for this application. They're cheap, plentiful, and useless. They are NOT designed for applications like this. The KDE1204PKVX that you can find anywhere and everywhere? It is objectively the worst. 10.8CFM, 27.5dBA, sounds great, right? It also has a static pressure of 0.27in H2O. These were designed for very free flow applications. The ICX chassis grills are the opposite.
I also don't recommend SanAce because the PSUs you all complain about on the 6610's? Those are SanAce fans. SanAces have a very distinct harmonic that everyone hates. That harmonic is also why SanAces perform the way they do. You can't have one without the other. If you think those 6610's are bad, at least there is much, much worse.

As far as selecting fans? The closer to stock flow, the better, however you achieve it. Installing 10CFM peak fans in an ICX6450-48 or 48P is a net cooling reduction of over 70%! So, you know, don't do that unless you want to run very hot or are not using POE! This is why I make angry noises at people who think they can just swap fans around in equipment like this. For the love of god people, the ICX6610 fan modules? 2x Delta FFB0412UHN's per module, that's over 50CFM per module, over 100CFM excluding the power supplies! Nobody puts that kind of brute force in unless they NEED it. Especially as those FFBs are close to $20/each at quantity.
But as I said: how you get there doesn't matter as long as you get there or accept the trade-off. Ducting a pair of 45CFM 120mm fans on the side of your 6450-48P? That'd work too. No, seriously! (Won't work on the exhaust side because you just don't have enough opening and it's too much pressure loss.) Add more, quieter fans? Also valid. Strap tiny quiet fans to the heatsinks? Absolutely will help. Only care about performance at all costs? Throw in some contra-rotating fans! Want to get stock cooling performance out of a 6450-48? Cut out 2 more fan holes, drop in 3 x 9CFM, dead silent and ready to rock.
 
Last edited:

Scarlet

Member
Jul 29, 2019
63
22
8
And while we're here, let's talk about the Sunon KDE MagLevs. Which will be a very short talk: KDE MagLevs are terrible for this application. They're cheap, plentiful, and useless.
I don't doubt your experience in the field, thanks for the information and recommendations that you provided.

But for me the much lower rated KDE1204PKV3 (regarding CFM, static pressure) are good enough. For my home usage "good enough" means they can keep my switches at homelab use at my usual ambient temperatures cool enough, i.e. the switches keep the fans running at speed 1 most of the time. If the fans ramp up to speed 2 they stay there for a few minutes to bring temps down to 59 celsius and are then automatially throttled back to speed 1.

My aim is to keep the switches reasonably cool at a sound level that I can tolerate next to my desk. In addition to the inferior internal fans in the switches I am actively exhausting warm air in the rack using silent 12" fans. *
 

KevinS

New Member
Sep 5, 2020
3
0
1
I picked up a ICX6610-48P, followed the fine instructions from @fohdeesha but when I restarted the switch after resetting to factory defaults, I'm getting the following and the switch just hangs... any suggestions?

ICX Boot Code Version 10.1.00 (grz10100)
Enter 'a' to stop at memory test
Enter 'b' to stop at boot monitor
BOOT INFO: load monitor from boot flash, cksum = 71f1
BOOT INFO: verify flash files.....
BOOT INFO: load image from primary copy...

platform type = 12
PCIE-1 LTSSM status: 22
PCIE Switch status: 0
..............................
Firmware integrity checksum passed
.......
Starting Main Task .Applying factory defaults..
INFO: startup config data is not available, try to read from backup
INFO: startup config data in the backup area is not available
..CPSS DxCh Version: cpss3.4p1 release
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,271
2,341
113
30
fohdeesha.com
I picked up a ICX6610-48P, followed the fine instructions from @fohdeesha but when I restarted the switch after resetting to factory defaults, I'm getting the following and the switch just hangs... any suggestions?

ICX Boot Code Version 10.1.00 (grz10100)
Enter 'a' to stop at memory test
Enter 'b' to stop at boot monitor
BOOT INFO: load monitor from boot flash, cksum = 71f1
BOOT INFO: verify flash files.....
BOOT INFO: load image from primary copy...

platform type = 12
PCIE-1 LTSSM status: 22
PCIE Switch status: 0
..............................
Firmware integrity checksum passed
.......
Starting Main Task .Applying factory defaults..
INFO: startup config data is not available, try to read from backup
INFO: startup config data in the backup area is not available
..CPSS DxCh Version: cpss3.4p1 release
wait a few minutes. at that point it takes like another 2 or 3 minutes to boot and it won't have any output
 
  • Like
Reactions: Jason Antes

fohdeesha

Kaini Industries
Nov 20, 2016
2,271
2,341
113
30
fohdeesha.com
Well, Ruckus has released a couple of months ago the 08.0.30u... Not many differences, maybe a couple of minor bugfixes, but still.
I've been holding off on updating the guide to use 8030u as initially the brocade subreddit and the ICX usergroup on Ruckus's support site kept giving vague reports that it was "buggy" and to just stay on 8030t. However I've been running it at home and in production for about a month now and haven't ran into any issues so I will probably upgrade the guide to it soon.

As for the icx7xxx series guides, still deciding what to do there as I'd like to recommend the latest 8092 train, however it's distributed differently and HAS to be flashed using a UFI image. The fun part: versions before 8080e don't support flashing UFI images. This makes the guide very difficult to write as the process will completely depend on what version your switch came with, I'd basically have to write two guides: one for people on firmware earlier than 8080e, in which case they'll have to jump to 8080e first, then to 8092, and another guide for people already on 8080e or later

I take pride in how simple and straightforward my docu is and this kind of mess/multiple paths breaks pretty much all my rules, so I'm trying to work out something simpler that covers everyone's switch regardless of state. It will probably involve flashing one of the later/latest bootloaders, then flashing the latest UFI 8092 image from within the bootloader - that way it won't rely on the user having a specific version of fastiron with UFI flashing support