Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[adman_c]

Good news everyone!

The Fan problem is now essentially SOLVED.

I just received my Arctic S4028-6K value pack.

Initial testing shows the new Arctic serie is the perfect fan for the 3-pin brocades. here's why:

before i am running 1x original fan with my tape mod. icx6450-48p completely idle with one hour running time:

#show chassis
The stack unit 1 chassis info:

Power supply 1 (NA - AC - PoE) present, status ok
Power supply 2 not present
Power supply 3 not present

Fan 1 failed
Fan 2 failed
Fan 3 failed

Fan controlled temperature: 55.0 deg-C

Fan speed switching temperature thresholds:
                Speed 1: NM<----->65       deg-C
                Speed 2:       56<-----> 79 deg-C (shutdown)

Sensor B Temperature Readings:
        Current temperature : 49.0 deg-C
Sensor A Temperature Readings:
        Current temperature : 55.0 deg-C
        Warning level.......: 69.0 deg-C
        Shutdown level......: 79.0 deg-C

Now i receive my 4028-6K value pack. Plus and minus wires needs to be switched. A value pack at german amazon.de is just under 30 EUR, includes 5 fans and 20 screws. Immediately as i open the pack i realize it's the real deal.

the 6K version are too slow for heavy usage, but as i'm writing this my 6450-48p is running at idle with 3x arctic 4028-6K's in ABSOLUTE ****ING SILENCE.

show chassis
The stack unit 1 chassis info:

Power supply 1 (NA - AC - PoE) present, status ok
Power supply 2 not present
Power supply 3 not present

Fan 1 ok, speed (auto): [[1]]<->2
Fan 2 ok, speed (auto): [[1]]<->2
Fan 3 ok, speed (auto): [[1]]<->2

Fan controlled temperature: 63.5 deg-C

Fan speed switching temperature thresholds:
                Speed 1: NM<----->65       deg-C
                Speed 2:       56<-----> 79 deg-C (shutdown)

Sensor B Temperature Readings:
        Current temperature : 56.5 deg-C
Sensor A Temperature Readings:
        Current temperature : 63.5 deg-C
        Warning level.......: 69.0 deg-C
        Shutdown level......: 79.0 deg-C

again, i repeat: ABSOLUTE SILENCE.
i have to put my ear right next to the fan outlets to hear that anything is going on.

Obviously, the Arctic 6K versions are insufficient for anything heavier that idle, i'm only a few degrees celcius for it kicking into high gear

BUT

first of all the arctic's are both voltage AND PWM controlled. 4.5V to 12V

a 5 fan value pack sells on amazon for 30 EUR, which includes 20 screws. That is great value. I'm gonna order the 15K version next, and depending on my final load, add 15K Arctics as necessary.

Again, i'm running my 6450-48p atm at ABSOLUTE SILENCE. presumably according to the specs they are running at 1400 RPM atm, at 4.5V lol.

Point is immediately as i received the 6k versions today i could feel holding them they are gonna be the perfect fans for the brocades, and remember, they are BOTH voltage AND PWM controlled, [4 pin] so users of the big beef qsfp switches take notice, it's possible these could serve as straight replacements!

this is a special alert! The perfect fans for all the brocade switches might very well have now been identified!!!!

View attachment 23028View attachment 23029View attachment 23030View attachment 23032

Intriguing. I wonder if a pair of the 6k models can adequately cool the 7250-24p or if they'd be constantly cycling between high and low speeds.

 

[sergi0]

I think you would need to undo the tagging and dual modes on 15 and 17 first.

Tremendous : it worked. I remove the config via GUI and then do it inside a terminal. Everything was configured as your conf. I did a couple of test, I get strange results when configuring using the GUI, but why bother if it works via terminal.
I did a couple of stress test and so far no problem.

ONE thousand THANKS for everyone that helps, especially Lone Wolf and Fohdeesha. I am also glad to have learn a couple of things too .

 

[heromode]

Intriguing. I wonder if a pair of the 6k models can adequately cool the 7250-24p or if they'd be constantly cycling between high and low speeds.

An update on my situation: Today on a warm summer day the fans did throttle up with switch still all idle, only one gigabit ethernet cable connected. So at low mode these don't move enough air. When they spin up they are audible, not noisy though. Nothing even close to one original fan at low speed.

It would be interesting to hear what the 15K version sound at low speed, but atm i'm planning on ordering 3x noctua nf-a4x10 flx, which are 10mm deep, and attach them to each ASIC heatspreader, and feed the directly from the 12V line coming from the PSU, and maybe even use the included voltage reduction adapter. I already have the T-Splitters to get 12V.

I believe the combination of 3 fans directly over the ASIC's and the 3x Arctic should get me a silent switch. The best thing about the S4028-6K's is they work perfectly at both 4.5V and 12V, and when they do spin up to high speed they actually move air decently, and the switch cools down, throttling down after a while.

Today i've been experimenting with adding the 2 extra arctic's from the valuepack inside the case, in the free area between the PCB and the rear, and blowing towards the rear in the same direction as other fans. I've strapped them with zip-ties to the PCB screws beneath the PCB. I'm missing some wires like splitted 3-pin fan cable extenders so i'm having to improvise some just to test. So far i have the 4th Arctic working by feeding it only voltage and ground from one fan header. I'm not connecting the tach pin so not to mess up the signal. It works and the 4th fan starts at full and throtthles down to 4.5V just as the other fans.

I'm still working on getting the other, 5th fan connected, and while i think this will help abit, i'm still probably gonna go with 3x noctua's directly on the ASIC chips. That combination of 3x noctua's and 3x Arctics i believe will result in a very quiet switch that still has the ability to ramp up the rear fans incase of heavy load.



Update. Now i have the 2 extra Arctic's connected, and the cover back on, so in an hour i'll know how much they help. It's not a silent switch, there is some minor noise at low speed now. You can hear it in a quiet room standing 1 meter away, but it's acceptable to me. Somehow attaching plus and ground to extra fans seems to have changed the sound of the rear fans abit, i dunno if i'm imagining it..

Update2. Now it's been running idle for an hour, and the temps have stabilized:

Temps kept rising even after an hour, and seem to have settled now.. This depends on window open or close, and outside temp ofcourse, but this ain't gonna cut it:

Fan controlled temperature: 61.5 deg-C

Fan speed switching temperature thresholds:
Speed 1: NM<----->65 deg-C
Speed 2: 56<-----> 79 deg-C (shutdown)

Sensor B Temperature Readings:
Current temperature : 57.5 deg-C
Sensor A Temperature Readings:
Current temperature : 61.5 deg-C
Warning level.......: 69.0 deg-C
Shutdown level......: 79.0 deg-C

So the 2 extra fans inside the case seem to have dropped the temps compared to only 3 Arctic rear fans by 2.5 1 degree for sensor B, and 5 2 degrees for sensor A. It's a reasonable result in my opinion. Not much help. At low speed these don't move enough air.

The fact is that without direct airflow over the hot ASIC heatspreaders, the only way to compensate that is with very high general airflow throughout the whole chassis, which is impossible to do quietly.

I'm convinced now that it's not possible to make the 6450 close to silent without having 40 x 40 x 10 fans attached on the ASIC's.

Next comes splurging the big bucks for the Noctua's.

Edit3: Instead of the expensive Noctua's i decided to try these 3V - 12V chinese fans, and connect them to the fan headers instead. Presumably at 4.5V they should rotate at low RPM, plus i can connect them with a simple fan cable splitter without tapping into the 12V line. that's 10€ instead of 45€.

 

[tubs-ffm]

ICX 7150 stuck at boot after firmware update


Hello,

I needs some help. The firmware update of my ICX 7150-24P from 09.0.10a to 09.0.10c failed. The switch does not boot up any more. It gets stuck with prompt

OS>

I updated the primary partitopn. When I press 'b' during boot and select by comand boot_secondary the secondary partition, the same happens. It gets stuck with prompt

OS>

Any advice what to do to get it alive again?

 

[fohdeesha]

ICX 7150 stuck at boot after firmware update


Hello,

I needs some help. The firmware update of my ICX 7150-24P from 09.0.10a to 09.0.10c failed. The switch does not boot up any more. It gets stuck with prompt

OS>

I updated the primary partitopn. When I press 'b' during boot and select by comand boot_secondary the secondary partition, the same happens. It gets stuck with prompt

OS>

Any advice what to do to get it alive again?

don't run alpha firmware lmao

 

[fohdeesha]

I'm experiementing with VLAN routing on ICX6610 and I would need some help. Two computers in different VLANs don't seem to be able to reach each other. I did it pretty much based on this video.

• Only two computers connected to the switch, both have static IPs configured
  • 10.0.10.100, default route to 10.0.10.1, VLAN 10
  • 10.0.20.100, default route to 10.0.20.1, VLAN 20
• Two VLANs, 10 and 20 with router-interfaces set
• Two virtual interfaces, 10 and 20
  • 10.0.10.1/24
  • 10.0.20.1/24

But the computers cannot see each other. I tried to ping the other machine from both machines, but they cannot see reach other. Pinging the switch IP 10.0.10.1 and 10.0.20.1 from the computers works fine. Pinging from the switch to the computers doesn't work. I checked the ARP table from the switch and both computers are there with their IPs and correct MAC addresses. I have tried to disable all firewalls etc. What am I doing wrong or is there something I have misunderstood about this?

View attachment 23037

post the full output of "show run"

 

[tubs-ffm]

ICX 7150 stuck at boot after firmware update

Now I am able to print the boot log.
Looks like going back to recovery mode is the only way to go.

Starting kernel ...

sd 0:0:0:0: [sda] No Caching mode page found
Mounting file system...
booting with 4.4 kernelsd 0:0:0:0: [sda] Assuming drive cache: write through

NAND Type: Micron NAND 2GiB (x 1)
PLATFORM MN
dd cmd: UBI
Config partition mounted
Resource partition is mounted!!
1 external USB Drive(s)/Partition(s) Mounted..Successfully read the SKU Reg, the value is: 0xff
dma_mem_base: 0x96000000, dma_mem_len: 0x8000000, warm_mem_base: 0x9f500000, warm_mem_len: 0xb00000
Please download UFI image to install packages
/etc/process_packages.sh: line 153: can't create /.pkg/error.log: nonexistent directory
Package installation is incomplete, related functionalities may not be available
Cannot boot FastIron application due to package installation failure
starting console application
server-execvp: python3: No such file or directory



OS>

 

[tubs-ffm]

Looks like going back to recovery mode is the only way to go.

I cannot tell what went wrong. Maybe there was an easy way to fix it. Going via recovery mode and reinstall from scratch for me was the easiest way because I had the step-by-step manual on hand. Backup of latest config I also had. All is fixed again.

don't run alpha firmware lmao

... and never change a running system.
Maybe I follow both advises for the future.

 

[Chow]

I have a quedtion to the Icx 7250 Switch:

How can i set an uplink port on which are all vlans goes tagged to the other Switch (trunk port) inclusive vlan 1?

I dont know how i can setup this :-( vlan 1 at least is almost untagged :-(

Can anyone help me? What i am doing wrong?

 

[kpfleming]

I have a quedtion to the Icx 7250 Switch:

How can i set an uplink port on which are all vlans goes tagged to the other Switch (trunk port) inclusive vlan 1?

I dont know how i can setup this :-( vlan 1 at least is almost untagged :-(

Can anyone help me? What i am doing wrong?

Which software version are you using?

It should be possible to remove untagged VLAN 1 from that port and add tagged VLAN 1 (along with all the other VLANs).

 

[Chow]

This is my Software Version:

ICX7250-48 Router>show version

Copyright (c) Ruckus Networks, Inc. All rights reserved.
UNIT 1: compiled on Dec 16 2021 at 03:45:30 labeled as SPR08095f
(33554432 bytes) from Primary SPR08095f.bin (UFI)
SW: Version 08.0.95fT213
Compressed Primary Boot Code size = 786944, Version:10.1.21T215 (spz10121)
Compiled on Wed Aug 25 09:27:31 2021

 

[fohdeesha]

I have a quedtion to the Icx 7250 Switch:

How can i set an uplink port on which are all vlans goes tagged to the other Switch (trunk port) inclusive vlan 1?

I dont know how i can setup this :-( vlan 1 at least is almost untagged :-(

Can anyone help me? What i am doing wrong?

why do you want to carry tagged vlan 1? vlan 1 shouldn't be used at all really for security and many other reasons, set your main vlan to something else like 10, and carry it tagged across your trunk

 

[Chow]

My Management VLAN 1 is used to manage Server, switch and so on. And on this uplink port is sitting my opnsense router, which have to receive VLAN 1 tagged too, so i can reach everything on my management VLAN.

On my Cisco Switch there is no problem to do so. There it works as expected.

Still the question, how to do this on Brocade / Ruckus :-D

 

[kpfleming]

Why not have VLAN 1 untagged on the router (and other network devices)? That's a much more common configuration.

In any case, if the ICX won't let you remove untagged VLAN 1 from the uplink port, then just create a 'dummy' VLAN on the ICX that isn't ever used for anything, make it the untagged VLAN on that port, and make VLAN 1 tagged on that port.

 

[fohdeesha]

My Management VLAN 1 is used to manage Server, switch and so on. And on this uplink port is sitting my opnsense router, which have to receive VLAN 1 tagged too, so i can reach everything on my management VLAN.

On my Cisco Switch there is no problem to do so. There it works as expected.

Still the question, how to do this on Brocade / Ruckus :-D

you can't tag the default/native vlan, for many reasons (half of them security related). fix your opnsense router (requiring vlan 1 tagged is broke) to operate on a different vlan, like vlan 10, then you can set this network up properly. for starters, on your cisco, if you have stuff on tagged vlan1, and it's being trunked while it's still the native vlan, your network is vulnerable to vlan hopping (google cisco vlan hopping with native vlan 1)

 

[AndroidCat]

I managed to configure Oxidized to fetch ICX6610 config.
Oxidized SSH library has to be configured to accept older ciphers/kex protocols to be able to connect to the switch.
Ciphers have to be specified in the global section of Oxidized config file, they don't work in groups or models sections:

vars:
#add encryption and key exchange protocol
   ssh_encryption: "+3des-cbc"
   ssh_kex: "+diffie-hellman-group-exchange-sha1"

Then I have credentials in groups section (already defined groups in LibreNMS -> Oxidized plugin) and enable in models:

groups:
  default:
    username: localaccount
    password: supersecretpassword
  icx6610:
    username: xxx
    password: yyy
    
... other groups...

models:
  ironware:
   vars:
     enable: zzz
     remove_secret: false

 

[danb35]

Thanks, @AndroidCat, I'll have to give that a try--I'd been having trouble myself. Are the credentials required even when you're using public-key authentication? And I'd expected the model would be fastiron, not ironware.

Edit: I see that the problem on my 6450 (firmware version 8.0.30u) is that it's using an output pager, even though oxidized tries to disable it with skip-page-display and terminal length 0. Since show version produces output more than one screen's worth of output, and Oxidized times out waiting for a prompt it's never going to see. Not sure what to do about that at this point, though.

Edit 2: Ah, I wasn't clear on the use of enable in the config fragment above, and had assumed it called for the enable password. Since I don't have one set, I didn't include this line. Adding that line with a dummy value made it work. Excellent!

 

[thebwack]

I got all four breakouts into the InstaPatch working using @fohdeesha wiring pic and diagram. Awesome! Now I'm playing with static LAG'ing them into 4 ports on an older unmanaged version of the Qnap-Qsw-1208-8c and I keep getting the non primary port states switched to Blocking. trying to decide if I should just get a 7250 or something like this:

https://www.amazon.com/MikroTik-12-...t=&hvlocphy=9027500&hvtargid=pla-814106935359

I need to get at least a few 10GB ports in addition to the trunk, and I need the switch to be near silent. I need the trunk bandwidth as this is a room fulll of video editors and creatives. When we come back from a shoot and dump TB's of footage it slows our editors down. all sharing one 10GB uplink for now.

Any thoughts on a quiet managed switch with 8+ 10GB ports that could handle this? RJ45 preferred. Or I'll eventually get the Qnap working here, we'll see.

 

[PANiCnz]

Just picked up a 6450 but it didn't come with the mounting brackets, can anyone recommend some generic brackets off Amazon, eBay etc that will fit?

 

[LodeRunner]

I got all four breakouts into the InstaPatch working using @fohdeesha wiring pic and diagram. Awesome! Now I'm playing with static LAG'ing them into 4 ports on an older unmanaged version of the Qnap-Qsw-1208-8c and I keep getting the non primary port states switched to Blocking. trying to decide if I should just get a 7250 or something like this:

https://www.amazon.com/MikroTik-12-...t=&hvlocphy=9027500&hvtargid=pla-814106935359

I need to get at least a few 10GB ports in addition to the trunk, and I need the switch to be near silent. I need the trunk bandwidth as this is a room fulll of video editors and creatives. When we come back from a shoot and dump TB's of footage it slows our editors down. all sharing one 10GB uplink for now.

Any thoughts on a quiet managed switch with 8+ 10GB ports that could handle this? RJ45 preferred. Or I'll eventually get the Qnap working here, we'll see.

I doubt you’ll get an unmanaged switch working. Static LAG still requires the other switch to be aware of what’s happening. Unless that Qnap is a smart or partially managed switch.

As far as footage offload, would this be multiple streams or single stream? The single stream speed over a LAG is the speed of a single interface.

Whats the disk system you’re ingesting to?