Drag to reposition cover

Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

noise850

New Member
Feb 28, 2020
10
0
1
I just replaced the stock fan in a 6450-48 non PoE with a Sunon MB40201V3-000U-G99 and I am a bit concerned about heat. The ambient temperature of the room is around 75 F right now and with the fan running on high from the dd set_pwrfan_high command I am getting the following temps.

Code:
Power supply 1 (NA - AC - Regular) present, status ok
Power supply 2 not present

Fan ok, speed (auto): [[1]]<->2

Fan speed switching temperature thresholds:
        1 -> 2 @ 69 deg-C
        1 <- 2 @ 64 deg-C

Sensor B Temperature Readings:
        Current temperature : 48.0 deg-C
Sensor A Temperature Readings:
        Current temperature : 60.5 deg-C
        Warning level.......: 66.0 deg-C
        Shutdown level......: 76.0 deg-C
The noise from the fan is noticeable in high mode and there is also a low pitch whirring noise only present with the lid on (can't hear it with the lid off on the bench). I've only got one Gb trunk line going in and one Gb access port in use as a test, with very low utilization and I am a bit worried about cooking this thing under a full load or if I turn the fan back down with dd set_pwrfan_low.

What do you all think?

Edit:

set_pwrfan_low as a test and ran it for 20 minutes and these are the current readings. Am I misunderstanding something as it looks like Sensor A is about to hit Warning level?

Code:
Fan ok, speed (auto): [[1]]<->2

Fan speed switching temperature thresholds:
        1 -> 2 @ 69 deg-C
        1 <- 2 @ 64 deg-C

Sensor B Temperature Readings:
        Current temperature : 53.0 deg-C
Sensor A Temperature Readings:
        Current temperature : 65.0 deg-C
        Warning level.......: 66.0 deg-C
        Shutdown level......: 76.0 deg-C
Edit 2:

It's now the hottest part of the day for this room. Sensor B is at 55. Sensor A at 67. Nothing in the log file except my ssh connections to check temps and logging level is set at ACDMEINW. I am now really confused.
 
Last edited:

TheCodeLife

New Member
Mar 29, 2019
25
3
3
@fohdeesha I was able to unlock the hidden bootloader and I do see a tftpput command. However, I'm struggling to understand how to dump the nand into a file. I was hoping to do something like piping the output from the nand read.raw command into the tftpput utility, but I've been unsuccesful with that so far. I can create empty files using the tftpput command, so I know it's connecting to my tftp server.

Here are the nand menu options if it's helpful at all:

Code:
nand info - show available NAND devices
nand device [dev] - show or set current device
nand read - addr off|partition size
nand write - addr off|partition size
    read/write 'size' bytes starting at offset 'off'
    to/from memory address 'addr', skipping bad blocks.
nand read.raw - addr off|partition [count]
nand write.raw - addr off|partition [count]
    Use read.raw/write.raw to avoid ECC and access the flash as-is.
nand erase[.spread] [clean] off size - erase 'size' bytes from offset 'off'
    With '.spread', erase enough for given file size, otherwise,
    'size' includes skipped bad blocks.
nand erase.part [clean] partition - erase entire mtd partition'
nand erase.chip [clean] - erase entire chip'
nand bad - show bad blocks
nand dump[.oob] off - dump page
nand scrub [-y] off size | scrub.part partition | scrub.chip
    really clean NAND erasing bad blocks (UNSAFE)
nand markbad off [...] - mark bad block(s) at offset (UNSAFE)
nand biterr off - make a bit error at offset (UNSAFE)

EDIT: I have discovered that nand read will put the data directly in RAM. I can then use tftpput to dump a user specified portion of the RAM into a file on the tftp server. The biggest problem with this process that I'm encountering is that nand read seems to hang instead of reading past the bad sectors. I was hoping to use nand read.raw to get around this issue, but nand read.raw doesn't seem to place the read data into RAM, so I don't know how to get it. nand read.raw also reads a different amount of data than nand.read.

nand read.raw example:
Code:
ICX7150-Boot>nand read.raw 0 0 1

NAND read:  4320 bytes read: OK
nand read example:
Code:
ICX7150-Boot>nand read 0 0 1

NAND read: device 0 offset 0x0, size 0x1
1 bytes read: OK
If anyone has any ideas I would appreciate the input.

EDIT 2: I'm also having problems with tftpput only working when uploading small amounts of data (~1KB). Larger amounts of data transfer nothing and just timeout. It might just be my TFTP program on Windows, so I'll try setting up a TFTP server on Linux later and try again.
 
Last edited:

aidenpryde

New Member
Apr 30, 2020
27
1
3
So, I'm looking to quiet down the 1 fan that this ICX 6450-24 has in it. I did a little searching in this thread and I've seen the NF-A4x20 FLX and the Sunon MB40201VX-000U-G99 has been used by a couple folks, but it looks like these fans will cause a boot loop until the stock fan is put back in.

Can anyone tell me if they've managed to get a quieter fan inside this that doesn't have that issue?
 

infoMatt

Active Member
Apr 16, 2019
222
100
43
So, I'm looking to quiet down the 1 fan that this ICX 6450-24 has in it. I did a little searching in this thread and I've seen the NF-A4x20 FLX and the Sunon MB40201VX-000U-G99 has been used by a couple folks, but it looks like these fans will cause a boot loop until the stock fan is put back in.

Can anyone tell me if they've managed to get a quieter fan inside this that doesn't have that issue?
6450 will boot even without any fans plugged in, no problem.
 

OptimusPrime

Member
Apr 21, 2020
44
6
8
This is fun. I got all 4 of my RJ45 SFP transceivers installed into my 6450. All three PCs have their 10Gbe cards. Looks like I can transfer between a PC’s M2 card and another PC’s SSD at 3.5 Gbps max…which is close to maxing out the SATA interface on SSD.

However, mechanical drives are disappointing. Even though they are SATA III, my transfer speeds when reading from one of the mechanical drives averages 111 MB/s. It’s a little less when writing to them.
 

infoMatt

Active Member
Apr 16, 2019
222
100
43
This is fun. I got all 4 of my RJ45 SFP transceivers installed into my 6450. All three PCs have their 10Gbe cards. Looks like I can transfer between a PC’s M2 card and another PC’s SSD at 3.5 Gbps max…which is close to maxing out the SATA interface on SSD.

However, mechanical drives are disappointing. Even though they are SATA III, my transfer speeds when reading from one of the mechanical drives averages 111 MB/s. It’s a little less when writing to them.
Yes, those are the expected performance out of a spinning hard drive. It will vary with the size of the disk, number of plates and speed, but it should start around 120-180MB/s for the outer tracks, and end at say 80-120MB/s for the innermost tracks.

To get faster speeds, you'll need a RAID 0/5/10 configuration. ;)
 

infoMatt

Active Member
Apr 16, 2019
222
100
43
I figured that out later. Will the 6450 take whatever 3-pin fan I give it?
It should... They are standard +12V GND and PWM signal, but pay attention that the pinout is not the standard one used for computer fans; a user made a video, reposted in the last few pages, in which it shows the correct pins to swap.
 

aidenpryde

New Member
Apr 30, 2020
27
1
3
Thanks. Does anyone have a guide on how to get these working with pfSense. I don't think I'm capable of translating some of the guides you see around for Netgear and Cisco switches into Brocade. If I can't get this working I'm going to have to return this or resell it.
 

bubsterboo

Member
Dec 15, 2019
42
18
8
Thanks. Does anyone have a guide on how to get these working with pfSense. I don't think I'm capable of translating some of the guides you see around for Netgear and Cisco switches into Brocade. If I can't get this working I'm going to have to return this or resell it.
Unless you need to get into vlans or any kind of more advanced configuration then there's really not much special to it.
Follow the guide from the OP to get your switch in a good default state. Plug the LAN side of your pfsense into any port on the switch and that's really it!
Ofcourse there's a ton of features and goodies if you want to get more into it. But you don't need to!
 

OptimusPrime

Member
Apr 21, 2020
44
6
8
Is there a good primer thread on pfSense for home hobbyists? It seems to be a recurring topic here, I'd like to learn more about it.
 

aidenpryde

New Member
Apr 30, 2020
27
1
3
Unless you need to get into vlans or any kind of more advanced configuration then there's really not much special to it.
Follow the guide from the OP to get your switch in a good default state. Plug the LAN side of your pfsense into any port on the switch and that's really it!
Ofcourse there's a ton of features and goodies if you want to get more into it. But you don't need to!
That's the thing, I don't want anything complex, but no matter what I do here it doesn't seem as though I'm understanding.

All I want is 5 VLANS similar to the guide here: pfSense baseline guide with VPN, Guest and VLAN support

But this is to too much for me to do via CLI without some kind of guide. If I could get the web interface up when connected to pfSense, maybe I could do it, but I can't do it with my knowledge.
 

bubsterboo

Member
Dec 15, 2019
42
18
8
That's the thing, I don't want anything complex, but no matter what I do here it doesn't seem as though I'm understanding.

All I want is 5 VLANS similar to the guide here: pfSense baseline guide with VPN, Guest and VLAN support

But this is to too much for me to do via CLI without some kind of guide. If I could get the web interface up when connected to pfSense, maybe I could do it, but I can't do it with my knowledge.
I don't mean to be a debby downer or anything. But.. That guide isn't exactly for a simple network setup. It's not the most complicated. It's not impossible. But it isn't simple either. It will require a lot of reading and some patience to get working properly. Likewise you will need a bit of patience to learn how to manage VLANs on the brocade switches. Its not complicated and you don't need to fear the CLI. It took me about half a day to learn about VLANs and read through the brocade fastiron documentation to get something similar going.
 

klui

Well-Known Member
Feb 3, 2019
824
453
63
Not a good experience with going to 08.0.92b on an ICX 7150-C12P from 08.0.70c.

I couldn't go directly from 70 to 92 because 70 is a non-UFI while 92 is. I had to go to 08.0.80 non-UFI first. I decided to have primary on 80f and secondary 92b. The upgrade to 80f was straightforward. copy from tftp to flash the 80f's bootrom, image to primary, then reload. The nice thing was after bootup the PoE FW is automatically updated. And the system prints progress to the console. The ICX6610 running 08.0.30u does not print progress but prevents you from rebooting until the PoE FW is done after a manual upgrade. show logging will display progress.

The problem was upgrading the secondary to 92b. Because it's a unified image, the bootrom is embedded in the ufi.bin file. When I boot back to 80f, I'm met with a boot-monitor version mismatch at startup. Back in 92b, it shows there are primary and secondary boot code partitions. They both are the version recommended for 92 but the copy to flash bootrom command has been deprecated and there are no commands to replace the primary boot code. Under 80f, there is only one flash bootrom command and there is no way to specify which partition to use. What's more 92b displays "Moving app to flash...." upon every boot and takes an annoyingly long (~15 sec) time. Going back from 92b to 80f loses some stacking/trunk port definitions if there is no pre-8090-startup-backup file. What was a convenience of PoE FW auto flashing is now a hassle. Switching between versions will cause during first reboot into the partition upgrade/downgrade of PoE FW which takes around 2 minutes.

The above served me right because reading the Software Upgrade guide shows Ruckus recommending upgrades of pre-08.0.80 by

flash 80 bootrom
flash non-UFI 80 image primary
reboot
flash 92 image primary
flash 92 image secondary
reboot

There is no support for different versions between partitions if their recommended bootroms are different.

Keeping 80f on primary for now.
 

aidenpryde

New Member
Apr 30, 2020
27
1
3
I don't mean to be a debby downer or anything. But.. That guide isn't exactly for a simple network setup. It's not the most complicated. It's not impossible. But it isn't simple either. It will require a lot of reading and some patience to get working properly. Likewise you will need a bit of patience to learn how to manage VLANs on the brocade switches. Its not complicated and you don't need to fear the CLI. It took me about half a day to learn about VLANs and read through the brocade fastiron documentation to get something similar going.
Yeah, okay, that's annoying. Figure I should have gone with Netgear or Cisco as this is essentially a dumb switch for me now.
 

gb00s

Well-Known Member
Jul 25, 2018
1,175
586
113
Poland
That's the thing, I don't want anything complex, but no matter what I do here it doesn't seem as though I'm understanding.

All I want is 5 VLANS similar to the guide here: pfSense baseline guide with VPN, Guest and VLAN support

But this is to too much for me to do via CLI without some kind of guide. If I could get the web interface up when connected to pfSense, maybe I could do it, but I can't do it with my knowledge.
Go to and check out Terry Henry's YT channel. he has some nice and easy to understand vid's about setting up & configuring a Brocade switch via CLI in FastIron OS. It's super cool and I'm glad I didn't give up. It's going to be fun to install a switch via CLI and you will be able to do it as well and learn something. Yes, you can buy a Cisco SG200-10 and just 'copy & paste' and learn nothing. But what if you want to change something in the setup later or you have to troubleshoot a problem and no blog in the WWW can guide you with your specific issue?

In my personal opinion, if you have the Layer3 setup on your future switches and you want to combine it with pfSense, it's not fun either CLI or not.

It's not hard to learn. I'm, just trying to push you and encourage you in a positive way. If you got your VLAN in FastIron working, you will be able to do the next in almost every Cisco switch via CLI as well. Both are so close to each other that you can use what you learned. Give yourself a push. Brocades are awesome.

Regards

Mike
 
Last edited:

infoMatt

Active Member
Apr 16, 2019
222
100
43
For basic L2/L3 tasks, pretty much every vendor has the same logics and "things" to setup via the CLI.
The biggest difference in Brocade-style vs Cisco-style CLI is the definition of VLAN and interface... With the former, you attach an interface to a vlan (conf t; vlan xxx; tagged ether 1/1/3), with the latter you define VLANs on a interface (conf t; interface ethe 1/1/3; switchport tagged vlan add xxx).
HP/Aruba AFIK are Brocade-like.

Aside for the cosmetic difference and "synonyms" for the different things (dammnit Brocade, why 'disable' and not 'shutdown' a-la Cisco??), the real value of those cheap boxes is to learn networking...

It's the same for say a programming language... in Python you'll iteate in a list in a different way than in Java or C++, but you know that to make a sum of all items you'll have to iterate over all of them. ;)
 
Last edited:

jd.developer

New Member
Jan 12, 2020
8
2
3
Concerning the 40G to 10G breakout setup on the ICX6610, I can’t seem to get the 40g to 10g breakout working. I have:

  1. a 40G-QSFP-SR-INT optic (also have the infiniband optic mentioned in an earlier post)
  2. fiber: female MPO to 8x LC OM4
  3. 10 G brocade optics and intel optics
I have connected the 40G optic to a far right qsfp port (should be a breakout port according to the earlier post by fohdeesha), connected the MPO cable, and then connected the LC end to a 10g optic, but no connection is established on the 10g side. I’ve confirmed the polarity is correct / matches the arrows on the 10g optic - light is coming out of the fiber. The switch also sees the 40G optic.

I’m using intel 10g nics, and usually the moment the fiber is plugged into a 10G connection, the nic port lights up indicating activity. Switching out intel and brocade optics on the 10g side seems to not make a difference. The host is a freenas box and the logs just show the interface as down when connected through the 10g breakout.

On the switch side, the breakout ports are enabled and the ICX6610 is not configured for stacking. It is running the the latest firmware, which should have breakout support. I wasn’t able to find any documentation on brocade’s site concerning fiber based breakouts.

Does anyone have an idea on how to get the breakouts working or any documentation I could read up further?
 
Last edited:
May 1, 2020
39
9
8
I just got my ICX6450 and am working through your guide. (finished actually but I'm trying to tweak things related to it)

I was wondering about the "enable" password. I created an account and I can login via ssh with a key which makes things really easy. However, when I want to elevate to the "enable" level, I still have to enter a password. Is it possible and safe to disable the enable password? or is there a way to authenticate it with they key itself?

I would think that the enable password is something that could be safely removed and handled purely based on user access levels. If I login as root, I can "enable" with no password (since I got in with the proper credentials to begin with), but if I'm a lower user I can't enable at all. Similar to sudo privileges on Linux.
 

infoMatt

Active Member
Apr 16, 2019
222
100
43
Is it possible and safe to disable the enable password? or is there a way to authenticate it with they key itself?
It's not possible to use a key authentication during the enable phase because you're in a strictly text-only phase... and it must work even on a serial console that doesn't know anything about SSH keys.
You can however disable it; not the wiser move in my opinion, but in a home/lab environment... why not?

I would think that the enable password is something that could be safely removed and handled purely based on user access levels. If I login as root, I can "enable" with no password (since I got in with the proper credentials to begin with), but if I'm a lower user I can't enable at all. Similar to sudo privileges on Linux.
You can define different security levels:

The privilege privilege-level parameter specifies the privilege level for the account. You can specify one of the following:
• 0 - Super User level (full read-write access)
• 4 - Port Configuration level
• 5 - Read Only level
The default privilege level is 0 .
(FastIron security guide)
Enable is more of a "su" than a "sudo" thing...