Drag to reposition cover

Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

fohdeesha

Kaini Industries
Nov 20, 2016
2,727
3,075
113
33
fohdeesha.com
Hehe yup, exactly why I mentioned reducing the amount of switchmode power supplies - evil noise! Are you a HAM by chance? (KD9DAL here). Used to dabble in rf amplifier design but never did much with it, mostly just did what I could to reduce the noisefloor here when I used to operate on HF regularly (and boy are switchmode power supplies horrific when it comes to HF noise)

if you or anyone else is interested, I might do a writeup on my equipment power measuring bench sometime, I made it up using all ebay gear (industrial grade equipment is hilariously cheap on ebay). I have an app that automatically pulls power input data from the wattnode's via modbus over RS485, then grabs DC output power data from the agilents over their serial adapter and automatically integrates it. However I think we're getting a little off topic hehe
 

Attachments

Last edited:
  • Like
Reactions: robo_geek_99

whitey

Moderator
Jun 30, 2014
2,766
868
113
41
Just a lil' off topic but I am smiling ear to ear here abt the 'geek speak' :-D You gents have issues...we're gonna need an intervention of sorts hah

Badass though!

EDIT: Still looking for my sub-$100 6450-24P, rather elusive, not as if I need it, my two 6610-24P's are just lonely I guess, want a lil' bro hah
 
Last edited:
  • Like
Reactions: Yunia and fohdeesha

fohdeesha

Kaini Industries
Nov 20, 2016
2,727
3,075
113
33
fohdeesha.com
Forgot to mention, the ICX6610 supports OpenFlow 1.3 in hardware, including hybrid port mode, if you want to play with a few hundred gbps of SDN :p
 
Last edited:
  • Like
Reactions: metag

Marsh

Moderator
May 12, 2013
2,642
1,496
113
Still looking for my sub-$100 6450-24P
Few days ago, a seller listed a bunch of ICX 6450 for $89 free shipping with BO.
My mistake #1 was offer 3 x $80 , within hour, the ICX 6450 were flying off the self.
I was lucky to purchased the last ICX 6450 at full price $89 with free shipping.

Same seller had 7 x ICX 6610-24 for $128, I got the last ICX 6610 as well.
I got to use the 10% ebay coupon.

These Brocade are not the POE model.
 

whitey

Moderator
Jun 30, 2014
2,766
868
113
41
Yeah, I am choosy for sure, luv me some poe but killer score, got my 6610-24p's for $100 each wooohoooo...it was like Christmas as a kiddo!
 
  • Like
Reactions: Marsh

abq

Active Member
May 23, 2015
675
204
43
Wow, where did you get the 6610-24P for $100 ea? That was a complete steal, sharing a link would be great!

Yeah, I am choosy for sure, luv me some poe but killer score, got my 6610-24p's for $100 each wooohoooo...it was like Christmas as a kiddo!
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,727
3,075
113
33
fohdeesha.com
Same seller had 7 x ICX 6610-24 for $128, I got the last ICX 6610 as well.
I got to use the 10% ebay coupon.

These Brocade are not the POE model.
If it's the seller I'm thinking of (has 2 or 3 negative feedbacks from the last couple months, had big 6450 and 6610 listings), Then the 6610 you won actually IS PoE.

He mis-listed both the ICX6450's and the ICX6610's - he listed the ICX6450's as the ICX6450-24P PoE model even though they weren't (he corrected this after I contacted him though). Then his ICX6610's were listed as ICX6610-24/48 non PoE, but if you looked at the pictures, they were definitely the PoE model.

When you get your 6610 delivered, look on the front where it says the model number and see if it has a P on the end (like ICX6610-24P). If it does, congrats you got PoE!

I need to add a warning about this to the main post here, it's unbelievable how often ebay sellers mislist PoE/non-PoE versions. Just ask @whitey :p
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,727
3,075
113
33
fohdeesha.com
Hidden Commands

All Fastiron platforms have hidden CLI commands, with the biggest one being "dm". The "dm" command is available (but hidden) for diagnostic use. Type dm (at the enable or config t level) and hit tab a couple times to see available options, like the below for example:


Code:
FCX3(config)#dm
  HEX                           Number
  802-1w                        show 802-1w internal information
  aaa-acct-session
  acl_tcam                      command for tcam manager
  aging-loop                    Turn off/on system aging loop
  allled                        Toggle -On/Off All LED
  alt-diag                      Test only, off/on
  app_vlan_debug                App VLAN table (shadow)
  auq-resync                    Sync Auq for a device
  auq-status                    show auq status
  badaddr                       Test only, will reboot
  blink                         Show gig link changes due to PHY blink
  buffer                        DMA CPU buffer related
  cancel-flash-timeout          set flash timeout to default
  clear_boot_count              Clear crash dump
  cpld-reg-dump                 Dump the CPLD register
There's hundreds of other options. These are low level debug commands, so if you don't know what they do you can break your switch (although usually fixed with a reboot). One useful command for example (that I don't understand why it's hidden) is "dm fan-speed" which shows you the current RPM/speed setting of each fan.


Hidden OS/Boot Monitor Console

This only works over a serial connection. It will not work over telnet/ssh/etc. Connect to a FastIron device over serial, and once it's fully booted and running, press ctrl+y, let go, then press m. Then hit enter. It should drop you down to the OS console:

Code:
OS>
Press the question mark on your keyboard for a list of all available options:

Code:
OS>
clear            Clear internal table
console          Attach console to a process
copy             Copy file
daw              Set data address watchpoint
dbg              Toggle debug mode
dq               Display memory in 64-bit quad word
dd               Display memory in 32-bit double word
del              Delete flash file
dir              List flash files
type             List file
erasesector      Erase Sector 1-512
dmsg             Display logged messages
etc etc .....
On the PowerPC platforms (ICX6610, FCX) there are many options here, you might recognize them as the same options available in the bootloader, but while the switch is booted - which can be very handy.

on the ARM models (ICX6450, ICX7xxx series), there are only a couple commands available in OS mode, compared to the hundred or so in PowerPC models:

Code:
ICX6450-24 Router>Switch to OS console...
OS>
show             Show information
set              Set config parameters
fill             Fill memory
dump             Dump register and memory
rdspr            Read SPR Register
wrspr            Write to a SPR Register
exit             Exit
reset            Reset
fastreset        GOS Reset
pci              Pci Conf/Mem Read/Write
pcie-ctrl        Pcie Controller Reg Read/Write
ppdev            Packet Processor Instance Mem Read/Write
cpld             CPLD Reg Read/Write
log_clear        Clear Core files
gpio             GPIO Reg Read/Write
test             Test command
OS>
There are fewer commands because ARM models are running on linux and u-boot, there's no custom boot monitor shell to drop down to. If you want to "drop down" to a lower level shell on ARM platforms, use the Accessing Linux guide
 

tommybackeast

Active Member
Jun 10, 2018
286
105
43
A sincere thank you to fohdeesha for his time and effort in getting the word out on this impressive switch.

The Brocade ICX-6450 is an amazing deal for the price. It's small in depth, so very quiet while running, and draws only 30-35 watts of power (no PoE being used).

fohdeesha's documentation is well written and clear.

I've only had the switch a short time, and have not learned enough about VLANs to execute the capabilities of this enterprise grade switch; but look forward to learning more.

Very helpful to know the Brocade can work with MikroTik's RJ45 10GBe $60 transceiver
 

tommybackeast

Active Member
Jun 10, 2018
286
105
43
They told me they would begin the process of working with the dev team and cataloging and providing all related source code - that should let me do quite a few interesting things with these switches.
Once the source code is available from Arris ; any examples on some of the things that becomes possible to do ?
 

Marsh

Moderator
May 12, 2013
2,642
1,496
113
I also like to thank @fohdeesha for the fantastic documentation.

The write up and research are top rated, it got me off my lazy bottom and buy a ICX6450 as well as ICX6610 from Ebay.

Both routers came fully licensed, I followed the direction to upgrade the firmware, configure the vlan, replaced the 10G router for a Proxmox cluster.

The ICX6450 40mm fan bearing is bad, the fan is screaming. I'll replace it soon.
 

whitey

Moderator
Jun 30, 2014
2,766
868
113
41
Wow, where did you get the 6610-24P for $100 ea? That was a complete steal, sharing a link would be great!
Those came w/ $30 shipped each so still a steal but yeah I had been hunting them after feeling sad then one night a couple months back scouring the original thread @fohdeesha started I went and looked at ebay and was shocked that some were JUST listed and priced right...couldn't let the 'STH effect' happen and there were only three that I snapped up, good thing too as I wanted two but messed up a BO submission and accidentally ended up with all three. Worked out though cause one of them had a failed PoE board and I sent that one back w/ upgraded firmware/licenses so they got a deal really.
 

arglebargle

H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈
Jul 15, 2018
657
244
43
Just a lil' off topic but I am smiling ear to ear here abt the 'geek speak' :-D You gents have issues...we're gonna need an intervention of sorts hah

Badass though!

EDIT: Still looking for my sub-$100 6450-24P, rather elusive, not as if I need it, my two 6610-24P's are just lonely I guess, want a lil' bro hah
I picked one up last week for $120 shipped. I threw a low-ball offer at a $175 listing for a dinged/scraped 24p to see if the seller would bite, UPS should have it here tomorrow. I'm really looking forward to replacing my collection of 5 port switches and PoE injectors with a single unit.

@fohdeesha thanks for pointing these out, the price/feature set/power draw are absolutely perfect for home labbing.
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,727
3,075
113
33
fohdeesha.com
Once the source code is available from Arris ; any examples on some of the things that becomes possible to do ?

Nothing too crazy, since the main networking application that drives the switching/routing ASIC is closed source. It would however allow me to easily re-add a lot of missing u-boot commands they ripped out that would make it easier to multi-boot a different linux build, give me back arbitrary I2C read/write commands (for programming custom serial numbers etc), stuff like that
 
  • Like
Reactions: maes

tommybackeast

Active Member
Jun 10, 2018
286
105
43
Nothing too crazy, since the main networking application that drives the switching/routing ASIC is closed source. It would however allow me to easily re-add a lot of missing u-boot commands they ripped out that would make it easier to multi-boot a different linux build, give me back arbitrary I2C read/write commands (for programming custom serial numbers etc), stuff like that
Thanks for your note back.
 

mangodoc

Member
Apr 26, 2017
32
21
8
@fohdeesha Thank you for all the help with this Brocade ICX-6610-24F switch, fantastic documentation and your overall help to the STH community.

When I received the switch, after a month in customs (SouthEast Asia), I found that the shipper (qncomputer) had included 8 SFPs. I wanted the "F" model because I have many SFP+ and DACs from my other switches and it would make for an easy switch over.

I turned it on and was surprised how loud it was, but pleasantly surprised after about a minute as it went almost silent. It had 07xxx image installed, I updated to 08030r.bin, the flash went smoothly, but I couldn't get an IP address to stay. It must have been a gummed up flash as I flashed 08030s.bin next and everything worked as expected.

I'm using one of the 40G slots to hook to a Mellanox ConnectX-3 CX314A 40G NIC via a 2m MC2210130-002 Mellanox Compatible 40G QSFP+ Passive DAC Cable on my desktop. One of the 40G Breakout ports hooked to a Dell 8024F switch via a FS.com #64268 2m Customized Compatibility 40G QSFP+ to 4x10G SFP+ Passive Direct Attach Copper Breakout Cable (just to test the port and breakout cable). I plan on purchasing a 10G NAS to replace my Asustor AS5004T, which is dated.