Drag to reposition cover

Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

rootwyrm

Member
Mar 25, 2017
74
93
18
www.rootwyrm.com
Also keep in mind, the FCX and 6610/6650 platforms are NOT the same CPU/architecture as the 6430/6450 and 7xxx series lines. The 6430, 6450, and at minimum the 7450 and lower 7k switches are all ARM-based CPUs, not whatever the others use (PowerPC? I can't remember) so they suck a LOT less power just by default.

I agree that you're probably SOL on lower-power with BGP through the ICX line, but just on the off-chance you haven't, have you actually checked out the noise on a 7450? I've had them running on my desk for pre-deployment configuring and they're not really that loud, even the PoE ones. The FCXs and the 6610s are screamers (ESPECIALLY the PoE ones), but I haven't found the 7450s that terrible.

Of course, I haven't had one permanently parked 10 feet from my chair either, so yeah.
Oh, the 7450's look pretty damn bad... they're rated 47dBA. I'm fan noise sensitive so 47dBA is an extremely high number to me. (Really, anything over 38dBA @ 1m is irritant level.) For comparison, the 6610's rated 39-48dBA @ 1m. So the 7450 actually is as loud as the 6610. But the 7450 doesn't state a low-side number, so I honestly don't know how quiet it is at low load. Any time it's going to be under high load, I'm already going to have hearing protection on anyways.

Really there's no reason to not have BGP other than market segmentation on Brocade/Ruckus' part. Contrary to popular belief, unless you're doing some really big tables, BGP is noisy but not high CPU demand. Usually it's RAM. For a smaller environment (couple dozen /22-/32's,) even something like the Juniper EX2200-12CT can handle it with ease. You sure as hell don't want it trying a significant table, but you could very easily do a pretty robust anycast setup on even an ICX6430 without putting a dent in performance. And with the 7000-series on a common ARM architecture, it kinda surprises me nobody just copy-pasted the bgpd over to a lower switch.
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,727
3,075
113
33
fohdeesha.com
Oh, the 7450's look pretty damn bad... they're rated 47dBA. I'm fan noise sensitive so 47dBA is an extremely high number to me. (Really, anything over 38dBA @ 1m is irritant level.) For comparison, the 6610's rated 39-48dBA @ 1m. So the 7450 actually is as loud as the 6610. But the 7450 doesn't state a low-side number, so I honestly don't know how quiet it is at low load. Any time it's going to be under high load, I'm already going to have hearing protection on anyways.

Really there's no reason to not have BGP other than market segmentation on Brocade/Ruckus' part. Contrary to popular belief, unless you're doing some really big tables, BGP is noisy but not high CPU demand. Usually it's RAM. For a smaller environment (couple dozen /22-/32's,) even something like the Juniper EX2200-12CT can handle it with ease. You sure as hell don't want it trying a significant table, but you could very easily do a pretty robust anycast setup on even an ICX6430 without putting a dent in performance. And with the 7000-series on a common ARM architecture, it kinda surprises me nobody just copy-pasted the bgpd over to a lower switch.
Like anything else it's just market segmentation. Pretty much the only people demanding something like BGP on the bottom end models designed to be stuck in a campus wiring closet are the people in this thread, and of course they are not the main revenue generators for brocade/ruckus/etc. I'd wager even on the 7450, less than 10% of customers are using bgp on the thing.

Theres also no bgpd daemon to copy, there's no daemons to speak of at all - all Linux does is launch a single monolithic fastiron binary (a huge one), which contains proprietary implementations of pretty much everything minus the standard broadcom sdk stuff to actually talk to the ASIC (and last I looked, the bgp & ospf stack in the binary are licensed from IP infusion). Also worth noting the 7000 series is actually not on a common ARM architecture, there's some outliers like the 7750 which are still PPC (but at least moved to uboot + linux). I believe the new 7850 they also finally moved to native 64bit on the new armv8 A57s versus the 32 bit armv7 stuff on the older 7000 series as well but I haven't had a chance to look at the firmware yet
 

BullCreek

New Member
Jan 5, 2016
18
6
3
55
Finally took the plunge and got a 6610 - it didn't come with a console cable unfortunately - and I see multiple links to the required brocade cable in these pages, but looking at the pictures when i follow those links, I don't get it. The recommended one on ebay shows a picture of a cable with an rj45 on one end and mini usb on the other, and then also what looks like an adapter that goes from rj45 to db9. I assume i plug the rj45 end of the cable into the switch, but how to i go from the miniusb to db9 that I can plug into my USB to serial adapter or am I totally not understanding how it is supposed to work?

FWIW, I have a cisco baby blue cable i use at work, and I tried plugging it into the rj45 on the switch via my usb to serial adapter, and I get output, but just squiggles - any chance I could make that work - maybe wrong settings?
 

rocketpanda40

Member
Dec 12, 2019
49
31
18
  • Like
Reactions: fohdeesha

rocketpanda40

Member
Dec 12, 2019
49
31
18
Finally took the plunge and got a 6610 - it didn't come with a console cable unfortunately - and I see multiple links to the required brocade cable in these pages, but looking at the pictures when i follow those links, I don't get it. The recommended one on ebay shows a picture of a cable with an rj45 on one end and mini usb on the other, and then also what looks like an adapter that goes from rj45 to db9. I assume i plug the rj45 end of the cable into the switch, but how to i go from the miniusb to db9 that I can plug into my USB to serial adapter or am I totally not understanding how it is supposed to work?

FWIW, I have a cisco baby blue cable i use at work, and I tried plugging it into the rj45 on the switch via my usb to serial adapter, and I get output, but just squiggles - any chance I could make that work - maybe wrong settings?
Cisco style cable works, just make sure you're 96008n1 with flow control off.
 
  • Like
Reactions: T_Minus

Wolfstar

Active Member
Nov 28, 2015
159
83
28
48
Finally took the plunge and got a 6610 - it didn't come with a console cable unfortunately - and I see multiple links to the required brocade cable in these pages, but looking at the pictures when i follow those links, I don't get it. The recommended one on ebay shows a picture of a cable with an rj45 on one end and mini usb on the other, and then also what looks like an adapter that goes from rj45 to db9. I assume i plug the rj45 end of the cable into the switch, but how to i go from the miniusb to db9 that I can plug into my USB to serial adapter or am I totally not understanding how it is supposed to work?

FWIW, I have a cisco baby blue cable i use at work, and I tried plugging it into the rj45 on the switch via my usb to serial adapter, and I get output, but just squiggles - any chance I could make that work - maybe wrong settings?
The cable people are referring to in this thread is for the 7250/7450 switches. The 6610 and 6450/6430 switches all use a standard RJ45 serial port. That said, the way you would use the cable is to plug the mini-USB port into the switch, then the RJ45 into the adapter, then the adapter into your USB-Serial dongle.
 

BullCreek

New Member
Jan 5, 2016
18
6
3
55
Thanks Wolfstar, that makes sense about the cable not being for the 6610 which doesn't have a miniusb to plug into - and I'm happy to say my cisco cable worked after I got the settings right per rocketpanda40's advice - I'm off to the races now.
 

Wolfstar

Active Member
Nov 28, 2015
159
83
28
48
Like anything else it's just market segmentation. Pretty much the only people demanding something like BGP on the bottom end models designed to be stuck in a campus wiring closet are the people in this thread, and of course they are not the main revenue generators for brocade/ruckus/etc. I'd wager even on the 7450, less than 10% of customers are using bgp on the thing.
Can confirm. I'm a contractor for an... exceptionally large organization, and we're deploying 7450s en masse. (Last number I heard was in excess of 2500 switches.) We use them entirely as Layer 2 devices, so not even inter-VLAN routing.
 
  • Like
Reactions: fohdeesha

vangoose

Active Member
May 21, 2019
326
104
43
Canada
Powered on my second 7250. Very surprised it's a POE version.

Doing firmware upgrade to 8080e and POE firmware is automatically upgraded, don't need to run it inline upgrade

SSL Client Certificate is successfully created
PoE Info: PoE module 1 of Unit 1 on ports 1/1/1 to 1/1/24 detected. Initializing....
2000 log entries of PoE Event Trace Log Buffer is allocated on unit 1 for unit 1
PoE Event Trace Logging enabled for unit 1
PoE Info: Current Firmware version 1.8.8, Recommended Firmware version 2.1.1, Upgrade Required.
PoE Info: PoE FW upgrade is required. Auto upgrade will start now.
Firmware version from File: 2.1.1
PoE Warning: Upgrading firmware in slot 1....DO NOT SWITCHOVER OR POWER DOWN THE UNIT.
PoE Info: FW Download on slot 1...sending download command...
PoE Info: FW Download on slot 1...TPE response received.
PoE Info: FW Download on slot 1...sending erase command...
PoE Info: FW Download on slot 1...erase command...accepted.
PoE Info: FW Download on slot 1...erasing firmware memory...
PoE Info: FW Download on slot 1...erasing firmware memory...completed
PoE Info: FW Download on slot 1...sending program command...
PoE Info: FW Download on slot 1...sending program command...accepted.
PoE Info: FW Download on slot 1...programming firmware...takes around 5 minutes....
U1-MSG: PoE Info: Firmware Download on slot 1.....10 percent completed.
U1-MSG: PoE Info: Firmware Download on slot 1.....20 percent completed.
U1-MSG: PoE Info: Firmware Download on slot 1.....30 percent completed.
U1-MSG: PoE Info: Firmware Download on slot 1.....40 percent completed.
U1-MSG: PoE Info: Firmware Download on slot 1.....50 percent completed.
U1-MSG: PoE Info: Firmware Download on slot 1.....60 percent completed.
U1-MSG: PoE Info: Firmware Download on slot 1.....70 percent completed.
U1-MSG: PoE Info: Firmware Download on slot 1.....80 percent completed.
U1-MSG: PoE Info: Firmware Download on slot 1.....90 percent completed.
U1-MSG: PoE Info: Firmware Download on slot 1.....100 percent completed.
PoE Info: FW Download on slot 1...programming firmware...completed.
PoE Info: FW Download on slot 1...upgrading firmware...completed. Module will be reset.
PoE Info: Resetting in slot 1....
 

rootwyrm

Member
Mar 25, 2017
74
93
18
www.rootwyrm.com
Theres also no bgpd daemon to copy, there's no daemons to speak of at all - all Linux does is launch a single monolithic fastiron binary (a huge one), which contains proprietary implementations of pretty much everything minus the standard broadcom sdk stuff to actually talk to the ASIC (and last I looked, the bgp & ospf stack in the binary are licensed from IP infusion). Also worth noting the 7000 series is actually not on a common ARM architecture, there's some outliers like the 7750 which are still PPC (but at least moved to uboot + linux). I believe the new 7850 they also finally moved to native 64bit on the new armv8 A57s versus the 32 bit armv7 stuff on the older 7000 series as well but I haven't had a chance to look at the firmware yet
Oh, where there's a will (and some time) there's a way. At the end of the day, it's Linux. If you can't break the security on a Linux appliance, you just aren't trying. I wonder if you might mixing up licensees though; Broadcom was a known IP Infusion licensee (as was F10 now Dell) which lives on the DCX family. I would be a bit surprised if they bailed on Foundry's BGP and OSPF stacks, which both predated and were significantly better than IP Infusion. But, well, let's just say none of the post-Foundry owners are the sharpest crayons in the shed! ¯\_(ツ)_/¯ (And if Broadcom/Brocade want to argue, I swear to god I will find the list of bugs I found on Silkworm, DCX, and DMM.)

Based on the spec sheet, I'd guess the ICX7750 is an NXP 4040 (e500mc) and the chassis stack uses the sRIO ports to an XAUI interface on the stack port, which means the stack members can operate in hardware lockstep (critical with a non-RTOS.) The RAM amount (3840MB) points pretty strongly to being a 32-bit core; that's 4GB (32-bit maximum) with a 256MB address hole. But it could also be the P5-series (e5500) at below-stock clock. Also means it may have the secure code controller enabled which would make software modification even with the appropriate tools a non-starter.

Amusingly, we actually know what the ICX7850 is under the hood; it's a BCM58712D. (Clicky linky.) Unfortunately, it's a <sigh> Broadcrap <sigh> so chances of getting actual meaningful documentation on the CPU are between "screw" and "you." Worse, it looks to be an all around mediocre part that tried to cherry pick from Vulcan. The NXP LS1046A looks to have way better I/O on the newer and just plain better A72 core.
 

ronclark

New Member
Dec 6, 2019
11
6
3
so i picked up a ICX6610-48P it should be here Monday, so its over kill on the ports for what i need right now. its clean has ears, two power supplies and two fan trays.

I have a baby blue serial to RJ45, it sounds like i am set for getting into the console.
I need to get a few more things I need to get hooked up let see if i did my homework right.
I am running three 649281-B21 ConnectX3 QSFP+ I have two QSFP+ DAC cables so i should be good for two of servers. for the workstation it looks like i need

HP-Mellanox-655874-B21-QSFP-SFP-Adapter-Kit-655902-001

Amphenol-571540002-2-meters-SFP-10GbE-Direct-Attach-Passive-Copper-Cable
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,727
3,075
113
33
fohdeesha.com
Oh, where there's a will (and some time) there's a way. At the end of the day, it's Linux. If you can't break the security on a Linux appliance, you just aren't trying. I wonder if you might mixing up licensees though; Broadcom was a known IP Infusion licensee (as was F10 now Dell) which lives on the DCX family. I would be a bit surprised if they bailed on Foundry's BGP and OSPF stacks, which both predated and were significantly better than IP Infusion. But, well, let's just say none of the post-Foundry owners are the sharpest crayons in the shed! ¯\_(ツ)_/¯ (And if Broadcom/Brocade want to argue, I swear to god I will find the list of bugs I found on Silkworm, DCX, and DMM.)

Based on the spec sheet, I'd guess the ICX7750 is an NXP 4040 (e500mc) and the chassis stack uses the sRIO ports to an XAUI interface on the stack port, which means the stack members can operate in hardware lockstep (critical with a non-RTOS.) The RAM amount (3840MB) points pretty strongly to being a 32-bit core; that's 4GB (32-bit maximum) with a 256MB address hole. But it could also be the P5-series (e5500) at below-stock clock. Also means it may have the secure code controller enabled which would make software modification even with the appropriate tools a non-starter.

Amusingly, we actually know what the ICX7850 is under the hood; it's a BCM58712D. (Clicky linky.) Unfortunately, it's a <sigh> Broadcrap <sigh> so chances of getting actual meaningful documentation on the CPU are between "screw" and "you." Worse, it looks to be an all around mediocre part that tried to cherry pick from Vulcan. The NXP LS1046A looks to have way better I/O on the newer and just plain better A72 core.

it's definitely IP Infusion: (lots of their sigs in the binary as well):



and yeah, I never went anywhere near the silkworm, DCX, FC etc crap, they seemed to have the total opposite engineering mindset as the fastiron guys that came from foundry (which weren't perfect, but at least weren't super authoritarian about every piece of the OS). the 7750 management plane is a freescale P2041 (four e500mc cores on one chip). Oddly a lot of the 7750 datasheets show 8GB of RAM but indeed it's limited to 4GB. Not sure if they were planning on going 64-bit and didn't last minute or what, but to this day a lot of the datasheets show 8GB, lol.

the whole ICX7xxx series is broadcom, I'd say it's at least a step up from the Marvell ASICs they were using in the 6 series and before, at least in terms of documentation. Been trying to work with up-n-atom on writing new values to the broadcom warpcore registers on the 7250 to alter the serdes datarate to enable 2.5gbps SFPs on the thing (the ASIC fully supports it), but they've made the broadcom shell super annoying to get at (he's made progress, though, have a whole list of ~800 hidden commands I've been meaning to publish, including the broadcom tor shell). I know the old marvell ASICs didn't have half the tools to allow such modifications, and the ones that did required massive license fees just to get documentation for
 
  • Like
Reactions: Wolfstar

cooldude

Member
Feb 20, 2018
50
13
8
40
I am deciding between juniper ex3300 and 7250...Need 48port poe and 4 1ogig ports.Looking to do routing on switch and vlan isolation.
Right now using unifi switch fortigate.Fortigate does all routing..
I cant decide on switch..Any help appreciated
 

vangoose

Active Member
May 21, 2019
326
104
43
Canada
I am deciding between juniper ex3300 and 7250...Need 48port poe and 4 1ogig ports.Looking to do routing on switch and vlan isolation.
Right now using unifi switch fortigate.Fortigate does all routing..
I cant decide on switch..Any help appreciated
Like both but I find ICX cli is simpler and easier to understand.
 
  • Like
Reactions: fohdeesha

artlessknave

New Member
Mar 16, 2017
13
2
3
38
do these brocades work with any 10gbe rj45? they seem tempting compared to CRS317-1G-16S+RM, but as far as i can tell, that supports basically every sfp, including rj45 copper 10gbe, while the brocades...do not.
I'm not sure I wanna spend 350$ for only 8x10gbe ports that can never do 10gbe rj45 when I could spend 500$ for 16 flexible ports (1/10/1.5gbe, fiber/DAC/rj45), particularly if they sound like they are trying to break the sound barrier while being 40 pounds. I only want a "dumb" switch, I don't care about l3, routing, vlans, or any of that. (this would be connected to my "core" lb6m/lb4m pair (stock FW) to bring 10gbe to a different room)
 

artlessknave

New Member
Mar 16, 2017
13
2
3
38
Where in the world did you hear that? Like 30% of the people in this thread are using RJ45 10gbE adapters. Even the LB6M should support them if you load my brocade firmware on it
the thread is 150 pages long. i have no idea how many people are using what, the first page, while full of a ton of info, doesn't seem to have that info that I can find.

i guess it would be more accurate to have made that "..do not?". and I'm getting this from the brocade site, which lists a bunch of sfp's, none of them 10gbe rj45, only one of them 1gbe rj45 thus my question. I couldn't find anything that said it did or did not support them
I realize that the lb6m could support them but I don't need them there atm so that would be a project for the future, since bringing my network offline for a fw upgrade (which is inherently a risk) isn't in the plan right now.
 

CED6688

New Member
Dec 4, 2019
15
10
3
I will 100% confirm that both the ICX6610 and ICX6450 (now have both) support the exact same RJ45 SFP+ modules that my Mikrotik (own CRS305, CRS309, and RB4011) devices accept, along with the same DAC and fiber modules. I literally pulled them from the MT and plugged into the ICX switches.

While the ICX6610 is a bit loud and need to find a more isolated spot before I plug it in again, my ICX6450 with unmodified fans is quiet enough that the wife didn't realize it was there when she went into the room. :)
 
Last edited: