Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

kapone

Well-Known Member
May 23, 2015
784
384
63
I'm new to this thread. But, don't they all support 10 Gbps over the the SPF+ ports of the ICX models?

Perhaps you meant the SPF+ ports only support 1 Gbps or 10 Gbps, nothing inbetween?
Correct. That in between is much newer, and these switches are already EOL, which is why you can buy a 0.5tbps switching bandwidth switch for ~$200 (vs $10K or more)
 

vangoose

Active Member
May 21, 2019
263
69
28
Canada
I thought about getting a 6650 for a while, but hated the noise of my 6610-48P. And the fact that there was nothing you can do about changing the modding the fans. Do you know if its possible to change/mod the fans on the 6650?
No, it's same as 6610. I opened the cover, looked at it and gave up.

I've changed my plan for now. I have 2 7250 now, may get another 7250 and use it as core. another 2 7250 will run in layer 2 since I want pfc. BGP will be handled by juniper.

I will get the AC unit for the room so I can close the door, then 6650 will go live. Really love it.
 

FlorianZ

New Member
Dec 10, 2019
6
5
3
Uh oh! My new 7250-48P is from g-electronic. They have a bad track record of selling busted junk or?
Damn, I purchased from them as well. Short of physically testing all ports for functionality and power delivery, is there anything I should watch out for to determine whether my unit is broken? Any diagnostics I should be running? Thank you!
 

eduncan911

Active Member
Jul 27, 2015
100
44
28
65
eduncan911.com
Fohdeesha actually listed several reasons why he dislikes them for homelab use, but it comes down to "the 6610 has more and better ports". Specifically, you have three module slots (one front, two rear) and you can use either 40G or 4x10G modules in each one. Whereas with the 6610, you've got 8x SFP+, 2x QSFP+, and 2x QSFP+ Breakout ports. There's just way more high-speed capacity. They also tend to be MUCH more expensive than the 7250, which loses any 40G option but gives you 8x SFP+ fixed ports.
Yep, the 6610 is more ideal for ports and a little cheaper for more features. But it's also more power hungry, louder, and doesn't run an OS I can build (my ultimate goal since I've switched to Coreboot on most of my key devices). Less watts = less heat, which is also another key goal of mine since my closet I keep them is sensitive to a lot of heat (I build my servers for ultra-low power - 96 TB, 16 core server that idles at 45W).
 
Last edited:

rootwyrm

Member
Mar 25, 2017
33
41
18
www.rootwyrm.com
Also keep in mind, the FCX and 6610/6650 platforms are NOT the same CPU/architecture as the 6430/6450 and 7xxx series lines. The 6430, 6450, and at minimum the 7450 and lower 7k switches are all ARM-based CPUs, not whatever the others use (PowerPC? I can't remember) so they suck a LOT less power just by default.

I agree that you're probably SOL on lower-power with BGP through the ICX line, but just on the off-chance you haven't, have you actually checked out the noise on a 7450? I've had them running on my desk for pre-deployment configuring and they're not really that loud, even the PoE ones. The FCXs and the 6610s are screamers (ESPECIALLY the PoE ones), but I haven't found the 7450s that terrible.

Of course, I haven't had one permanently parked 10 feet from my chair either, so yeah.
Oh, the 7450's look pretty damn bad... they're rated 47dBA. I'm fan noise sensitive so 47dBA is an extremely high number to me. (Really, anything over 38dBA @ 1m is irritant level.) For comparison, the 6610's rated 39-48dBA @ 1m. So the 7450 actually is as loud as the 6610. But the 7450 doesn't state a low-side number, so I honestly don't know how quiet it is at low load. Any time it's going to be under high load, I'm already going to have hearing protection on anyways.

Really there's no reason to not have BGP other than market segmentation on Brocade/Ruckus' part. Contrary to popular belief, unless you're doing some really big tables, BGP is noisy but not high CPU demand. Usually it's RAM. For a smaller environment (couple dozen /22-/32's,) even something like the Juniper EX2200-12CT can handle it with ease. You sure as hell don't want it trying a significant table, but you could very easily do a pretty robust anycast setup on even an ICX6430 without putting a dent in performance. And with the 7000-series on a common ARM architecture, it kinda surprises me nobody just copy-pasted the bgpd over to a lower switch.
 

fohdeesha

Kaini Industries
Nov 20, 2016
1,909
1,722
113
29
fohdeesha.com
Oh, the 7450's look pretty damn bad... they're rated 47dBA. I'm fan noise sensitive so 47dBA is an extremely high number to me. (Really, anything over 38dBA @ 1m is irritant level.) For comparison, the 6610's rated 39-48dBA @ 1m. So the 7450 actually is as loud as the 6610. But the 7450 doesn't state a low-side number, so I honestly don't know how quiet it is at low load. Any time it's going to be under high load, I'm already going to have hearing protection on anyways.

Really there's no reason to not have BGP other than market segmentation on Brocade/Ruckus' part. Contrary to popular belief, unless you're doing some really big tables, BGP is noisy but not high CPU demand. Usually it's RAM. For a smaller environment (couple dozen /22-/32's,) even something like the Juniper EX2200-12CT can handle it with ease. You sure as hell don't want it trying a significant table, but you could very easily do a pretty robust anycast setup on even an ICX6430 without putting a dent in performance. And with the 7000-series on a common ARM architecture, it kinda surprises me nobody just copy-pasted the bgpd over to a lower switch.
Like anything else it's just market segmentation. Pretty much the only people demanding something like BGP on the bottom end models designed to be stuck in a campus wiring closet are the people in this thread, and of course they are not the main revenue generators for brocade/ruckus/etc. I'd wager even on the 7450, less than 10% of customers are using bgp on the thing.

Theres also no bgpd daemon to copy, there's no daemons to speak of at all - all Linux does is launch a single monolithic fastiron binary (a huge one), which contains proprietary implementations of pretty much everything minus the standard broadcom sdk stuff to actually talk to the ASIC (and last I looked, the bgp & ospf stack in the binary are licensed from IP infusion). Also worth noting the 7000 series is actually not on a common ARM architecture, there's some outliers like the 7750 which are still PPC (but at least moved to uboot + linux). I believe the new 7850 they also finally moved to native 64bit on the new armv8 A57s versus the 32 bit armv7 stuff on the older 7000 series as well but I haven't had a chance to look at the firmware yet
 

BullCreek

New Member
Jan 5, 2016
14
5
3
51
Finally took the plunge and got a 6610 - it didn't come with a console cable unfortunately - and I see multiple links to the required brocade cable in these pages, but looking at the pictures when i follow those links, I don't get it. The recommended one on ebay shows a picture of a cable with an rj45 on one end and mini usb on the other, and then also what looks like an adapter that goes from rj45 to db9. I assume i plug the rj45 end of the cable into the switch, but how to i go from the miniusb to db9 that I can plug into my USB to serial adapter or am I totally not understanding how it is supposed to work?

FWIW, I have a cisco baby blue cable i use at work, and I tried plugging it into the rj45 on the switch via my usb to serial adapter, and I get output, but just squiggles - any chance I could make that work - maybe wrong settings?
 

rocketpanda40

New Member
Dec 12, 2019
5
2
3
  • Like
Reactions: fohdeesha

rocketpanda40

New Member
Dec 12, 2019
5
2
3
Finally took the plunge and got a 6610 - it didn't come with a console cable unfortunately - and I see multiple links to the required brocade cable in these pages, but looking at the pictures when i follow those links, I don't get it. The recommended one on ebay shows a picture of a cable with an rj45 on one end and mini usb on the other, and then also what looks like an adapter that goes from rj45 to db9. I assume i plug the rj45 end of the cable into the switch, but how to i go from the miniusb to db9 that I can plug into my USB to serial adapter or am I totally not understanding how it is supposed to work?

FWIW, I have a cisco baby blue cable i use at work, and I tried plugging it into the rj45 on the switch via my usb to serial adapter, and I get output, but just squiggles - any chance I could make that work - maybe wrong settings?
Cisco style cable works, just make sure you're 96008n1 with flow control off.
 
  • Like
Reactions: T_Minus

Wolfstar

Active Member
Nov 28, 2015
155
81
28
45
Finally took the plunge and got a 6610 - it didn't come with a console cable unfortunately - and I see multiple links to the required brocade cable in these pages, but looking at the pictures when i follow those links, I don't get it. The recommended one on ebay shows a picture of a cable with an rj45 on one end and mini usb on the other, and then also what looks like an adapter that goes from rj45 to db9. I assume i plug the rj45 end of the cable into the switch, but how to i go from the miniusb to db9 that I can plug into my USB to serial adapter or am I totally not understanding how it is supposed to work?

FWIW, I have a cisco baby blue cable i use at work, and I tried plugging it into the rj45 on the switch via my usb to serial adapter, and I get output, but just squiggles - any chance I could make that work - maybe wrong settings?
The cable people are referring to in this thread is for the 7250/7450 switches. The 6610 and 6450/6430 switches all use a standard RJ45 serial port. That said, the way you would use the cable is to plug the mini-USB port into the switch, then the RJ45 into the adapter, then the adapter into your USB-Serial dongle.
 

BullCreek

New Member
Jan 5, 2016
14
5
3
51
Thanks Wolfstar, that makes sense about the cable not being for the 6610 which doesn't have a miniusb to plug into - and I'm happy to say my cisco cable worked after I got the settings right per rocketpanda40's advice - I'm off to the races now.
 

Wolfstar

Active Member
Nov 28, 2015
155
81
28
45
Like anything else it's just market segmentation. Pretty much the only people demanding something like BGP on the bottom end models designed to be stuck in a campus wiring closet are the people in this thread, and of course they are not the main revenue generators for brocade/ruckus/etc. I'd wager even on the 7450, less than 10% of customers are using bgp on the thing.
Can confirm. I'm a contractor for an... exceptionally large organization, and we're deploying 7450s en masse. (Last number I heard was in excess of 2500 switches.) We use them entirely as Layer 2 devices, so not even inter-VLAN routing.
 
  • Like
Reactions: fohdeesha

vangoose

Active Member
May 21, 2019
263
69
28
Canada
Powered on my second 7250. Very surprised it's a POE version.

Doing firmware upgrade to 8080e and POE firmware is automatically upgraded, don't need to run it inline upgrade

SSL Client Certificate is successfully created
PoE Info: PoE module 1 of Unit 1 on ports 1/1/1 to 1/1/24 detected. Initializing....
2000 log entries of PoE Event Trace Log Buffer is allocated on unit 1 for unit 1
PoE Event Trace Logging enabled for unit 1
PoE Info: Current Firmware version 1.8.8, Recommended Firmware version 2.1.1, Upgrade Required.
PoE Info: PoE FW upgrade is required. Auto upgrade will start now.
Firmware version from File: 2.1.1
PoE Warning: Upgrading firmware in slot 1....DO NOT SWITCHOVER OR POWER DOWN THE UNIT.
PoE Info: FW Download on slot 1...sending download command...
PoE Info: FW Download on slot 1...TPE response received.
PoE Info: FW Download on slot 1...sending erase command...
PoE Info: FW Download on slot 1...erase command...accepted.
PoE Info: FW Download on slot 1...erasing firmware memory...
PoE Info: FW Download on slot 1...erasing firmware memory...completed
PoE Info: FW Download on slot 1...sending program command...
PoE Info: FW Download on slot 1...sending program command...accepted.
PoE Info: FW Download on slot 1...programming firmware...takes around 5 minutes....
U1-MSG: PoE Info: Firmware Download on slot 1.....10 percent completed.
U1-MSG: PoE Info: Firmware Download on slot 1.....20 percent completed.
U1-MSG: PoE Info: Firmware Download on slot 1.....30 percent completed.
U1-MSG: PoE Info: Firmware Download on slot 1.....40 percent completed.
U1-MSG: PoE Info: Firmware Download on slot 1.....50 percent completed.
U1-MSG: PoE Info: Firmware Download on slot 1.....60 percent completed.
U1-MSG: PoE Info: Firmware Download on slot 1.....70 percent completed.
U1-MSG: PoE Info: Firmware Download on slot 1.....80 percent completed.
U1-MSG: PoE Info: Firmware Download on slot 1.....90 percent completed.
U1-MSG: PoE Info: Firmware Download on slot 1.....100 percent completed.
PoE Info: FW Download on slot 1...programming firmware...completed.
PoE Info: FW Download on slot 1...upgrading firmware...completed. Module will be reset.
PoE Info: Resetting in slot 1....
 

rootwyrm

Member
Mar 25, 2017
33
41
18
www.rootwyrm.com
Theres also no bgpd daemon to copy, there's no daemons to speak of at all - all Linux does is launch a single monolithic fastiron binary (a huge one), which contains proprietary implementations of pretty much everything minus the standard broadcom sdk stuff to actually talk to the ASIC (and last I looked, the bgp & ospf stack in the binary are licensed from IP infusion). Also worth noting the 7000 series is actually not on a common ARM architecture, there's some outliers like the 7750 which are still PPC (but at least moved to uboot + linux). I believe the new 7850 they also finally moved to native 64bit on the new armv8 A57s versus the 32 bit armv7 stuff on the older 7000 series as well but I haven't had a chance to look at the firmware yet
Oh, where there's a will (and some time) there's a way. At the end of the day, it's Linux. If you can't break the security on a Linux appliance, you just aren't trying. I wonder if you might mixing up licensees though; Broadcom was a known IP Infusion licensee (as was F10 now Dell) which lives on the DCX family. I would be a bit surprised if they bailed on Foundry's BGP and OSPF stacks, which both predated and were significantly better than IP Infusion. But, well, let's just say none of the post-Foundry owners are the sharpest crayons in the shed! ¯\_(ツ)_/¯ (And if Broadcom/Brocade want to argue, I swear to god I will find the list of bugs I found on Silkworm, DCX, and DMM.)

Based on the spec sheet, I'd guess the ICX7750 is an NXP 4040 (e500mc) and the chassis stack uses the sRIO ports to an XAUI interface on the stack port, which means the stack members can operate in hardware lockstep (critical with a non-RTOS.) The RAM amount (3840MB) points pretty strongly to being a 32-bit core; that's 4GB (32-bit maximum) with a 256MB address hole. But it could also be the P5-series (e5500) at below-stock clock. Also means it may have the secure code controller enabled which would make software modification even with the appropriate tools a non-starter.

Amusingly, we actually know what the ICX7850 is under the hood; it's a BCM58712D. (Clicky linky.) Unfortunately, it's a <sigh> Broadcrap <sigh> so chances of getting actual meaningful documentation on the CPU are between "screw" and "you." Worse, it looks to be an all around mediocre part that tried to cherry pick from Vulcan. The NXP LS1046A looks to have way better I/O on the newer and just plain better A72 core.
 

ronclark

New Member
Dec 6, 2019
9
3
3
so i picked up a ICX6610-48P it should be here Monday, so its over kill on the ports for what i need right now. its clean has ears, two power supplies and two fan trays.

I have a baby blue serial to RJ45, it sounds like i am set for getting into the console.
I need to get a few more things I need to get hooked up let see if i did my homework right.
I am running three 649281-B21 ConnectX3 QSFP+ I have two QSFP+ DAC cables so i should be good for two of servers. for the workstation it looks like i need

HP-Mellanox-655874-B21-QSFP-SFP-Adapter-Kit-655902-001

Amphenol-571540002-2-meters-SFP-10GbE-Direct-Attach-Passive-Copper-Cable
 

fohdeesha

Kaini Industries
Nov 20, 2016
1,909
1,722
113
29
fohdeesha.com
Oh, where there's a will (and some time) there's a way. At the end of the day, it's Linux. If you can't break the security on a Linux appliance, you just aren't trying. I wonder if you might mixing up licensees though; Broadcom was a known IP Infusion licensee (as was F10 now Dell) which lives on the DCX family. I would be a bit surprised if they bailed on Foundry's BGP and OSPF stacks, which both predated and were significantly better than IP Infusion. But, well, let's just say none of the post-Foundry owners are the sharpest crayons in the shed! ¯\_(ツ)_/¯ (And if Broadcom/Brocade want to argue, I swear to god I will find the list of bugs I found on Silkworm, DCX, and DMM.)

Based on the spec sheet, I'd guess the ICX7750 is an NXP 4040 (e500mc) and the chassis stack uses the sRIO ports to an XAUI interface on the stack port, which means the stack members can operate in hardware lockstep (critical with a non-RTOS.) The RAM amount (3840MB) points pretty strongly to being a 32-bit core; that's 4GB (32-bit maximum) with a 256MB address hole. But it could also be the P5-series (e5500) at below-stock clock. Also means it may have the secure code controller enabled which would make software modification even with the appropriate tools a non-starter.

Amusingly, we actually know what the ICX7850 is under the hood; it's a BCM58712D. (Clicky linky.) Unfortunately, it's a <sigh> Broadcrap <sigh> so chances of getting actual meaningful documentation on the CPU are between "screw" and "you." Worse, it looks to be an all around mediocre part that tried to cherry pick from Vulcan. The NXP LS1046A looks to have way better I/O on the newer and just plain better A72 core.

it's definitely IP Infusion: (lots of their sigs in the binary as well):



and yeah, I never went anywhere near the silkworm, DCX, FC etc crap, they seemed to have the total opposite engineering mindset as the fastiron guys that came from foundry (which weren't perfect, but at least weren't super authoritarian about every piece of the OS). the 7750 management plane is a freescale P2041 (four e500mc cores on one chip). Oddly a lot of the 7750 datasheets show 8GB of RAM but indeed it's limited to 4GB. Not sure if they were planning on going 64-bit and didn't last minute or what, but to this day a lot of the datasheets show 8GB, lol.

the whole ICX7xxx series is broadcom, I'd say it's at least a step up from the Marvell ASICs they were using in the 6 series and before, at least in terms of documentation. Been trying to work with up-n-atom on writing new values to the broadcom warpcore registers on the 7250 to alter the serdes datarate to enable 2.5gbps SFPs on the thing (the ASIC fully supports it), but they've made the broadcom shell super annoying to get at (he's made progress, though, have a whole list of ~800 hidden commands I've been meaning to publish, including the broadcom tor shell). I know the old marvell ASICs didn't have half the tools to allow such modifications, and the ones that did required massive license fees just to get documentation for
 
  • Like
Reactions: Wolfstar

cooldude

Member
Feb 20, 2018
34
2
8
36
I am deciding between juniper ex3300 and 7250...Need 48port poe and 4 1ogig ports.Looking to do routing on switch and vlan isolation.
Right now using unifi switch fortigate.Fortigate does all routing..
I cant decide on switch..Any help appreciated
 

vangoose

Active Member
May 21, 2019
263
69
28
Canada
I am deciding between juniper ex3300 and 7250...Need 48port poe and 4 1ogig ports.Looking to do routing on switch and vlan isolation.
Right now using unifi switch fortigate.Fortigate does all routing..
I cant decide on switch..Any help appreciated
Like both but I find ICX cli is simpler and easier to understand.
 
  • Like
Reactions: fohdeesha

artlessknave

New Member
Mar 16, 2017
11
2
3
34
do these brocades work with any 10gbe rj45? they seem tempting compared to CRS317-1G-16S+RM, but as far as i can tell, that supports basically every sfp, including rj45 copper 10gbe, while the brocades...do not.
I'm not sure I wanna spend 350$ for only 8x10gbe ports that can never do 10gbe rj45 when I could spend 500$ for 16 flexible ports (1/10/1.5gbe, fiber/DAC/rj45), particularly if they sound like they are trying to break the sound barrier while being 40 pounds. I only want a "dumb" switch, I don't care about l3, routing, vlans, or any of that. (this would be connected to my "core" lb6m/lb4m pair (stock FW) to bring 10gbe to a different room)