Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

kroy

New Member
May 17, 2019
24
7
3
So I’ve finally navigated the Brocade desert and have found the promised land.

  • 6450-48p was the first switch that I ordered. It worked perfectly but I suffered from a bit of a layer 8 issue here. I didn’t do my homework and discovered this switch doesn’t support dynamic IPv6
  • 6650. I ordered this for a bit of a different project. This switch was DOA and immediately RMAd.
  • 7250-24p #1. Since I was looking for dynamic routing on v6, this seemed like the obvious choice. This one arrived with obviously fried ports (lights stuck on), and no working POE. It took me a while to figure out because I had to hack out the cable. RMAd.
  • 7250-24p #2. Despite being “fully tested”, this switch had a bad POE board. To make things worse, it tried to start a fire while I was trying to troubleshoot with the seller. If not for the fire attempt, I might have kept it because POE wasn’t a deal-killer for me. RMAd.
  • 7250-24p #3. The eBay seller had listed this wrong. It was actually a 7250-24G. No 10Gb, no routing, I didn’t even take this one fully out of the box before RMAing.
  • 6610-24pe. And this point I was sick of the 7250 problems, and tried this switch. Despite mentioning POE all over the listing, and “all ports POE fully working”, what was delivered was a 6610-24-PE, not a 6610-24P-PE. I’ll definitely cop a bit to being a bit of a layer 8 issue here. I believed the listing, but I also didn’t know the difference between P-PE and just -PE. Went right back into the box once someone here pointed this out. I was a bit thankful because with one PSU, this bastard was LOUD. RMAd.
  • 7250-24p #4. FINALLY. A fully working switch. I paid a bit more, but this is a super clean switch.
It’s definitely been a journey, but I really do love these switches. The 7250s are especially nice as they eliminate dumb things like the `dual-mode` ports.

Thanks to the people here that have helped me along the way, especially through the layer8 problems, and especially to @fohdeesha for all the work he’s done on these things.
 
  • Like
Reactions: fohdeesha

Wolfstar

Active Member
Nov 28, 2015
155
78
28
45
It’s definitely been a journey, but I really do love these switches. The 7250s are especially nice as they eliminate dumb things like the `dual-mode` ports.
Okay, I gotta ask. Which dumb thing with dual-mode ports? We use dual-mode on the FCXs and ICX6450s at work, haven't had a chance to really dig on the 7450s we're using yet. We use dual-mode as a way of assigning what Cisco calls a native VLAN, which makes tons of sense to my (admittedly network-oriented) brain.
 

kroy

New Member
May 17, 2019
24
7
3
Okay, I gotta ask. Which dumb thing with dual-mode ports? We use dual-mode on the FCXs and ICX6450s at work, haven't had a chance to really dig on the 7450s we're using yet. We use dual-mode as a way of assigning what Cisco calls a native VLAN, which makes tons of sense to my (admittedly network-oriented) brain.
Just with the way the rest of the VLANs work, it’s clunky and disconnected. I ran into some goofy circular issues trying to change the native VLAN on a port to like 9, then tag in some other ones.

It’s just weird to have all your VLANs: you tag your trunk ports, untag your access ports, but then you have to go to a different section to have the untagged VLAN for a trunk port.

In the later Brocade versions that run on the 7250, you just add the port as untagged on the VLAN. It just makes more sense from a configuration standpoint.
 

Wolfstar

Active Member
Nov 28, 2015
155
78
28
45
Just with the way the rest of the VLANs work, it’s clunky and disconnected. I ran into some goofy circular issues trying to change the native VLAN on a port to like 9, then tag in some other ones.

It’s just weird to have all your VLANs: you tag your trunk ports, untag your access ports, but then you have to go to a different section to have the untagged VLAN for a trunk port.

In the later Brocade versions that run on the 7250, you just add the port as untagged on the VLAN. It just makes more sense from a configuration standpoint.
Yeah, my major complaint about... well, just about EVERYONE that isn't Cisco last I looked (and maybe Juniper, haven't used their stuff), but Brocade/Ruckus in particular is the way they handle VLANs and tagged vs. untagged.

What actually is nice for anything running 08.0.30 code is that if you set up per-port authentication, you can actually define access port VLANs on the interface - you just use `auth-default-vlan XXX` and as long as the port's authenticated it will work on that VLAN.

I will admit, dual-mode command on the port after you just did `no tagged ethe 1/1/1` for 15 VLANs is really freaking irritating.
 

kroy

New Member
May 17, 2019
24
7
3
Yeah, my major complaint about... well, just about EVERYONE that isn't Cisco last I looked (and maybe Juniper, haven't used their stuff), but Brocade/Ruckus in particular is the way they handle VLANs and tagged vs. untagged.
I definitely hear you there. I’ve been tearing through switches in the last year and have seen much of the good/bad/ugly/etc.

Cisco pretty much simplifies it down on a per interface basis to:
  • Choose native vlan
  • Choose tagged VLANs to allow/deny
  • Choose trunk vs access
The fs.com switches are pretty Cisco-like. They have a nice convenience trick where when you allow a VLAN on an interface, it’s automatically added to the VLAN database. Which is nice for a quick 2-4094.

Arubas go a bit in reverse and make you create profile and then assign that profile to a port. It works well but it’s obnoxious when you have a port that’s a bit of an outlier as you end up creating a profile specifically for that port.

In comparison to those, Brocade definitely isn’t terrible. They go a bit in reverse like the Arubas, and make you assign ports to VLANs instead of VLANs to ports. That would probably be super obnoxious, but the way Brocade accepts port and vlan ranges is definitely helpful.
 

DRAGONKZ

Member
Apr 9, 2018
71
9
8
37
Long time reader, first time poster! (in this thread)

I’m looking at getting 2 x 6610-48Ps and stacking them.

I may have missed posts about this topic, but I just wanted to check that 2 units can still be stacked using the non-breakout 40Gb ports?

I’m ideally wanting to use the 8 x 10Gb front ports, 2 x 40Gb breakout ports as 8 x 10Gb, and still use the 2 x 40Gb non-breakout ports to stack together.

Failing that, can the breakout ports be used for stacking instead?

Thanks!
 

mb300sd

Active Member
Aug 1, 2016
197
74
28
30
Anyone ever have just a single port PoE die suddenly? I rearranged some power cables today, taking the switch down, and now the port I had my AP on no longer comes up. Plugged it into a different one and it works. show inline power says 30000 allocated for a few seconds befoing going back to 0, but the AP doesn't power up.
 

gustav9797

New Member
Dec 30, 2016
16
7
3
22
Checked the power draw of the ICX6450-24P for fun. Looks like it draws 50W at startup and then goes up to a stable 78W after booting, this is with nothing connected but the power cord. A bit far from 21W, is there anything wrong with it?
 

infoMatt

Active Member
Apr 16, 2019
142
53
28
The PoE unit draws a lot more power.
No, it shouldn't... during normal operating conditions, the difference should be negligeable (say 1-2W), other than the load required for the PoE devices themselves, of course.
The "partial load" consumption found on the datasheet is measured as 5% ethernet traffic, full load of PoE PD ;), that's why it's not much different than the 100% load figure.

Personally, I don't have (at the moment?) a power meter, but just by feeling, to me the 75W sounds way off the tracks... I mean, it was pretty hot here in Italy in the last few days, and the exhaust of my ICX 6450-24P felt "by hand" barely warmer than the surrounding environment, even after an entire afternoon in a not-so-well-ventilated room with an ambient temperature of 28-29°C (no aircon of course)... Inline power was enabled on all ports, but there were just two access points connected.
 
Last edited:

pyroguysf

New Member
Aug 17, 2019
3
0
1
I'm looking to upgrade my wireless router + unmanaged switch, and the 6450 seems perfect for future expansion. There's a lot of great resources here, but I had a few questions.
  1. What would I need to set up the 10gb connections to my FreeNAS box and Windows workstation for faster transfers? I was looking at the Intel X520-DA1, but would the Mellanox cards mentioned here work in FreeNAS? It looks like I'd need some transceivers in addtion to specific cables, but I'm having trouble grasping the compatibility. A QSFP breakout seems a little cluttered and unnecessary for my usage.
  2. Would I be able to add an unmanaged switch downstream, or would that cause issues/functionality loss?
 

infoMatt

Active Member
Apr 16, 2019
142
53
28
What would I need to set up the 10gb connections to my FreeNAS box and Windows workstation for faster transfers? I was looking at the Intel X520-DA1, but would the Mellanox cards mentioned here work in FreeNAS? It looks like I'd need some transceivers in addtion to specific cables, but I'm having trouble grasping the compatibility. A QSFP breakout seems a little cluttered and unnecessary for my usage.
Yep, both the Intel and the ConnectX would work, pick the one that you can find at the lowest price.
Other than the NICs you'll need the cabes, and you have basically two choices:
  • SFP+ DAC (direct attach cable), if you need to cover a short distance and they don't need to be routed through the walls/conduits, or
  • SFP+ optics and fiber optic: these switches don't care about the brand of the optics, but if it's not Brocade you wouldn't see the monitoring details (ie. optical power received and temperature, among the others...). If you choose this route you'll have to match fiber types and optics at both ends (single mode vs multimode, for in-house purpose basically consider the shortest range optics, ie. SR for MM and LR for single mode). Don't quote me on that, but from my memory the Intel NICs by default are picky about the optics, and require Intel ones, but I've not tried myself (this is what I've been told at $dayjob, but can't confirm or deny).
EDIT: QSFP breakouts are used at the switch side for the 40G ports, and there are none on the 6450; if you find a 40G QSFP NIC you'll need a QSFP-to-SFP+ converter (look at the last few pages).

Would I be able to add an unmanaged switch downstream, or would that cause issues/functionality loss?
No problems; watch for any loops in the network, because if you make a loop ad activate the spanning tree, the switch might ERR-DISABLE the downlink port ;).
 
Last edited:

pyroguysf

New Member
Aug 17, 2019
3
0
1
Thanks Matt.
  • SFP+ DAC (direct attach cable), if you need to cover a short distance and they don't need to be routed through the walls/conduits, or
  • SFP+ optics and fiber optic: these switches don't care about the brand of the optics
So you don't need optics with a DAC? I guess the direct attach part makes sense now. When I was looking at FS, their DACs had a different option for each manufacturer - does that matter if optics aren't involved, or is it just more of a marketing thing? Should I just get one that matches the NIC, or would I need a dual compatible one?

Similarly, I read about Intel NICs being optics locked, so if I went that route, I'd want an Intel transceiver at each end with an LC-LC OM1 cable? If I went Mellanox, would the Brocade transceiver work on that end for the extra details?

QSFP breakouts are used at the switch side for the 40G ports, and there are none on the 6450; if you find a 40G QSFP NIC you'll need a QSFP-to-SFP+ converter (look at the last few pages).
That makes much more sense, I don't know how I missed that. I swear I was reading it as people using breakouts on the NIC side which made no sense.

Edit: I've read conflicting reports - if I connect my NAS and workstation to the SFP+, will I be able to transfer at 10Gb?
 
Last edited:

French Chamallow

New Member
Mar 15, 2019
17
3
3
Checked the power draw of the ICX6450-24P for fun. Looks like it draws 50W at startup and then goes up to a stable 78W after booting, this is with nothing connected but the power cord. A bit far from 21W, is there anything wrong with it?
can other users with a powermeter confirm that the ICX6450-24P consumes so much watt in idle ?
 

klui

Active Member
Feb 3, 2019
123
45
28
Not according to the main post and the specifications here: Ruckus ICX 6450-24P
:rolleyes:
Check your meter's other measurements. I have a brand new ICX7150-C12P and my Kill-a-Watt measures 36W with only the console cable connected.

This Kill-a-Watt is used intermittently, but I have shut off the circuit while it was plugged in before. Its LCD is currently slow to update and you can see the LCD segments' refresh rate is slow even when values aren't changing. The voltage reading is on the outlet with nothing plugged in is 180V and that isn't right. It displays that voltage for every other single-phase circuit in my house. A CatII multimeter shows normal voltage at 120V. So the measurement for wattage will be off due to voltage, at least. If I convert using 120 instead of 180, I get roughly 24W or so. Shutting off PoE on all ports don't change the wattage value from the Kill-a-Watt.