Drag to reposition cover

Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

fohdeesha

Kaini Industries
Nov 20, 2016
2,332
2,473
113
31
fohdeesha.com
it's been a while since I've messed with IGMP, it's possible I never even tried on the default LAN as my usual practice is discarding the default vlan 1, defining vlan's for everything and putting ports in them as needed (then you never need dual-mode tagging for one, since you will never have to pass the "1" tag, which has to be untagged)

you can certainly define a lag without shutting down ports, but before you define the lag, if you plug both links, it has no way of knowing they're going to be a lag in the future, so it's just going to see a loop and shut them down if stp is enabled (or allow a broadcast storm if it's not). What behavior were you expecting? if RSTP is on, and you plug in two links which make a loop, it's going to get disabled regardless of vendor :p
 
  • Like
Reactions: juey

Fryguy8

New Member
Jul 27, 2017
5
0
1
37
Is there any reason to consider 6430s, or is it worth it to just go to a 6450? I'm a bit worried about the power consumption and noise. And I don't really need the layer3 stuff, but hey it's a homelab I'll find something to do with it!

Everybody is saying that these are quiet, but the spec sheet is saying 38db, which is already maybe louder than I want. And in the case of the -48p model, 55db, which is clearly way too loud for me

I'd only be using 1-2 poe devices (maybe 3-4 in the future), so don't want to take a fan hit if I can avoid it.

Any real world numbers about fan noise, both with stock fans and sunons/noctuas?
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,332
2,473
113
31
fohdeesha.com
The 6430's have no 10gbe

The 6450 24's with one fan (that isn't bad or failing) are already near silent, with a noctua or similar, probably literally silent

Like everything else in their spec sheet, they're a bit conservative. For instance the real world power draw of the 6610 24 is a good 20 watts lower than stated. As for the power draw of the 6450 24, it's barely 20 watts, you're not going to get 4x 10gbe less than that anywhere
 

juey

Member
Oct 1, 2018
56
15
8
Germany
it's been a while since I've messed with IGMP, it's possible I never even tried on the default LAN as my usual practice is discarding the default vlan 1, defining vlan's for everything and putting ports in them as needed (then you never need dual-mode tagging for one, since you will never have to pass the "1" tag, which has to be untagged)

you can certainly define a lag without shutting down ports, but before you define the lag, if you plug both links, it has no way of knowing they're going to be a lag in the future, so it's just going to see a loop and shut them down if stp is enabled (or allow a broadcast storm if it's not). What behavior were you expecting? if RSTP is on, and you plug in two links which make a loop, it's going to get disabled regardless of vendor :p
Im usually doing the same, but never deleted vlan 1 (for whatever reason), can i switch the default vlan ? Didn´t find any information on that in the L2/L3 Guide.

Yeah but i was hoping the switch to be smart enough to take the lag ports as one lag virtual interface in terms of stp/rstp, so after you hit "deploy" it stops blocking all but one member port(s), but instead i have to manually shut the other ports, then deploy and then take them up again, kind of weird but im slowly getting used to it :D
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,332
2,473
113
31
fohdeesha.com
oh, so the issue is rstp blocks the port (as expected), but after deploying the LAG it does not come back up? According to the manual (and my experience), once a LAG is deployed, STP should unblock any ports previously blocked. It might not be doing that because you manually shut them beforehand? If so, don't shut anything, just let RSTP block one, configure and deploy the lag, and it should get unblocked on its own


re: default vlan: page 368 of the l2 guide. "default-vlan-id" granted some protocols don't like the default vlan so I typically just leave the default vlan 1, create a new vlan like "vlan 10 public" then put ports into it as needed
 

juey

Member
Oct 1, 2018
56
15
8
Germany
def vlan: Ouch, yeah found it, pretty straight forward, thanks !

lag: im usually setting up the cabling first, then shut the secondary ports, deploy the lag and then they come back as expected (e.g. rstp state forwarding), i tried your way on the SFP+ Ports and the ports just stayed in blocking. Gotta test that again without shutting them down...
 
  • Like
Reactions: fohdeesha

AiC

New Member
Oct 28, 2018
17
6
3
Got the console cable today. 6450 is up and running. I'm currently running a very basic config, not sure what all I need or want to do.
Big thanks for the guide way back on page one!!
 
  • Like
Reactions: fohdeesha

Dave Corder

Active Member
Dec 21, 2015
163
82
28
39
Just bought three 48 port Poe icx6610's for $153!

Brocade ICX 6610-48 48 Ports Gigabit Ethernet Switch ICX6610-48 | eBay

Two are probably junk, but one ... C'mon!

Hopefully there is a master reset jumper in there. Then a little (a LOT) help from @fohdeesha and 40gb will be a go.
Ah, so it was YOU who outbid me at the last second :)

Oh, well, probably good I didn't win the auction. My wife likely wouldn't be too happy with me when those showed up on my doorstep...
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,332
2,473
113
31
fohdeesha.com
the first 2 are faulty hardware, but the power supply and fan trays from them more than equal what you paid

switch 3 should work perfectly fine minus the faulty PoE. There's even a chance the PoE is perfectly fine but did not come up on port 29 and 30 because the passworded config from previous owner simply had poe turned off on those ports

the "factory set-default" command in my update guide removes any previous passwords
 
  • Like
Reactions: nedimzukic2

mattaw

Member
Jul 30, 2018
56
15
8
Ah, so it was YOU who outbid me at the last second :)

Oh, well, probably good I didn't win the auction. My wife likely wouldn't be too happy with me when those showed up on my doorstep...
I'm going to get it but good when they do arrive, sadly.
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,332
2,473
113
31
fohdeesha.com
FYI found out you can NOT have a front 10gbE port (SFP+) and rear port (QSFP+ breakout) in a LACP group together. You can have a LACP with any combination of rear ports you want, and any combo of front ports you want, but you can't cross the streams

Only found out as I was trying to find ways to make the switch barf, was trying a giant 16x 10gbE port lag. 8x from the front or 8x from the back works great though
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,332
2,473
113
31
fohdeesha.com


Introducing: ICX7150-C12P
This lil' microbeefer has 12 1gbE ports (all PoE capable), 2x 10gbE SFP+ ports, and is totally fanless. It draws around 10 to 15 watts. Being an ICX7xxx series model, it will run the latest bleeding edge codetrain (currently v8080). v8080 is about 98% identical to v8030, with the 2% being minor improvements to vlan syntax and a couple other things.

The big change in v8080 however is licenses are no longer required. They still exist, but they are honor based and can be activated with a simple "license install perpetual xxx". This is why I'm finally mentioning these (they were 1gbE-only paperweights before).

Price: since these are much newer there are less listings and they vary wildly, but average seems to be from $200 to $300

Here's a listing at $185 with 3x available, let's see how fast STH can make them disappear

Video overview of features and the overall switch from my favorite Brocade/Ruckus engineer Terry Henry:
 
Last edited:

fohdeesha

Kaini Industries
Nov 20, 2016
2,332
2,473
113
31
fohdeesha.com


Introducing: ICX7250

This medium-rare beefer has 24 or 48 1gbE ports (PoE models available) with 8x 10gbE SFP+ ports. You can think of it as a big ICX6450, or a little ICX6610. It has fans, has the same sound level as the ICX6450, and draws about 50 watts. If you wished the ICX6610 cut it's power draw and sound level in half, and were willing to trade the rear QSFP+ ports and redundant PSU's for that, this is your switch.

Again, this 7 series runs v8080, so no licenses physically required.

  • L3 - all the same L3 features as the ICX6610 (ipv4/ipv6 routing, OSPF, VRRP, PIM, VRF's) except BGP
  • Same ACL features as 6450/6610
  • non-PoE models easily converted to 12v DC power (like the ICX6450)
  • 256gbps switching capacity
  • 190mpps forwarding capacity
  • ASIC: BCM56344
  • Management CPU: 1GHZ dual-core ARM
  • Datasheet

Price: the price on these vary more wildly. If you knew what I picked some up for, you'd stab me. They seemed to have levelled out at around $300 to $400. Here's a 24 port PoE model with no ears for $288

I picked a few of these up personally, so I was able to do my usual deep-dive on them. Below are my resulting notes:

Code:
##ICX7250 NOTES

Going price: $350 with wild swings both directions
24/48 1gbps copper ports (PoE models available)
8x 10gbE SFP+

All the L3 features of ICX6610 except BGP
A lower power quieter alternative to the ICX6610 without the QSFP ports or redundant PSU's
Runs latest FI codetrain - v8080
v8080 - honor based licensing (no actual licenses needed)


ASIC: Broadcom Helix 4 / BCM56344
CPU: ARM Cortex - A9 dualcore 1GHZ
(ARM management CPU is integrated into the ASIC - one P/N from BRDCM for whole package)

RAM: 2GB
Flash: 2GB

(Main flash is 2GB of NAND - stores everything but bootloader)
(8MB of separate SPI flash present only for bootloader/u-boot)

ARM JTAG header pads unpopulated (but present)

CPLD: Altera MAX V
CPLD JTAG header populated

Still plain (but branded) u-boot booting Linux
Linux then launches monolithic Fastiron binary application
(Same method & init scripts as ICX64xx)

47 watts power draw with 3 ports active
Number of ports active doesn't have an effect on power draw
To increase power draw to 55w, ~90gbps of L3 required

Same noise level as 6450
Lower pitched fan noise compared to ICX6450's higher pitched whine

Fans: PIA040H12P
Extra harness near fan connectors NOT for fan power (PSU sense harness - PoE PSU only?)

Hardware watchdog = shuts down switch with no fans
Unplugging all but 1 fan = OK


7250-2#dm fan-speed
Fan 1 Speed at 164 RPM (unplugged)
Fan 2 Speed at 6490 RPM

Fan header voltage at full speed - 12V
Fan header voltage at idle speed = 4.5V


ICX 7250-24 : One fan per device
ICX 7250-24P: Two fans per device
ICX 7250-48: Two fans per device
ICX 7250-48P: Three fans per device

Inline power (PoE) is enabled by default on ports as of FastIron release 08.0.70

#fully license a switch
enable
license install perpetual 1 8x10g
license install perpetual 1 l3-prem
conf t
interface ethernet 1/2/1 to 1/2/8
no speed-duplex 1000-full
write mem


Boot Console Output
Linux dmesg Output
Possible u-boot & Linux Commands
hi-res internal images


Info on stock fans that was hard to find (keep in mind these specs are at full bore 12v, while the switch usually feeds it 4.5v):



 
Last edited:

juey

Member
Oct 1, 2018
56
15
8
Germany
FYI found out you can NOT have a front 10gbE port (SFP+) and rear port (QSFP+ breakout) in a LACP group together. You can have a LACP with any combination of rear ports you want, and any combo of front ports you want, but you can't cross the streams

Only found out as I was trying to find ways to make the switch barf, was trying a giant 16x 10gbE port lag. 8x from the front or 8x from the back works great though
Yeah, this is remarked somewhere in the L2 Switching Guide, you cannot mix Media Types in LAGs, not sure if you can mix the front SFP+ with copper and glas transceiver.
 

u238

Member
Aug 11, 2018
40
10
8
Dang. That 7250 would be absolutely perfect for my setup. I'm going to keep an eye out for a cheap one.
 
  • Like
Reactions: fohdeesha

fohdeesha

Kaini Industries
Nov 20, 2016
2,332
2,473
113
31
fohdeesha.com
Yeah, this is remarked somewhere in the L2 Switching Guide, you cannot mix Media Types in LAGs, not sure if you can mix the front SFP+ with copper and glas transceiver.
It doesn't care about media types, you just can't mix speeds - from it's point of view the rear qsfp breakout with an SR4 optic and the front SFP+ were both optics anyway. The error message thrown out was "Error - Trunk port 1/3/3 and 2/2/2 do not have same default port speeds" so it seems it's a bit of a bug, nothing on purpose.

It's not checking the current configured speed of the port like it should be, but instead the default speed. And for the front 10gbE ports, that's 1gbE (with no license). Which obviously doesn't match the default 10gbE speed of the rear breakout port - even though they're both currently running at 10gbE. Something i'd imagine they could fix in software if they had any motivation, but considering we're not supposed to be using the rear ports at all I doubt that'll happen :p
 

juey

Member
Oct 1, 2018
56
15
8
Germany
It doesn't care about media types, you just can't mix speeds - from it's point of view the rear qsfp breakout with an SR4 optic and the front SFP+ were both optics anyway. The error message thrown out was "Error - Trunk port 1/3/3 and 2/2/2 do not have same default port speeds" so it seems it's a bit of a bug, nothing on purpose.

It's not checking the current configured speed of the port like it should be, but instead the default speed. And for the front 10gbE ports, that's 1gbE (with no license). Which obviously doesn't match the default 10gbE speed of the rear breakout port - even though they're both currently running at 10gbE. Something i'd imagine they could fix in software if they had any motivation, but considering we're not supposed to be using the rear ports at all I doubt that'll happen :p
I think this is just another reason why it fails
"[...]A combination of copper and fiber ports, even if they are of the same speed, cannot be members of the same LAG.[...]"
LAG formation rules