Drag to reposition cover

Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

StarsAndBars

New Member
Jul 13, 2020
21
2
3
Thank you, fohdeesha!

stack unit 1
module 1 icx6610-48p-poe-port-management-module
module 2 icx6610-qsfp-10-port-160g-module
module 3 icx6610-8-port-10g-dual-mode-module
priority 128
stack-port 1/2/1 1/2/6
stack unit 2
module 1 icx6610-48p-poe-port-management-module
module 2 icx6610-qsfp-10-port-160g-module
module 3 icx6610-8-port-10g-dual-mode-module
stack-trunk 2/2/6 to 2/2/7
stack-port 2/2/1 2/2/6
stack enable
stack mac 748e.f8ce.2e9c
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,001
1,821
113
29
fohdeesha.com
Thank you, fohdeesha!

stack unit 1
module 1 icx6610-48p-poe-port-management-module
module 2 icx6610-qsfp-10-port-160g-module
module 3 icx6610-8-port-10g-dual-mode-module
priority 128
stack-port 1/2/1 1/2/6
stack unit 2
module 1 icx6610-48p-poe-port-management-module
module 2 icx6610-qsfp-10-port-160g-module
module 3 icx6610-8-port-10g-dual-mode-module
stack-trunk 2/2/6 to 2/2/7
stack-port 2/2/1 2/2/6
stack enable
stack mac 748e.f8ce.2e9c

enable
conf t
stack unit 1
stack-trunk 1/2/6 to 1/2/7
exit
exit
write mem


might need to reload the stack after, not sure
 

StarsAndBars

New Member
Jul 13, 2020
21
2
3
Thanks again, still getting an error though. I run these commands on unit 1, correct?

telnet@ICX6610-1#conf t
telnet@ICX6610-1(config)#stack unit 1
telnet@ICX6610-1(config-unit-1)#stack-trunk 1/2/6 to 2/2/7
Error! 2/2/7 is not in unit 1.
telnet@ICX6610-1(config-unit-1)#
 

kapone

Well-Known Member
May 23, 2015
824
406
63
Thanks again, still getting an error though. I run these commands on unit 1, correct?

telnet@ICX6610-1#conf t
telnet@ICX6610-1(config)#stack unit 1
telnet@ICX6610-1(config-unit-1)#stack-trunk 1/2/6 to 2/2/7
Error! 2/2/7 is not in unit 1.
telnet@ICX6610-1(config-unit-1)#
@fohdeesha said... "stack-trunk 1/2/6 to 1/2/7"
You executed..."stack-trunk 1/2/6 to 2/2/7"
 
  • Like
Reactions: Jason Antes

TheCodeLife

New Member
Mar 29, 2019
25
3
3
@LodeRunner Thanks for the information! I will have to look into building an image for it. I have to admit my patience is wearing thin on this switch. I don't have a lot of time and I may just give up on the switch and do a NAND scrub as you did.

@sean I will have to briefly look into musl-cross-make and see if there's a way I can cross-compile the tpm-tools package. However, as I said before, I don't have much time to spare on this, so if I can't get it working relatively easily, I'll probably wipe the device.
 

infoMatt

Active Member
Apr 16, 2019
182
76
28
Sorry, missed a notification and life got me away from this thread (not worry, not for bad things).

The 6450-24P says it uses around 25 watts of power which is nice
Yea... it says... buuut...

Also, FYI my new 6450-24P uses a constant 35ish watts (no inline power on), which is about 10 higher than I expected.
Mine does this as well, same power usage as another user in this thread. I don't know if it was a typo in the brochure or if the power supply used has changed during the lifetime of the model, or whatnot... but yea, the power usage is higher than what it says.

for another 15 watts I also may have lived with another 6450-48P or a 7250-48P (50 watts) for the cheaper price of the switches. lol Ohh well.
Fair enough, but the real question is: are we sure that the 6450-48P will draw the stated 50W, or would it exceed like the little brother? I don't have an answer about this.
 

Wesumat

New Member
Oct 16, 2019
3
1
3
Germany
Hi,

has anyone an idea if it´s possible to replace the complete PSU (main and POE) from an ICX6450-24P with parts from an ICX6400-EPS1500? Is it just the same main psu with 3 parallel POE supplies or am i missing something?
 

infoMatt

Active Member
Apr 16, 2019
182
76
28
Hi,

has anyone an idea if it´s possible to replace the complete PSU (main and POE) from an ICX6450-24P with parts from an ICX6400-EPS1500? Is it just the same main psu with 3 parallel POE supplies or am i missing something?
I don't know if it fits inside the chassis... The EPS1500 has a way beefer PSU inside, and I think it's bigger than the one inside the 24P.
 

Weppel

New Member
Aug 22, 2015
4
2
3
Fair enough, but the real question is: are we sure that the 6450-48P will draw the stated 50W, or would it exceed like the little brother? I don't have an answer about this.
My 6450-48P (MY 2016) does 57.7W (230V) booted, idle, no ports connected.
 

kh78

New Member
Mar 31, 2020
27
6
3
G'day Brocadeers

****
EDIT: To clarify, I've got LAG's running to windows hosts and a linux based NAS, but it's OS is fundamentally different to the more vanilla systemd, Clear Linux distro I'm trying below, so not of much help...
****

Has anyone managed to get a LACP (802.3ad) LAG running between an ICX 6610 and a bond on a Linux server?
What settings were you running?

I'm finding a few things, most notably that the Brocade is basically putting the ports straight into LACP-BLOCKED. Secondly, looking at it with wireshark, it doesn't seem to be respecting the 'lacp-timeout long' command, as whilst it's setting the flag, it's still spewing the lacpdu's out at 1 per second, which is contrary to what brocade's and all other documentation states...perhaps it just spits out them out quickly until the LAG comes up and then backs off?

The closest I have been able to achieve thus far with this distro, results in flapping at best on the LAG. I'm trying to get it running on just two physical gig eth ports with no vlan tagging as a starting point. I've tried two different physical boxes and sets of cables (though both running the same Clear Linux distro, latest release).

Any hints on debugging on the brocade?
I've looked at:
sh int lag id 3
sh lag <name>
debug trunk ctrl
debug trunk event

So far, they haven't been of much help...

The linux server is definitely running mode 4 (802.3ad) and I've tried lacp_rate at both fast and slow, to no avail in either case.
 

kh78

New Member
Mar 31, 2020
27
6
3
Hmmm, seems like there is something suspicious in NetworkManager (at least on Clear Linux). Just built 'the same' bond using iproute2 commands, and the LAG on the ICX 6610 has come up.

Catting around in these two:
/proc/net/bonding/bond0/
/sys/class/net/bond0/bonding/
Suggests the configuration is identical in the major areas (mode, miimon, lacp_rate)....but one works and the other doesn't seem too....

Looking at the flags in Wireshark, there seems to be different lacp flags coming from the Linux box depending on if the bond was config'd via NetworkManager compared to iproute2's 'ip link add/set' commands. Smells....buggy, or else a nasty difference in defaults on the other parameters. This is going to be a long and boring dig to find out why...
 

kh78

New Member
Mar 31, 2020
27
6
3
Disregard, I had ',' instead of '.' in the first nmcli command in the script that I failed to notice (bond.options).

Don't worry, I'll show myself out.....
 
  • Like
Reactions: fohdeesha

dragonian

New Member
Jan 3, 2020
13
1
3
I'm trying to figure out the best or easiest way to fix my inter-vlan routing issues.

I currently an using a OPNsense firewall/ router (Protectli) for the router on the stick paradigm connected via 2x 1GbE LAG to ICX6450. I'm not trying to do anything crazy for VLANs, just LAN, GUEST, MGMT, and CAMERA.
I am seeing an issue with WAN timeouts when the router is forced to route from CAMERA to LAN (for storage).

I'd like to keep opnsense for most DHCP, firewall duties, IPV6, multicast, etc. I'm sure that a lot of this is due to the nice GUI, and visualizations. Maybe this is wrong but i have fear that the 5 year old L3 routing code is not always going to be sufficient.

I've looked a lot of posts in this thread with similar topics, but haven't seen a "good solution" [in my probably flawed opinion]. Re: here or here , etc.

I had hoped that the LAG would give another path for WAN packets for streaming music & skype connections to not be interfered when the CAMERA copy is taking place, but that doesn't seem to be the case. [Un]fortunately my NAS can sustain ~350 MBps writes, so bi directional will kill the 1 GbE link.
10 GbE is not an option for this firewall box at the moment.
I don't think this issue is CPU bound on the opnsense box.. It gets up to 40-50%.. unless it's a single core issue.

Is there some LAG configuration that I could use to make this connection better?
Would some QoS PCP values make anything better?
I'd probably even consider limiting the bandwidth coming out of the CAMERA vlan/intfc. Is there a good way to do that?

Is there some way that I can add simple ACL/ route for just 192.168.60.10 <=>192.168.10.12, so this traffic doesn't need to go up to the router?
I tried to do this as per various tutorials, but since the gateway is opnsense, it goes there to get routed to the other vlan, and doesn't use the VE's that I setup on the 6450.

Or am i forced to give up this opnsense router on a stick, change the gateways, add Transit VLAN, etc. Embrace change!
I know if wanted to route at 10G, then i'd need to go there, but I would think that this setup could handle a 1G link.

Thanks!
 

csementuh

Member
Oct 7, 2019
30
8
8
Pittsburgh, PA
Thanks for the power talk guys. I'm happy enough with this 6450-24P. It sits in my office for now and will then get a 10G fiber link to my shop at some point.


I'm trying to figure out the best or easiest way to fix my inter-vlan routing issues.
How many and what types of cameras do you have? The NAS should be irrelevant, most IP cams use like 4-8 mbps of bandwidth.

If you want the camera traffic to stay in one subnet/VLAN then just setup a VE and set the gateway of the camera to the VE's IP. It will then route on the switch at wire speed directly to the NAS interface on the other subnet/VLAN. Your router will never even see the traffic. The router only needs to see some other interface to view/manage the cams, port forward for external viewing etc.

I do this at home with 12 POE IP cams, a NVR box, a 7250-48P and an Untangle UTM setup.

Have you tried that?
 

dragonian

New Member
Jan 3, 2020
13
1
3
How many and what types of cameras do you have? The NAS should be irrelevant, most IP cams use like 4-8 mbps of bandwidth.

If you want the camera traffic to stay in one subnet/VLAN then just setup a VE and set the gateway of the camera to the VE's IP. It will then route on the switch at wire speed directly to the NAS interface on the other subnet/VLAN. Your router will never even see the traffic. The router only needs to see some other interface to view/manage the cams, port forward for external viewing etc.
Only 2 cams (one is 4k), but I'm using a Blue Iris sw running on a little windows box as the NVR. They are recording to local to that box storage, and then archiving after a day or so to the NAS. The slug happens when it copies 8GB to the nas from blueiris.

I tried to set the gateway on the blue iris box to the VE, but that didn't work. It made the network angry.. and seemed like one way comms. I'm assuming that I would have also needed a default route back from the firewall. This is the point that I realized that I think i needed a transit vlan.
 

dragonian

New Member
Jan 3, 2020
13
1
3
I don't think this issue is CPU bound on the opnsense box.. It gets up to 40-50%.. unless it's a single core issue.
hmm, maybe I am CPU bound. :(
Code:
CPU:  0.4% user,  0.0% nice, 25.7% system, 20.8% interrupt, 53.1% idle
That seems like 1 core doing firewall checks, 1 handling getting packets off the wire.
 

Epsilonson

New Member
Jul 29, 2020
3
0
1
Hi All,
Just picked up a ICX-6610-48p and was getting started on update and configuration for it. Forgive me if this was answered earlier in the thread I couldn't find an answer via search.

Mine has two licenses on it but I want to update the firmware. Will that wipe out the license files?

I have 4 VLANs set up via a pfsense box + L2 switch. Is it best practice to move over everything that only sees LAN (3 of them) to the switch?

Any tips or tricks when setting these switches up I should know about? I am new to Brocade hardware.

Thanks