Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)
- Thread starter fohdeesha
- Start date
Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
Yeah I had a typo in my original post, edited it about 30 seconds later, bad timing. Follow what it says now
Thanks, I wondered until I looked more carefully at it. I think I am good now. Thanks again for helping me with this!
@LodeRunner Thanks for the information! I will have to look into building an image for it. I have to admit my patience is wearing thin on this switch. I don't have a lot of time and I may just give up on the switch and do a NAND scrub as you did.
@sean I will have to briefly look into musl-cross-make and see if there's a way I can cross-compile the tpm-tools package. However, as I said before, I don't have much time to spare on this, so if I can't get it working relatively easily, I'll probably wipe the device.
@sean I will have to briefly look into musl-cross-make and see if there's a way I can cross-compile the tpm-tools package. However, as I said before, I don't have much time to spare on this, so if I can't get it working relatively easily, I'll probably wipe the device.
Sorry, missed a notification and life got me away from this thread (not worry, not for bad things).
Yea... it says... buuut...
Mine does this as well, same power usage as another user in this thread. I don't know if it was a typo in the brochure or if the power supply used has changed during the lifetime of the model, or whatnot... but yea, the power usage is higher than what it says.
Fair enough, but the real question is: are we sure that the 6450-48P will draw the stated 50W, or would it exceed like the little brother? I don't have an answer about this.
I don't know if it fits inside the chassis... The EPS1500 has a way beefer PSU inside, and I think it's bigger than the one inside the 24P.
G'day Brocadeers
****
EDIT: To clarify, I've got LAG's running to windows hosts and a linux based NAS, but it's OS is fundamentally different to the more vanilla systemd, Clear Linux distro I'm trying below, so not of much help...
****
Has anyone managed to get a LACP (802.3ad) LAG running between an ICX 6610 and a bond on a Linux server?
What settings were you running?
I'm finding a few things, most notably that the Brocade is basically putting the ports straight into LACP-BLOCKED. Secondly, looking at it with wireshark, it doesn't seem to be respecting the 'lacp-timeout long' command, as whilst it's setting the flag, it's still spewing the lacpdu's out at 1 per second, which is contrary to what brocade's and all other documentation states...perhaps it just spits out them out quickly until the LAG comes up and then backs off?
The closest I have been able to achieve thus far with this distro, results in flapping at best on the LAG. I'm trying to get it running on just two physical gig eth ports with no vlan tagging as a starting point. I've tried two different physical boxes and sets of cables (though both running the same Clear Linux distro, latest release).
Any hints on debugging on the brocade?
I've looked at:
sh int lag id 3
sh lag <name>
debug trunk ctrl
debug trunk event
So far, they haven't been of much help...
The linux server is definitely running mode 4 (802.3ad) and I've tried lacp_rate at both fast and slow, to no avail in either case.
****
EDIT: To clarify, I've got LAG's running to windows hosts and a linux based NAS, but it's OS is fundamentally different to the more vanilla systemd, Clear Linux distro I'm trying below, so not of much help...
****
Has anyone managed to get a LACP (802.3ad) LAG running between an ICX 6610 and a bond on a Linux server?
What settings were you running?
I'm finding a few things, most notably that the Brocade is basically putting the ports straight into LACP-BLOCKED. Secondly, looking at it with wireshark, it doesn't seem to be respecting the 'lacp-timeout long' command, as whilst it's setting the flag, it's still spewing the lacpdu's out at 1 per second, which is contrary to what brocade's and all other documentation states...perhaps it just spits out them out quickly until the LAG comes up and then backs off?
The closest I have been able to achieve thus far with this distro, results in flapping at best on the LAG. I'm trying to get it running on just two physical gig eth ports with no vlan tagging as a starting point. I've tried two different physical boxes and sets of cables (though both running the same Clear Linux distro, latest release).
Any hints on debugging on the brocade?
I've looked at:
sh int lag id 3
sh lag <name>
debug trunk ctrl
debug trunk event
So far, they haven't been of much help...
The linux server is definitely running mode 4 (802.3ad) and I've tried lacp_rate at both fast and slow, to no avail in either case.
Hmmm, seems like there is something suspicious in NetworkManager (at least on Clear Linux). Just built 'the same' bond using iproute2 commands, and the LAG on the ICX 6610 has come up.
Catting around in these two:
/proc/net/bonding/bond0/
/sys/class/net/bond0/bonding/
Suggests the configuration is identical in the major areas (mode, miimon, lacp_rate)....but one works and the other doesn't seem too....
Looking at the flags in Wireshark, there seems to be different lacp flags coming from the Linux box depending on if the bond was config'd via NetworkManager compared to iproute2's 'ip link add/set' commands. Smells....buggy, or else a nasty difference in defaults on the other parameters. This is going to be a long and boring dig to find out why...
Catting around in these two:
/proc/net/bonding/bond0/
/sys/class/net/bond0/bonding/
Suggests the configuration is identical in the major areas (mode, miimon, lacp_rate)....but one works and the other doesn't seem too....
Looking at the flags in Wireshark, there seems to be different lacp flags coming from the Linux box depending on if the bond was config'd via NetworkManager compared to iproute2's 'ip link add/set' commands. Smells....buggy, or else a nasty difference in defaults on the other parameters. This is going to be a long and boring dig to find out why...
I'm trying to figure out the best or easiest way to fix my inter-vlan routing issues.
I currently an using a OPNsense firewall/ router (Protectli) for the router on the stick paradigm connected via 2x 1GbE LAG to ICX6450. I'm not trying to do anything crazy for VLANs, just LAN, GUEST, MGMT, and CAMERA.
I am seeing an issue with WAN timeouts when the router is forced to route from CAMERA to LAN (for storage).
I'd like to keep opnsense for most DHCP, firewall duties, IPV6, multicast, etc. I'm sure that a lot of this is due to the nice GUI, and visualizations. Maybe this is wrong but i have fear that the 5 year old L3 routing code is not always going to be sufficient.
I've looked a lot of posts in this thread with similar topics, but haven't seen a "good solution" [in my probably flawed opinion]. Re: here or here , etc.
I had hoped that the LAG would give another path for WAN packets for streaming music & skype connections to not be interfered when the CAMERA copy is taking place, but that doesn't seem to be the case. [Un]fortunately my NAS can sustain ~350 MBps writes, so bi directional will kill the 1 GbE link.
10 GbE is not an option for this firewall box at the moment.
I don't think this issue is CPU bound on the opnsense box.. It gets up to 40-50%.. unless it's a single core issue.
Is there some LAG configuration that I could use to make this connection better?
Would some QoS PCP values make anything better?
I'd probably even consider limiting the bandwidth coming out of the CAMERA vlan/intfc. Is there a good way to do that?
Is there some way that I can add simple ACL/ route for just 192.168.60.10 <=>192.168.10.12, so this traffic doesn't need to go up to the router?
I tried to do this as per various tutorials, but since the gateway is opnsense, it goes there to get routed to the other vlan, and doesn't use the VE's that I setup on the 6450.
Or am i forced to give up this opnsense router on a stick, change the gateways, add Transit VLAN, etc. Embrace change!
I know if wanted to route at 10G, then i'd need to go there, but I would think that this setup could handle a 1G link.
Thanks!
I currently an using a OPNsense firewall/ router (Protectli) for the router on the stick paradigm connected via 2x 1GbE LAG to ICX6450. I'm not trying to do anything crazy for VLANs, just LAN, GUEST, MGMT, and CAMERA.
I am seeing an issue with WAN timeouts when the router is forced to route from CAMERA to LAN (for storage).
I'd like to keep opnsense for most DHCP, firewall duties, IPV6, multicast, etc. I'm sure that a lot of this is due to the nice GUI, and visualizations. Maybe this is wrong but i have fear that the 5 year old L3 routing code is not always going to be sufficient.
I've looked a lot of posts in this thread with similar topics, but haven't seen a "good solution" [in my probably flawed opinion]. Re: here or here , etc.
I had hoped that the LAG would give another path for WAN packets for streaming music & skype connections to not be interfered when the CAMERA copy is taking place, but that doesn't seem to be the case. [Un]fortunately my NAS can sustain ~350 MBps writes, so bi directional will kill the 1 GbE link.
10 GbE is not an option for this firewall box at the moment.
I don't think this issue is CPU bound on the opnsense box.. It gets up to 40-50%.. unless it's a single core issue.
Is there some LAG configuration that I could use to make this connection better?
Would some QoS PCP values make anything better?
I'd probably even consider limiting the bandwidth coming out of the CAMERA vlan/intfc. Is there a good way to do that?
Is there some way that I can add simple ACL/ route for just 192.168.60.10 <=>192.168.10.12, so this traffic doesn't need to go up to the router?
I tried to do this as per various tutorials, but since the gateway is opnsense, it goes there to get routed to the other vlan, and doesn't use the VE's that I setup on the 6450.
Or am i forced to give up this opnsense router on a stick, change the gateways, add Transit VLAN, etc. Embrace change!
I know if wanted to route at 10G, then i'd need to go there, but I would think that this setup could handle a 1G link.
Thanks!
Thanks for the power talk guys. I'm happy enough with this 6450-24P. It sits in my office for now and will then get a 10G fiber link to my shop at some point.
If you want the camera traffic to stay in one subnet/VLAN then just setup a VE and set the gateway of the camera to the VE's IP. It will then route on the switch at wire speed directly to the NAS interface on the other subnet/VLAN. Your router will never even see the traffic. The router only needs to see some other interface to view/manage the cams, port forward for external viewing etc.
I do this at home with 12 POE IP cams, a NVR box, a 7250-48P and an Untangle UTM setup.
Have you tried that?
How many and what types of cameras do you have? The NAS should be irrelevant, most IP cams use like 4-8 mbps of bandwidth.
If you want the camera traffic to stay in one subnet/VLAN then just setup a VE and set the gateway of the camera to the VE's IP. It will then route on the switch at wire speed directly to the NAS interface on the other subnet/VLAN. Your router will never even see the traffic. The router only needs to see some other interface to view/manage the cams, port forward for external viewing etc.
I do this at home with 12 POE IP cams, a NVR box, a 7250-48P and an Untangle UTM setup.
Have you tried that?
Only 2 cams (one is 4k), but I'm using a Blue Iris sw running on a little windows box as the NVR. They are recording to local to that box storage, and then archiving after a day or so to the NAS. The slug happens when it copies 8GB to the nas from blueiris.
I tried to set the gateway on the blue iris box to the VE, but that didn't work. It made the network angry.. and seemed like one way comms. I'm assuming that I would have also needed a default route back from the firewall. This is the point that I realized that I think i needed a transit vlan.
hmm, maybe I am CPU bound.
Code:
CPU: 0.4% user, 0.0% nice, 25.7% system, 20.8% interrupt, 53.1% idleHi All,
Just picked up a ICX-6610-48p and was getting started on update and configuration for it. Forgive me if this was answered earlier in the thread I couldn't find an answer via search.
Mine has two licenses on it but I want to update the firmware. Will that wipe out the license files?
I have 4 VLANs set up via a pfsense box + L2 switch. Is it best practice to move over everything that only sees LAN (3 of them) to the switch?
Any tips or tricks when setting these switches up I should know about? I am new to Brocade hardware.
Thanks
Just picked up a ICX-6610-48p and was getting started on update and configuration for it. Forgive me if this was answered earlier in the thread I couldn't find an answer via search.
Mine has two licenses on it but I want to update the firmware. Will that wipe out the license files?
I have 4 VLANs set up via a pfsense box + L2 switch. Is it best practice to move over everything that only sees LAN (3 of them) to the switch?
Any tips or tricks when setting these switches up I should know about? I am new to Brocade hardware.
Thanks
Umm...yeah...about 204 pages worth.
Grab a cold one.Guess I asked for that lol..how many pages when you chop out all of the incredibly specific troubleshooting/scenario chaff?Umm...yeah...about 204 pages worth.
