Drag to reposition cover

Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

kh78

New Member
Mar 31, 2020
27
6
3
Have you tried mtu-exceed forward in your configuration? I believe this changes the switch's default behavior of dropping packets larger than a port's MTU, so that it will instead fragment and pass them along.
No, I wasn't aware of that. If I ever get excited/bored enough, I'll revisit it. For now it has all just been rolled back and will stay as 1500 mtu other than the SAN switch, as that's all hardware that's happily playing along with jumbo's.
 

gregsachs

Active Member
Aug 14, 2018
321
88
28
Just got a 6450-48p to replace my aruba s2500-24p, and have it all up and running now. I noticed in the console log a few messages like this:
Error: I2C access failed for device 0x50, command -1071879421, I2C code = 0x1, SIM Code = 14, TWSI Sts = 0xf8
Anyone know what that means?
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,003
1,823
113
29
fohdeesha.com
No, I wasn't aware of that. If I ever get excited/bored enough, I'll revisit it. For now it has all just been rolled back and will stay as 1500 mtu other than the SAN switch, as that's all hardware that's happily playing along with jumbo's.
keep in mind as well fragmenting isn't really supposed to happen, so "mtu-exceed forward" is a workaround, and it's done in CPU (too large packets are forwarded to the CPU to be fragmented, then sent back to the ASIC) so it's sloooww
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,003
1,823
113
29
fohdeesha.com
Just got a 6450-48p to replace my aruba s2500-24p, and have it all up and running now. I noticed in the console log a few messages like this:
Error: I2C access failed for device 0x50, command -1071879421, I2C code = 0x1, SIM Code = 14, TWSI Sts = 0xf8
Anyone know what that means?
I can't remember what device 0x50 is on the 6450. You can try running and posting the output of "dm i2c diag" at the enable cli level. There's also a command in the u-boot bootloader to list the i2c devices by name but I can't remember it off hand. Could be a temp sensor, could be an LED controller, could be the PoE controller, etc. I would test PoE to make sure it's not that
 

gregsachs

Active Member
Aug 14, 2018
321
88
28
I can't remember what device 0x50 is on the 6450. You can try running and posting the output of "dm i2c diag" at the enable cli level. There's also a command in the u-boot bootloader to list the i2c devices by name but I can't remember it off hand. Could be a temp sensor, could be an LED controller, could be the PoE controller, etc. I would test PoE to make sure it's not that
It may have been related to an optic, hasn't shown since I removed one I wasn't using.
dm i2c diag shows:
stack: 008779b4 0171327c 016e9b34 0123f218 0123f68c 00454a94 00459238 00455c8c 00459ab0 00455c8c 00459ab0 0044f510 00879f54 002c0438 00a2342c 019d86e0 019d93f0 00a23dec 016d9d0c 0228f2ac 02fc7790

POE is working, at least on the ports I have tested.
Thanks!
 
  • Like
Reactions: fohdeesha

gregsachs

Active Member
Aug 14, 2018
321
88
28
ICX6450-48p power draw;
Just as a data point, under low load, the UPS which runs my switch/usg/cable modem is showing a typical load of 73w with 17w of PoE devices. I suspect that there is a minimum value the P/S and electronics will draw regardless of PoE load. This is really no different than my S2500-24p was, with ~10 gb ports and 1 SFP+ used.
 

shank

New Member
Jul 19, 2020
3
0
1
I am having an issue where two of the four ports when using a breakout cable remain down. I have managed to get all 4 up after a switch reboot, however after disabling and re-enabling the port 2 of them never come back. This is repeatable.


I have tried looking into logs and STP, but not too sure what the issue is here, the ports are not in a LAG, as ESXi does not require it. Any ideas?

The switch is on the firmware that is listed in the guide, license is good and the cable is an FS.com QSFP+ to SFP+ breakout cable. The NICs on the server are intel x520's.
 
Last edited:

Spearfoot

Member
Apr 22, 2015
68
26
18
62
I am having an issue where two of the four ports when using a breakout cable remain down. I have managed to get all 4 up after a switch reboot, however after disabling and re-enabling the port 2 of them never come back. This is repeatable.


I have tried looking into logs and STP, but not too sure what the issue is here, the ports are not in a LAG, as ESXi does not require it. Any ideas?

The switch is on the firmware that is listed in the guide, license is good and the cable is an FS.com QSFP+ to SFP+ breakout cable. The NICs on the server are intel x520's.
I've had the same experience with one of my ESXi servers equipped with an Intel X520-DA2 connected to the ICX 6610 breakout ports. I found that reloading the NIC with these esxcli commands brings the connections back up:
Code:
#!/bin/sh

# Reloading the ICX6610 seems to take down the Intel X520-DA2 NIC
# installed in FALCON. This simple script brings it back up.

ssh root@falcon esxcli network nic up -n vmnic4
ssh root@falcon esxcli network nic up -n vmnic5
 

Jason Antes

Member
Feb 28, 2020
101
22
18
Twin Cities
I have a Netgear ReadyNAS 2120 that I want to bond the NIC's on. The NAS supports Adaptive Load Balancing, Round-robin, IEEE 802.3ad LACP, Transmit Load balancing, XOR, Active Backup, and broadcast as bond types. I figured I'd want Adaptive as I don't think that would require config at the switch and be easiest. Would I need to set up a LAG or anything else on the switch for RR, ALB, LACP, or TLB? Those are the ones I am considering. I know that I don't have to do anything with Windows 2012R2 for the bonds as I use the switch independent option.

My switch is a 6610, though I may set it up on my VDX6740 at some point if I decide to switch to it since I am only using the 6610 for POE and I only have 1 POE device. The VDX uses less power.
 
Jan 10, 2019
99
18
8
blog.azureinfra.com
I have a Netgear ReadyNAS 2120 that I want to bond the NIC's on. The NAS supports Adaptive Load Balancing, Round-robin, IEEE 802.3ad LACP, Transmit Load balancing, XOR, Active Backup, and broadcast as bond types. I figured I'd want Adaptive as I don't think that would require config at the switch and be easiest. Would I need to set up a LAG or anything else on the switch for RR, ALB, LACP, or TLB? Those are the ones I am considering. I know that I don't have to do anything with Windows 2012R2 for the bonds as I use the switch independent option.

My switch is a 6610, though I may set it up on my VDX6740 at some point if I decide to switch to it since I am only using the 6610 for POE and I only have 1 POE device. The VDX uses less power.
See if you can enable SMB3.1 on that NAS.. if that is the case, you will automatically use SMB multichannel and each NIC can run independently with its own IP address.. you clients (with 10G or dual NIC's) will use the bandwidth automatically - Breaking the 1Gbps barrier with Synology and Windows 10
[edit] should be supported on their latest updates[/edit]
 
  • Like
Reactions: Jason Antes

gregsachs

Active Member
Aug 14, 2018
321
88
28
Is it possible to rate-limit traffic on a specific vlan?
IE I'm running all VLANs trunked to firewall currently on e 1/1/1. I'd like to limit one of those vlans to say 15MB/s, to preserve bandwidth for other uses. Is that possible?
 
  • Like
Reactions: tommybackeast

shank

New Member
Jul 19, 2020
3
0
1
I've had the same experience with one of my ESXi servers equipped with an Intel X520-DA2 connected to the ICX 6610 breakout ports. I found that reloading the NIC with these esxcli commands brings the connections back up:
Code:
#!/bin/sh

# Reloading the ICX6610 seems to take down the Intel X520-DA2 NIC
# installed in FALCON. This simple script brings it back up.

ssh root@falcon esxcli network nic up -n vmnic4
ssh root@falcon esxcli network nic up -n vmnic5
So essentially just bringing the interfaces back up. Hmm has anyone found out the cause of this? Does it happen with other nics?

It's weird though, I actually rebooted the hosts and the interfaces didn't come up back.
 
Last edited:

Spearfoot

Member
Apr 22, 2015
68
26
18
62
So essentially just bringing the interfaces back up. Hmm has anyone found out the cause of this? Does it happen with other nics?

It's weird though, I actually rebooted the hosts and the interfaces didn't come up back.
I don't recall this happening with my other ESXi server connected to the ICX 6610 breakout ports -- it's running a SolarFlare SFN6122F NIC.

Maybe a quirk of the Intel firmware? I dunno...
 

Jason Antes

Member
Feb 28, 2020
101
22
18
Twin Cities
See if you can enable SMB3.1 on that NAS.. if that is the case, you will automatically use SMB multichannel and each NIC can run independently with its own IP address.. you clients (with 10G or dual NIC's) will use the bandwidth automatically - Breaking the 1Gbps barrier with Synology and Windows 10
[edit] should be supported on their latest updates[/edit]
I'll check it, I know it does SMB3. Having both set doesn't immediately boot me off the console so that's a good sign. This NAS only has dual Gb connections.
 
  • Like
Reactions: tommybackeast

ICXGURU

Member
Jun 22, 2020
36
15
8
Is it possible to rate-limit traffic on a specific vlan?
IE I'm running all VLANs trunked to firewall currently on e 1/1/1. I'd like to limit one of those vlans to say 15MB/s, to preserve bandwidth for other uses. Is that possible?
You can assign a VE to the vlan and rate-limit that.

Create your traffic policy to set the rate, then assign it to the traffic of interest in an ACL, then apply the ACL to the VE.

traffic-policy limiter rate-limit fixed 15000 exceed-action drop
!
ip access-list extended ratelimiter
sequence 10 permit ip any any traffic-policy limiter
!
interface ve 1
ip access-group ratelimiter in
 

gregsachs

Active Member
Aug 14, 2018
321
88
28
You can assign a VE to the vlan and rate-limit that.

Create your traffic policy to set the rate, then assign it to the traffic of interest in an ACL, then apply the ACL to the VE.

traffic-policy limiter rate-limit fixed 15000 exceed-action drop
!
ip access-list extended ratelimiter
sequence 10 permit ip any any traffic-policy limiter
!
interface ve 1
ip access-group ratelimiter in
Thanks, I'm going need to play with that!
 

infoMatt

Active Member
Apr 16, 2019
183
77
28
You can assign a VE to the vlan and rate-limit that.

Create your traffic policy to set the rate, then assign it to the traffic of interest in an ACL, then apply the ACL to the VE.

traffic-policy limiter rate-limit fixed 15000 exceed-action drop
!
ip access-list extended ratelimiter
sequence 10 permit ip any any traffic-policy limiter
!
interface ve 1
ip access-group ratelimiter in
But applying a rate limit on a VE will affect only the traffic routed by the switch itself, am I right?
@gregsachs said that

I'm running all VLANs trunked to firewall currently on e 1/1/1.
so the gateway is the firewall and not the VE on the switch... sooooo, it isn't going to be policed...
 

gregsachs

Active Member
Aug 14, 2018
321
88
28
But applying a rate limit on a VE will affect only the traffic routed by the switch itself, am I right?
@gregsachs said that



so the gateway is the firewall and not the VE on the switch... sooooo, it isn't going to be policed...
Yeah, I caught that, part of why i said I need to play with it.
Thanks