Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[fohdeesha]

Just got a 6450-48p to replace my aruba s2500-24p, and have it all up and running now. I noticed in the console log a few messages like this:
Error: I2C access failed for device 0x50, command -1071879421, I2C code = 0x1, SIM Code = 14, TWSI Sts = 0xf8
Anyone know what that means?

I can't remember what device 0x50 is on the 6450. You can try running and posting the output of "dm i2c diag" at the enable cli level. There's also a command in the u-boot bootloader to list the i2c devices by name but I can't remember it off hand. Could be a temp sensor, could be an LED controller, could be the PoE controller, etc. I would test PoE to make sure it's not that

 

[gregsachs]

I can't remember what device 0x50 is on the 6450. You can try running and posting the output of "dm i2c diag" at the enable cli level. There's also a command in the u-boot bootloader to list the i2c devices by name but I can't remember it off hand. Could be a temp sensor, could be an LED controller, could be the PoE controller, etc. I would test PoE to make sure it's not that

It may have been related to an optic, hasn't shown since I removed one I wasn't using.
dm i2c diag shows:
stack: 008779b4 0171327c 016e9b34 0123f218 0123f68c 00454a94 00459238 00455c8c 00459ab0 00455c8c 00459ab0 0044f510 00879f54 002c0438 00a2342c 019d86e0 019d93f0 00a23dec 016d9d0c 0228f2ac 02fc7790

POE is working, at least on the ports I have tested.
Thanks!

 

[hmw]

There's also a command in the u-boot bootloader to list the i2c devices by name but I can't remember it off hand.

i2c devices

 

[gregsachs]

ICX6450-48p power draw;
Just as a data point, under low load, the UPS which runs my switch/usg/cable modem is showing a typical load of 73w with 17w of PoE devices. I suspect that there is a minimum value the P/S and electronics will draw regardless of PoE load. This is really no different than my S2500-24p was, with ~10 gb ports and 1 SFP+ used.

 

[Spearfoot]

I am having an issue where two of the four ports when using a breakout cable remain down. I have managed to get all 4 up after a switch reboot, however after disabling and re-enabling the port 2 of them never come back. This is repeatable.


I have tried looking into logs and STP, but not too sure what the issue is here, the ports are not in a LAG, as ESXi does not require it. Any ideas?

The switch is on the firmware that is listed in the guide, license is good and the cable is an FS.com QSFP+ to SFP+ breakout cable. The NICs on the server are intel x520's.

I've had the same experience with one of my ESXi servers equipped with an Intel X520-DA2 connected to the ICX 6610 breakout ports. I found that reloading the NIC with these esxcli commands brings the connections back up:

#!/bin/sh

# Reloading the ICX6610 seems to take down the Intel X520-DA2 NIC
# installed in FALCON. This simple script brings it back up.

ssh root@falcon esxcli network nic up -n vmnic4
ssh root@falcon esxcli network nic up -n vmnic5

 

[Jason Antes]

I have a Netgear ReadyNAS 2120 that I want to bond the NIC's on. The NAS supports Adaptive Load Balancing, Round-robin, IEEE 802.3ad LACP, Transmit Load balancing, XOR, Active Backup, and broadcast as bond types. I figured I'd want Adaptive as I don't think that would require config at the switch and be easiest. Would I need to set up a LAG or anything else on the switch for RR, ALB, LACP, or TLB? Those are the ones I am considering. I know that I don't have to do anything with Windows 2012R2 for the bonds as I use the switch independent option.

My switch is a 6610, though I may set it up on my VDX6740 at some point if I decide to switch to it since I am only using the 6610 for POE and I only have 1 POE device. The VDX uses less power.

 

[Roelf Zomerman]

I have a Netgear ReadyNAS 2120 that I want to bond the NIC's on. The NAS supports Adaptive Load Balancing, Round-robin, IEEE 802.3ad LACP, Transmit Load balancing, XOR, Active Backup, and broadcast as bond types. I figured I'd want Adaptive as I don't think that would require config at the switch and be easiest. Would I need to set up a LAG or anything else on the switch for RR, ALB, LACP, or TLB? Those are the ones I am considering. I know that I don't have to do anything with Windows 2012R2 for the bonds as I use the switch independent option.

My switch is a 6610, though I may set it up on my VDX6740 at some point if I decide to switch to it since I am only using the 6610 for POE and I only have 1 POE device. The VDX uses less power.

See if you can enable SMB3.1 on that NAS.. if that is the case, you will automatically use SMB multichannel and each NIC can run independently with its own IP address.. you clients (with 10G or dual NIC's) will use the bandwidth automatically - Breaking the 1Gbps barrier with Synology and Windows 10
[edit] should be supported on their latest updates[/edit]

 

[gregsachs]

Is it possible to rate-limit traffic on a specific vlan?
IE I'm running all VLANs trunked to firewall currently on e 1/1/1. I'd like to limit one of those vlans to say 15MB/s, to preserve bandwidth for other uses. Is that possible?

 

[Spearfoot]

So essentially just bringing the interfaces back up. Hmm has anyone found out the cause of this? Does it happen with other nics?

It's weird though, I actually rebooted the hosts and the interfaces didn't come up back.

I don't recall this happening with my other ESXi server connected to the ICX 6610 breakout ports -- it's running a SolarFlare SFN6122F NIC.

Maybe a quirk of the Intel firmware? I dunno...

 

[Jason Antes]

See if you can enable SMB3.1 on that NAS.. if that is the case, you will automatically use SMB multichannel and each NIC can run independently with its own IP address.. you clients (with 10G or dual NIC's) will use the bandwidth automatically - Breaking the 1Gbps barrier with Synology and Windows 10
[edit] should be supported on their latest updates[/edit]

I'll check it, I know it does SMB3. Having both set doesn't immediately boot me off the console so that's a good sign. This NAS only has dual Gb connections.

 

[ICXGURU]

Is it possible to rate-limit traffic on a specific vlan?
IE I'm running all VLANs trunked to firewall currently on e 1/1/1. I'd like to limit one of those vlans to say 15MB/s, to preserve bandwidth for other uses. Is that possible?

You can assign a VE to the vlan and rate-limit that.

Create your traffic policy to set the rate, then assign it to the traffic of interest in an ACL, then apply the ACL to the VE.

traffic-policy limiter rate-limit fixed 15000 exceed-action drop
!
ip access-list extended ratelimiter
sequence 10 permit ip any any traffic-policy limiter
!
interface ve 1
ip access-group ratelimiter in

 

[gregsachs]

You can assign a VE to the vlan and rate-limit that.

Create your traffic policy to set the rate, then assign it to the traffic of interest in an ACL, then apply the ACL to the VE.

traffic-policy limiter rate-limit fixed 15000 exceed-action drop
!
ip access-list extended ratelimiter
sequence 10 permit ip any any traffic-policy limiter
!
interface ve 1
ip access-group ratelimiter in

Thanks, I'm going need to play with that!

 

[infoMatt]

You can assign a VE to the vlan and rate-limit that.

Create your traffic policy to set the rate, then assign it to the traffic of interest in an ACL, then apply the ACL to the VE.

traffic-policy limiter rate-limit fixed 15000 exceed-action drop
!
ip access-list extended ratelimiter
sequence 10 permit ip any any traffic-policy limiter
!
interface ve 1
ip access-group ratelimiter in

But applying a rate limit on a VE will affect only the traffic routed by the switch itself, am I right?
@gregsachs said that

I'm running all VLANs trunked to firewall currently on e 1/1/1.

so the gateway is the firewall and not the VE on the switch... sooooo, it isn't going to be policed...

 

[gregsachs]

But applying a rate limit on a VE will affect only the traffic routed by the switch itself, am I right?
@gregsachs said that



so the gateway is the firewall and not the VE on the switch... sooooo, it isn't going to be policed...

Yeah, I caught that, part of why i said I need to play with it.
Thanks

 

[windwalka1990]

Very new to Brocade switches, I've followed the guide for initial setup and was just about to begin configuring this new switch as my core but I'm pretty stumped on these syslogs I'm getting in the console...


SYSLOG: <14> Aug 7 09:31:54 CORE-POE1 System: SSL server 45.33.2.79:443 is disconnected
SYSLOG: <14> Aug 7 09:32:24 CORE-POE1 System: SSL server 45.33.23.183:443 is disconnected
SYSLOG: <14> Aug 7 09:32:54 CORE-POE1 System: SSL server 45.56.79.23:443 is disconnected
SYSLOG: <14> Aug 7 09:33:24 CORE-POE1 System: SSL server 45.79.19.196:443 is disconnected
SYSLOG: <14> Aug 7 09:33:54 CORE-POE1 System: SSL server 96.126.123.244:443 is disconnected
SYSLOG: <14> Aug 7 09:34:24 CORE-POE1 System: SSL server 198.58.118.167:443 is disconnected

I've gone over a command guide and looked over my running-config and nothing is jumping out at me. Any ideas to check?

EDIT: In case it matters this is an ICX 7250p-24p model.

Really appreciate any help on this!

 

[Spearfoot]

Very new to Brocade switches, I've followed the guide for initial setup and was just about to begin configuring this new switch as my core but I'm pretty stumped on these syslogs I'm getting in the console...


SYSLOG: <14> Aug 7 09:31:54 CORE-POE1 System: SSL server 45.33.2.79:443 is disconnected
SYSLOG: <14> Aug 7 09:32:24 CORE-POE1 System: SSL server 45.33.23.183:443 is disconnected
SYSLOG: <14> Aug 7 09:32:54 CORE-POE1 System: SSL server 45.56.79.23:443 is disconnected
SYSLOG: <14> Aug 7 09:33:24 CORE-POE1 System: SSL server 45.79.19.196:443 is disconnected
SYSLOG: <14> Aug 7 09:33:54 CORE-POE1 System: SSL server 96.126.123.244:443 is disconnected
SYSLOG: <14> Aug 7 09:34:24 CORE-POE1 System: SSL server 198.58.118.167:443 is disconnected

I've gone over a command guide and looked over my running-config and nothing is jumping out at me. Any ideas to check?

EDIT: In case it matters this is an ICX 7250p-24p model.

Really appreciate any help on this!

Strange. Those are all hosts on the linode.com domain:

45.33.2.79 : li956-79.members.linode.com
45.33.23.183 : li977-183.members.linode.com
45.56.79.23 : li929-23.members.linode.com
45.79.19.196 : li1118-196.members.linode.com
96.126.123.244 : li372-244.members.linode.com
198.58.118.167 : li647-167.members.linode.com

I noticed something similar when I was setting up my ICX 7150-C12P switches. I looked up the IPs and they were AWS servers. I speculated that the switch was reaching out and downloading POE firmware, or something like that. Here are some of the entries from my SSH log, all showing disconnections from port 443 just like your switch:

34.208.35.231 : ec2-34-208-35-231.us-west-2.compute.amazonaws.com
54.191.58.33 :  ec2-54-191-58-33.us-west-2.compute.amazonaws.com 
54.186.0.83 : ec2-54-186-0-83.us-west-2.compute.amazonaws.com

I never saw anything odd like this setting up my ICX 6610s.

I have no idea what's going on here, and it's a little alarming.

 

[windwalka1990]

Hmm, I'd be more comfortable if it were some AWS servers but when I saw linode I immediately disconnected it from my LAN. I don't see any traffic on other hosts to these domains so I'm pretty sure it's the switch. What gets me is I can't find any trace in the configuration, it's pretty much default except a few VLANs I created. Where would something like this be configured if not there, the OS??

I'll be honest I'm likely going to return this for a Cisco switch, I'm just over my head on these Brocades..

 

[fohdeesha]

It's part of their new SmartZone central management thing, the IP's it's contacting are ruckus services. This behavior should be off by default in the 8080 firmware on my guide, I believe it was 8090 where they enabled it by default. It can be completely disabled and turned off with the following commands:

enable
conf t
no sz registrar
sz disable
write mem

you can confirm it's disabled by running "show sz status", the top line should have "Operation Status: Disabled" and below that "State: DISABLED"

This should not be required on v8080, however if it is, let me know so I can add those commands to the guide. However my money is on the two of you skipping ahead to v8090 or above

 

[i386]

Page 207

 

[spatialbob]

Has anybody tried stacking these switches (I'm interested in ICX6450) with 10GBASE-T (copper)?