Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[BobTB]

Dont ban me I turned "jumbo" on my icx 6610 . Reload. Now all my ports show MTU 10200. Everything still works. Now I have a few VLANs, nothing fancy. No routing. One is for connecting my NAS for NFS traffic. Four ports only. Nothing larger than 1500 or so passes through. What am I missing? My devices are on MTU9000 on both sides, (ESXI with vswitch set to MTU9000 + vmknic set to MTU9000). What should I set on the switch. I tried setting an interface to "ip mtu 1500" just for fun. Nothing happens. Should the swithc be rebooted everytime?

Can I set all the ports to MTU1500 somehow in one go, and then these four ports to larger MTU? As far as I can see it can only be done port by port. Is this really so?

 

[fohdeesha]

Correct, though more accurately, taking PoE off a port on the switch. So you'd just run a cable along the side over to port 1 and off to the races you go.

Also, I just found the loveliest bug in 08.0.30u...

icx6450(config)# sho run
...
lag ds1511 dynamic id 1
ports ethernet 1/1/35 to 1/1/36
primary-port 1/1/35
deploy
port-name "DS1511.eth0" ethernet 1/1/35
port-name DS1511.eth1 ethernet 1/1/36
...
icx6450(config)#lag ds1511 dynamic id 1
icx6450(config-lag-ds1511)#no port-name "DS1511.eth0" ethernet 1/1/35
Port name doesn't match
icx6450(config-lag-ds1511)#no port-name DS1511.eth0 ethernet 1/1/35
Port name doesn't match

The only way to fix it? Write the config to tftp or scp (don't - you have to make your system insecure as hell because 08.0.30 is that obsolete and dangerous,) manually edit, and reload the whole thing.

how'd you even get it to save the quotation marks into the config, did you manually edit it? It strips them out of the name before saving them to the config, I can't get them in there even if I try. works perfectly fine here:

testicle(config-lag-ds1511)#
testicle(config-lag-ds1511)#port-name "DS1511.eth0" ethernet 1/1/20
testicle(config-lag-ds1511)#port-name DS1511.eth1 ethernet 1/1/21

testicle(config-lag-ds1511)#exit
testicle(config)#sh run | inc port-name
 port-name DS1511.eth0 ethernet 1/1/20
 port-name DS1511.eth1 ethernet 1/1/21


testicle(config)#lag ds1511 dynamic id 1
testicle(config-lag-ds1511)#no port-name DS1511.eth0 ethernet 1/1/20
testicle(config-lag-ds1511)#no port-name DS1511.eth1 ethernet 1/1/21
testicle(config-lag-ds1511)#sh run | inc port-name
testicle(config-lag-ds1511)#

don't - you have to make your system insecure as hell because 08.0.30 is that obsolete and dangerous

SO DANGEROUS BRO

 

[ICXGURU]

how'd you even get it to save the quotation marks into the config, did you manually edit it? It strips them out of the name before saving them to the config, I can't get them in there even if I try. works perfectly fine here:

testicle(config-lag-ds1511)#
testicle(config-lag-ds1511)#port-name "DS1511.eth0" ethernet 1/1/20
testicle(config-lag-ds1511)#port-name DS1511.eth1 ethernet 1/1/21

testicle(config-lag-ds1511)#exit
testicle(config)#sh run | inc port-name
port-name DS1511.eth0 ethernet 1/1/20
port-name DS1511.eth1 ethernet 1/1/21


testicle(config)#lag ds1511 dynamic id 1
testicle(config-lag-ds1511)#no port-name DS1511.eth0 ethernet 1/1/20
testicle(config-lag-ds1511)#no port-name DS1511.eth1 ethernet 1/1/21
testicle(config-lag-ds1511)#sh run | inc port-name
testicle(config-lag-ds1511)#

SO DANGEROUS BRO

Why would you use a quotes when there is no space in the name? The system will remove them anyway as fohdeesha shows above.

 

[ICXGURU]

Dont ban me I turned "jumbo" on my icx 6610 . Reload. Now all my ports show MTU 10200. Everything still works. Now I have a few VLANs, nothing fancy. No routing. One is for connecting my NAS for NFS traffic. Four ports only. Nothing larger than 1500 or so passes through. What am I missing? My devices are on MTU9000 on both sides, (ESXI with vswitch set to MTU9000 + vmknic set to MTU9000). What should I set on the switch. I tried setting an interface to "ip mtu 1500" just for fun. Nothing happens. Should the swithc be rebooted everytime?

Can I set all the ports to MTU1500 somehow in one go, and then these four ports to larger MTU? As far as I can see it can only be done port by port. Is this really so?

You do not need to reboot. You will see the MTU change in the show interface e x/x/x when you make the changes.
Yes, you need to do it per interface (or with a script)
Try "mtu-exceed forward" globally to fragment packets if needed.

 

[hmw]

That is interesting about the Wiitek linking at 2.5 and 5Gig. What did the 7250 say the link speed was in the show interface? I would think it reported 10gig even if the other end was 2.5 or 5 since those ASICs are not multigig.

The Wiitek is based on the Marvell 88x3310 and is indeed multi-gig, however other than the Aquantia ACQ-107, all the other Marvell based transceivers (Ipolex for example) I have tested, cause problems when linking with anything other than 10G (using an ICX-6610)

I've tried traffic shaping and rate limiting the ICX side ports without success (https://forums.servethehome.com/ind...t-marvell-88x3300-v-s-aquantia-aqs-107.30004/)

It would be awesome if you know & can share how to successfully rate limit and get full nbase-t performance on these ICX switches ...

 

[BobTB]

You do not need to reboot. You will see the MTU change in the show interface e x/x/x when you make the changes.
Yes, you need to do it per interface (or with a script)
Try "mtu-exceed forward" globally to fragment packets if needed.

I do not see the MTU change if I do the show interface. It just stays at MTU 10200, I can set it to whatever I want and it will still not change. Any setting that can prevent it to change?

 

[fohdeesha]

I do not see the MTU change if I do the show interface. It just stays at MTU 10200, I can set it to whatever I want and it will still not change. Any setting that can prevent it to change?

are you changing the mtu per interface as well or just using the global command?

 

[ICXGURU]

The Wiitek is based on the Marvell 88x3310 and is indeed multi-gig, however other than the Aquantia ACQ-107, all the other Marvell based transceivers (Ipolex for example) I have tested, cause problems when linking with anything other than 10G (using an ICX-6610)

I've tried traffic shaping and rate limiting the ICX side ports without success (https://forums.servethehome.com/ind...t-marvell-88x3300-v-s-aquantia-aqs-107.30004/)

It would be awesome if you know & can share how to successfully rate limit and get full nbase-t performance on these ICX switches ...

The ICX does not have a multi-gig ASIC, is was designed long before multi-gig existed so it does not know how to negotiate to 2.5 or 5.

I do not have any 6xxx series to test with but Rate-limiting is inbound only so that is not going to help you. You would need to output rate shape like so:
Core(config-if-e1000-1/1/2)# rate-limit output shaping 100000
Outbound Rate Shaping on Port 1/1/2 Config: 100000 Kbps, Actual: 100000 Kbps

 

[BobTB]

are you changing the mtu per interface as well or just using the global command?

I used the global command 'jumbo' to enable it, and then tried to lower individual interfaces one by one, but it does not change the MTU down at all, it just stays at the max of 10200

 

[rootwyrm]

how'd you even get it to save the quotation marks into the config, did you manually edit it? It strips them out of the name before saving them to the config, I can't get them in there even if I try. works perfectly fine here:

Your guess there is as good as mine, but, I confirmed it in copy tftp output and actually on the switch. Conclusion? Has to be a bug. But damned if I could tell you how the hell to trigger it. May be from the 08.0.30t to 30u upgrade itself. I did find a second workaround: delete the whole LAG, then 'no port-name' each individual port, then reload. (Boy, that sounds like something fun to do in production, doesn't it?) It also breaks LAG port-names after it, ordered by the dynamic id.

Also found what I think may be the other 30u bug people were spooked about; IPv6 multicast behavior is just not right. Even when I took dvSwitches out of the equation, the 6450 is just not learning IPv6 IGMP correctly. Which results in double-master and a lot of fabric flooding. So if you're doing anything like CARP or VRRP, I would say stay on 30t. raguard also seems flaky, but that may be purely related to the IGMP in my architecture.
Thankfully I'm moving all of that to OSPFv3 (if I can figure out why the hell frr is sending out the wrong interfaces) so ripping it out was a non-issue. If I was a paying customer? I'd probably be demanding some serious trade-in credit on a 7450 for the outage.

I do not see the MTU change if I do the show interface. It just stays at MTU 10200, I can set it to whatever I want and it will still not change. Any setting that can prevent it to change?

This is one of the most brain-dead aspects of the ICX family, and I mean it is just bad. All of the documentation will tell you "ip mtu 1536" - nope. This is a lie and only works on VRFs and only when you have an IP address assigned. This is NOT what the documentation says. The documentation says "all models can do this." They cannot. You only have three MTU options for non-VRF stuff, period.
1508 (1500), 10200 (9216), or 1536.
"Wait, 1536?!"
Yeah. Brocade is exceedingly stupid when it comes to frames, trust me. And even worse at documentation half the time. If you're passing tagged traffic from anything sane, you need to set jumbo globally first (this sets a register in the silicon because of course the silicon's that bad about it too.) Then set aggregated-vlan which is also a global. aggregated-vlan flips the software switch to accept 1536(1542) frames. Where this gets truly absurdist is: if you set aggregated-vlan without jumbo, it still frags at 1500. And if you just set jumbo and try to do 1536 frames on a port it's decided is 1500, it just chokes because "you didn't set the MTU appropriately!"

Once I figured that out? Yeah. Now I can pass frames between the Junipers and the ICX6450.

 

[BobTB]

If you're passing tagged traffic from anything sane, you need to set jumbo globally first (this sets a register in the silicon because of course the silicon's that bad about it too.) Then set aggregated-vlan which is also a global. aggregated-vlan flips the software switch to accept 1536(1542) frames. Where this gets truly absurdist is: if you set aggregated-vlan without jumbo, it still frags at 1500. And if you just set jumbo and try to do 1536 frames on a port it's decided is 1500, it just chokes because "you didn't set the MTU appropriately!"

Once I figured that out? Yeah. Now I can pass frames between the Junipers and the ICX6450.

Seems you got this figured out,. This is how far I am now - I set the jumbo already, and reload (nice to do when in production), now what I want to do is to have it pass 9000 frames - which it just doesn't - on a few interfaces in one VLAN, I dont have any ve IP address assigned. Should I ? Aggregated-vlan?

 

[Derelict]

I was running Wireshark on Win box first, but then switched to Linux and results were the same. I am pretty sure it is something on the switch, I just don't know if it is expected behavior or not. The reason I am saying this is because:

1. monitored port is on an isolated empty vlan
2. mirror port is on isolated vlan with only Wireshark box connected to it
3. I am not seeing any traffic until I issue command "mirror-port ethernet 1/1/24"
4. I am seeing traffic from multiple totally unrelated VLANs that have nothing to do with mirror or monitor ports
5. All traffic that I am seeing is multicast traffic + some TCP re transmits

I was trying to use mirroring for troubleshooting an issue, but I was able to resolve that issue, so this is not really an issue for me anymore. It would still be nice to figure out why it is happening.

I know this is really old but I am seeing exactly the same thing. The 6610 seems to mirror the monitored port traffic plus at least all multicast traffic on all VLANs on the switch. Once I enable the mirror port it starts sending all the multicast traffic even if the only port being monitored is down. And if I disable the mirror, the traffic doesn't stop. I have to reload the switch. This means, until I reload the stack, anything on that switch port receives all the IPv6 RAs, etc, on every VLAN on the switch.

The traffic is definitely not generated by the capture workstation. It is VRRP, CARP, RAs, etc from other VLANs on the switch stack.

I would love to find a solution because as it is, mirror/monitor is about useless to me.

Code level: FCXR08030u.bin

 

[virulent]

Received my ICX6450-24p today -- everything seems to check out!

Judging by the hostname it was in a barn environment.. opened it up and cleaned out that dust..

Before:

http://imgur.com/45LnTPX

After:

http://imgur.com/Su4Ju5d

Fans are surprisingly not destroyed by the dust.. Idle ~40 dBa (basically same as spec sheet's 39 dBa) and boot ~56 dBa.
For me it is effectively silent next to my T320s! No issues with PoE it looks like either.

I do have a pair of Delta FFB0412HN-5T20 I picked up to compare with, but not sure if I will attempt to swap the pins around tonight or some other time. It's already effectively quiet but I'm more concerned about the dust now. Might as well while I wait for licenses

 

[dswartz]

Very pleasantly surprised by the noise level of my 2 new 2450-24 switches.

 

[virulent]

Okay, swapping fan pins was a lot simpler than I thought (the pins come right out when pushing on the tab with a flathead.)

The acoustics are definitely better. The full speed noise is significantly less than stock. I attempted to measure it, but you can only go so far with a phone app & rough measurements but you get the idea.

ICX6450-24P

Stock fans (Nidec UltraFlo W40S12BMA5-52):
Idle: ~40.6 dBa
Boot: ~56.7 dBa

Delta FFB0412HN-5T20:
(In room with rack)
Idle: ~silent (NB I couldn't even hear the fans, so too quiet to register above server fans with my fake db meter?)
Boot: ~47.9 dBa

(In much quieter room)
Idle: ~40.8 dBa (even though this measures about the same as Nidec, they sound much less 'loud', so I think it's just my phone app's issue or I was incorrectly not the same distance away.)
Boot: ~49.2 dBa

(NB background noise taken out with this calculator)

In the quieter room I agree with @adman_c's mention that the harmonics might not be great to listen to all day. If your room already has other fans going the noise is non-existent.
Thanks for the great recommendation, I had grabbed the last pair off of amazon.com -- even with shipping to Canada it was well worth it.

 

[koifish59]

I'm looking to pick up this ICX 6610 switch but a few posts above says they had issues with vLAN and MTU working properly? This has me worried.

I plan to hook up a ESXi host with vSAN, vMotion, and LAN traffic running through a single 40GbE port on different vLANs on a mellanox connectX-4 NIC, and trunking it to the 40GbE port on this switch. Will this be of any issues?

 

[squadfer]

New member saying Hi!

I wanted to first thank @fohdeesha for sharing his knowledge of these switches and putting this thread and accompanying guides/documents together.

I've been reading through this thread for the past month as I am in the process of upgrading from a cisco SG300-28p switch which is my core switch. Up until I stumbled upon this thread I was leaning towards going the Ubiquiti route though I was hesitant as I wanted the switch to be layer3 and handle the vlan ACL's instead of going up to the router. I have three Supermicro servers and a desktop that have dual 10gbE ports and figured it was time to utilize them. Only 1 server will be utilizing both ports while the others will be just 1 for now as I am aware that my storage layer won't be able to saturate these links.

One of my main concerns with my systems is noise. My rack is not located in a spot that is closed/separate from the rest of my house. It is the opposite, located in the middle of an open floor/hallway. Think game room setup that has three walls for the room next to stairs. With that, I have made all attempts to keep my systems quiet. Currently, the loudest component of my setup is the exhaust fan located in the attic but piped to my rack that pulls the hot air out of the back which is producing 51db a foot from the front of my rack. As much as I would of loved to get the 6610 for future expansions to 10gb, I just couldn't accept the noise level without some serious alterations to the fan setup, which @dodgy route did an excellent execution on with his take (here).

So, I have acquired a 7250-48p as 8 ports of 10gb will hold me over for the foreseeable future. Upon delivery I plugged the switch in to get a reference to how loud the stock fans are. Prior to doing this I was aware that boot process with the fans at full tilt would be loud. But, even after the fans spun down to idle it was still too loud for my taste. I utilized the fans that @RoachedCoach mentioned on post 3,179 (here). The way I have mine wired up is (header on pcb) FanA is controlling both the Sunon fan on the ASIC heatsink via the 12v and ground wire and also one of the mechatronics fans with the tach wire hooked up to the mechatronics fan. Fan B/C are going to the other mechatronics fan respectively. Again, the boot process with fans at full tilt is louder than I would like but it is only temporary so its bearable. When the fans spin back down to idle I can not hear them from my rack so this is working perfectly for me.

I have not been able to get to the CLI of this device yet but that is due to me waiting on the shipment of a mini-usb to rj45 Brocade console cable. Once that arrives I'll be able to begin getting this device setup as my core switch.

 

[psc]

I've found a 6610 for what seems like reasonable money, but since it's the 48-port and I really only need 24 (and don't need the extra power consumption...) I'm only really interested if it's Rev B or C PSUs. The vendor has responded that the power supplies are "MREV 04" but I'm not sure what they tells me... To avoid going back and asking the same question again, and potentially still not buying, does anyone know how that translates?

 

[richrichgreen]

I've found a 6610 for what seems like reasonable money, but since it's the 48-port and I really only need 24 (and don't need the extra power consumption...) I'm only really interested if it's Rev B or C PSUs. The vendor has responded that the power supplies are "MREV 04" but I'm not sure what they tells me... To avoid going back and asking the same question again, and potentially still not buying, does anyone know how that translates?

Here is a picture of the REV A and B psu that I have. The REV B is labeled "MREV 04" and the A is "MREV 03". Its still possible that the MREV isn't directly tied to the revision (Could be batch or inspection date or manufacture year or just about anything) You should still ask just to be safe

 

[hmw]

I've found a 6610 for what seems like reasonable money, but since it's the 48-port and I really only need 24 (and don't need the extra power consumption...) I'm only really interested if it's Rev B or C PSUs. The vendor has responded that the power supplies are "MREV 04" but I'm not sure what they tells me... To avoid going back and asking the same question again, and potentially still not buying, does anyone know how that translates?

I don't think you can rely 100% on the MREV number (although I have three Rev B's and one Rev C and all the Rev B's are MREV 04)

Why take a risk? Ask the seller to send a pic of the PSU or the show chassis output

(edit) - found an eBay listing where a Rev A PSU is MREV 08 (Brocade 1000W Power Supply RPS16-E 23-0000142-02 Delta AWF-2DC-1000W-E 881119179654 | eBay), so perhaps the MREV doesn't increment linearly with the REV ?