Drag to reposition cover

Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

rootwyrm

Member
Mar 25, 2017
44
48
18
www.rootwyrm.com
and now I can finally close this thread
Nope. Because I'm working on a clip-on/bolt-on to mount double 120mm fans to the side-flows.
You're on your own for the electronics, unless I can remember how to trick the ports into providing power. Then you can use two of these ramped down to about 32-36V. (They're what I found that's easy to order off Amazon.)
Remember our airflow target is ~90CFM, but if you don't provide for the entire side, you either cook the PSU or you cook the POE board. Definitely better fans out there for it. But shortening the ramp to wedge ~100CFM from 242x121mm surface into those side vents at a <4" total thickness comes with VERY high static pressure demand (which means: loud.) And 48VDC is generally only used for "noise no longer matters" applications.

Long ramp 48VDC, probably would go SanAce 9G1248M401 (but MOQ is 60) for 59CFM @ 29dbA/ea.
Short ramp 48VDC, definitely need more thinking time.
Running off 12V, anything 50CFM or above would be fine, but you have a STRICT limit of 0.45A @ 12V off the board headers. (5.4W!)
I'd much prefer 24VDC, because then you can use Mechatronics G1225S24B-FSR's. 54CFM @ 26dBA and $11/ea. Just an excellent impeller design on those. 1500RPM with a pretty flat static pressure curve and 54CFM? Sign me the hell up.
 

dodgy route

Member
Aug 12, 2020
9
26
13
Australia
Nope. Because I'm working on a clip-on/bolt-on to mount double 120mm fans to the side-flows.
You're on your own for the electronics, unless I can remember how to trick the ports into providing power. Then you can use two of these ramped down to about 32-36V. (They're what I found that's easy to order off Amazon.)
Remember our airflow target is ~90CFM, but if you don't provide for the entire side, you either cook the PSU or you cook the POE board. Definitely better fans out there for it. But shortening the ramp to wedge ~100CFM from 242x121mm surface into those side vents at a <4" total thickness comes with VERY high static pressure demand (which means: loud.) And 48VDC is generally only used for "noise no longer matters" applications.

Long ramp 48VDC, probably would go SanAce 9G1248M401 (but MOQ is 60) for 59CFM @ 29dbA/ea.
Short ramp 48VDC, definitely need more thinking time.
Running off 12V, anything 50CFM or above would be fine, but you have a STRICT limit of 0.45A @ 12V off the board headers. (5.4W!)
I'd much prefer 24VDC, because then you can use Mechatronics G1225S24B-FSR's. 54CFM @ 26dBA and $11/ea. Just an excellent impeller design on those. 1500RPM with a pretty flat static pressure curve and 54CFM? Sign me the hell up.
All the power to different setups, whatever works is the best solution.
If you prefer some 24V fans, chuck both of them in series, they will run fine.

*EDIT* Just realised you are talking about some PoE port on the board itself.
If you are talking about the 6610, the electronics parts are detailed relatively well in my post above for the pulse generator for all 3 fans.
 

rootwyrm

Member
Mar 25, 2017
44
48
18
www.rootwyrm.com
All the power to different setups, whatever works is the best solution.
If you prefer some 24V fans, chuck both of them in series, they will run fine.

*EDIT* Just realised you are talking about some PoE port on the board itself.
If you are talking about the 6610, the electronics parts are detailed relatively well in my post above for the pulse generator for all 3 fans.
Correct, though more accurately, taking PoE off a port on the switch. So you'd just run a cable along the side over to port 1 and off to the races you go.

Also, I just found the loveliest bug in 08.0.30u...

Code:
icx6450(config)# sho run
...
lag ds1511 dynamic id 1
 ports ethernet 1/1/35 to 1/1/36
 primary-port 1/1/35
 deploy
 port-name "DS1511.eth0" ethernet 1/1/35
 port-name DS1511.eth1 ethernet 1/1/36
...
icx6450(config)#lag ds1511 dynamic id 1
icx6450(config-lag-ds1511)#no port-name "DS1511.eth0" ethernet 1/1/35
Port name doesn't match
icx6450(config-lag-ds1511)#no port-name DS1511.eth0 ethernet 1/1/35
Port name doesn't match
The only way to fix it? Write the config to tftp or scp (don't - you have to make your system insecure as hell because 08.0.30 is that obsolete and dangerous,) manually edit, and reload the whole thing.
 

dodgy route

Member
Aug 12, 2020
9
26
13
Australia
Correct, though more accurately, taking PoE off a port on the switch. So you'd just run a cable along the side over to port 1 and off to the races you go.
There is PoE to DC converters to run Pi's and such, worth a look into those :) but obviously something with suitable voltage.
I would be surprised if there isnt a variable DC-DC power supply to PoE adapter
 
Last edited:

BobTB

Member
Jul 19, 2019
37
4
8
Dont ban me :) I turned "jumbo" on my icx 6610 . Reload. Now all my ports show MTU 10200. Everything still works. Now I have a few VLANs, nothing fancy. No routing. One is for connecting my NAS for NFS traffic. Four ports only. Nothing larger than 1500 or so passes through. What am I missing? My devices are on MTU9000 on both sides, (ESXI with vswitch set to MTU9000 + vmknic set to MTU9000). What should I set on the switch. I tried setting an interface to "ip mtu 1500" just for fun. Nothing happens. Should the swithc be rebooted everytime?

Can I set all the ports to MTU1500 somehow in one go, and then these four ports to larger MTU? As far as I can see it can only be done port by port. Is this really so?
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,001
1,821
113
29
fohdeesha.com
Correct, though more accurately, taking PoE off a port on the switch. So you'd just run a cable along the side over to port 1 and off to the races you go.

Also, I just found the loveliest bug in 08.0.30u...

Code:
icx6450(config)# sho run
...
lag ds1511 dynamic id 1
ports ethernet 1/1/35 to 1/1/36
primary-port 1/1/35
deploy
port-name "DS1511.eth0" ethernet 1/1/35
port-name DS1511.eth1 ethernet 1/1/36
...
icx6450(config)#lag ds1511 dynamic id 1
icx6450(config-lag-ds1511)#no port-name "DS1511.eth0" ethernet 1/1/35
Port name doesn't match
icx6450(config-lag-ds1511)#no port-name DS1511.eth0 ethernet 1/1/35
Port name doesn't match
The only way to fix it? Write the config to tftp or scp (don't - you have to make your system insecure as hell because 08.0.30 is that obsolete and dangerous,) manually edit, and reload the whole thing.
how'd you even get it to save the quotation marks into the config, did you manually edit it? It strips them out of the name before saving them to the config, I can't get them in there even if I try. works perfectly fine here:


Code:
testicle(config-lag-ds1511)#
testicle(config-lag-ds1511)#port-name "DS1511.eth0" ethernet 1/1/20
testicle(config-lag-ds1511)#port-name DS1511.eth1 ethernet 1/1/21

testicle(config-lag-ds1511)#exit
testicle(config)#sh run | inc port-name
 port-name DS1511.eth0 ethernet 1/1/20
 port-name DS1511.eth1 ethernet 1/1/21


testicle(config)#lag ds1511 dynamic id 1
testicle(config-lag-ds1511)#no port-name DS1511.eth0 ethernet 1/1/20
testicle(config-lag-ds1511)#no port-name DS1511.eth1 ethernet 1/1/21
testicle(config-lag-ds1511)#sh run | inc port-name
testicle(config-lag-ds1511)#
don't - you have to make your system insecure as hell because 08.0.30 is that obsolete and dangerous
SO DANGEROUS BRO
 

ICXGURU

Member
Jun 22, 2020
36
15
8
how'd you even get it to save the quotation marks into the config, did you manually edit it? It strips them out of the name before saving them to the config, I can't get them in there even if I try. works perfectly fine here:


Code:
testicle(config-lag-ds1511)#
testicle(config-lag-ds1511)#port-name "DS1511.eth0" ethernet 1/1/20
testicle(config-lag-ds1511)#port-name DS1511.eth1 ethernet 1/1/21

testicle(config-lag-ds1511)#exit
testicle(config)#sh run | inc port-name
port-name DS1511.eth0 ethernet 1/1/20
port-name DS1511.eth1 ethernet 1/1/21


testicle(config)#lag ds1511 dynamic id 1
testicle(config-lag-ds1511)#no port-name DS1511.eth0 ethernet 1/1/20
testicle(config-lag-ds1511)#no port-name DS1511.eth1 ethernet 1/1/21
testicle(config-lag-ds1511)#sh run | inc port-name
testicle(config-lag-ds1511)#


SO DANGEROUS BRO
Why would you use a quotes when there is no space in the name? The system will remove them anyway as fohdeesha shows above.
 

ICXGURU

Member
Jun 22, 2020
36
15
8
Dont ban me :) I turned "jumbo" on my icx 6610 . Reload. Now all my ports show MTU 10200. Everything still works. Now I have a few VLANs, nothing fancy. No routing. One is for connecting my NAS for NFS traffic. Four ports only. Nothing larger than 1500 or so passes through. What am I missing? My devices are on MTU9000 on both sides, (ESXI with vswitch set to MTU9000 + vmknic set to MTU9000). What should I set on the switch. I tried setting an interface to "ip mtu 1500" just for fun. Nothing happens. Should the swithc be rebooted everytime?

Can I set all the ports to MTU1500 somehow in one go, and then these four ports to larger MTU? As far as I can see it can only be done port by port. Is this really so?
You do not need to reboot. You will see the MTU change in the show interface e x/x/x when you make the changes.
Yes, you need to do it per interface (or with a script)
Try "mtu-exceed forward" globally to fragment packets if needed.
 

hmw

Active Member
Apr 29, 2019
260
83
28
That is interesting about the Wiitek linking at 2.5 and 5Gig. What did the 7250 say the link speed was in the show interface? I would think it reported 10gig even if the other end was 2.5 or 5 since those ASICs are not multigig.
The Wiitek is based on the Marvell 88x3310 and is indeed multi-gig, however other than the Aquantia ACQ-107, all the other Marvell based transceivers (Ipolex for example) I have tested, cause problems when linking with anything other than 10G (using an ICX-6610)

I've tried traffic shaping and rate limiting the ICX side ports without success (https://forums.servethehome.com/ind...t-marvell-88x3300-v-s-aquantia-aqs-107.30004/)

It would be awesome if you know & can share how to successfully rate limit and get full nbase-t performance on these ICX switches ...
 

BobTB

Member
Jul 19, 2019
37
4
8
You do not need to reboot. You will see the MTU change in the show interface e x/x/x when you make the changes.
Yes, you need to do it per interface (or with a script)
Try "mtu-exceed forward" globally to fragment packets if needed.
I do not see the MTU change if I do the show interface. It just stays at MTU 10200, I can set it to whatever I want and it will still not change. Any setting that can prevent it to change?
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,001
1,821
113
29
fohdeesha.com
I do not see the MTU change if I do the show interface. It just stays at MTU 10200, I can set it to whatever I want and it will still not change. Any setting that can prevent it to change?
are you changing the mtu per interface as well or just using the global command?
 

ICXGURU

Member
Jun 22, 2020
36
15
8
The Wiitek is based on the Marvell 88x3310 and is indeed multi-gig, however other than the Aquantia ACQ-107, all the other Marvell based transceivers (Ipolex for example) I have tested, cause problems when linking with anything other than 10G (using an ICX-6610)

I've tried traffic shaping and rate limiting the ICX side ports without success (https://forums.servethehome.com/ind...t-marvell-88x3300-v-s-aquantia-aqs-107.30004/)

It would be awesome if you know & can share how to successfully rate limit and get full nbase-t performance on these ICX switches ...
The ICX does not have a multi-gig ASIC, is was designed long before multi-gig existed so it does not know how to negotiate to 2.5 or 5.

I do not have any 6xxx series to test with but Rate-limiting is inbound only so that is not going to help you. You would need to output rate shape like so:
Core(config-if-e1000-1/1/2)# rate-limit output shaping 100000
Outbound Rate Shaping on Port 1/1/2 Config: 100000 Kbps, Actual: 100000 Kbps
 

BobTB

Member
Jul 19, 2019
37
4
8
are you changing the mtu per interface as well or just using the global command?
I used the global command 'jumbo' to enable it, and then tried to lower individual interfaces one by one, but it does not change the MTU down at all, it just stays at the max of 10200
 

rootwyrm

Member
Mar 25, 2017
44
48
18
www.rootwyrm.com
how'd you even get it to save the quotation marks into the config, did you manually edit it? It strips them out of the name before saving them to the config, I can't get them in there even if I try. works perfectly fine here:
Your guess there is as good as mine, but, I confirmed it in copy tftp output and actually on the switch. Conclusion? Has to be a bug. But damned if I could tell you how the hell to trigger it. May be from the 08.0.30t to 30u upgrade itself. I did find a second workaround: delete the whole LAG, then 'no port-name' each individual port, then reload. (Boy, that sounds like something fun to do in production, doesn't it?) It also breaks LAG port-names after it, ordered by the dynamic id.

Also found what I think may be the other 30u bug people were spooked about; IPv6 multicast behavior is just not right. Even when I took dvSwitches out of the equation, the 6450 is just not learning IPv6 IGMP correctly. Which results in double-master and a lot of fabric flooding. So if you're doing anything like CARP or VRRP, I would say stay on 30t. raguard also seems flaky, but that may be purely related to the IGMP in my architecture.
Thankfully I'm moving all of that to OSPFv3 (if I can figure out why the hell frr is sending out the wrong interfaces) so ripping it out was a non-issue. If I was a paying customer? I'd probably be demanding some serious trade-in credit on a 7450 for the outage.

I do not see the MTU change if I do the show interface. It just stays at MTU 10200, I can set it to whatever I want and it will still not change. Any setting that can prevent it to change?
This is one of the most brain-dead aspects of the ICX family, and I mean it is just bad. All of the documentation will tell you "ip mtu 1536" - nope. This is a lie and only works on VRFs and only when you have an IP address assigned. This is NOT what the documentation says. The documentation says "all models can do this." They cannot. You only have three MTU options for non-VRF stuff, period.
1508 (1500), 10200 (9216), or 1536.
"Wait, 1536?!"
Yeah. Brocade is exceedingly stupid when it comes to frames, trust me. And even worse at documentation half the time. If you're passing tagged traffic from anything sane, you need to set jumbo globally first (this sets a register in the silicon because of course the silicon's that bad about it too.) Then set aggregated-vlan which is also a global. aggregated-vlan flips the software switch to accept 1536(1542) frames. Where this gets truly absurdist is: if you set aggregated-vlan without jumbo, it still frags at 1500. And if you just set jumbo and try to do 1536 frames on a port it's decided is 1500, it just chokes because "you didn't set the MTU appropriately!"

Once I figured that out? Yeah. Now I can pass frames between the Junipers and the ICX6450.
 
  • Like
Reactions: istamov

BobTB

Member
Jul 19, 2019
37
4
8
If you're passing tagged traffic from anything sane, you need to set jumbo globally first (this sets a register in the silicon because of course the silicon's that bad about it too.) Then set aggregated-vlan which is also a global. aggregated-vlan flips the software switch to accept 1536(1542) frames. Where this gets truly absurdist is: if you set aggregated-vlan without jumbo, it still frags at 1500. And if you just set jumbo and try to do 1536 frames on a port it's decided is 1500, it just chokes because "you didn't set the MTU appropriately!"

Once I figured that out? Yeah. Now I can pass frames between the Junipers and the ICX6450.
Seems you got this figured out,. This is how far I am now - I set the jumbo already, and reload (nice to do when in production), now what I want to do is to have it pass 9000 frames - which it just doesn't - on a few interfaces in one VLAN, I dont have any ve IP address assigned. Should I ? Aggregated-vlan?
 
Last edited:

Derelict

New Member
Jun 19, 2020
1
0
1
Chattanooga, Tennessee
I was running Wireshark on Win box first, but then switched to Linux and results were the same. I am pretty sure it is something on the switch, I just don't know if it is expected behavior or not. The reason I am saying this is because:
  1. monitored port is on an isolated empty vlan
  2. mirror port is on isolated vlan with only Wireshark box connected to it
  3. I am not seeing any traffic until I issue command "mirror-port ethernet 1/1/24"
  4. I am seeing traffic from multiple totally unrelated VLANs that have nothing to do with mirror or monitor ports
  5. All traffic that I am seeing is multicast traffic + some TCP re transmits
I was trying to use mirroring for troubleshooting an issue, but I was able to resolve that issue, so this is not really an issue for me anymore. It would still be nice to figure out why it is happening.
I know this is really old but I am seeing exactly the same thing. The 6610 seems to mirror the monitored port traffic plus at least all multicast traffic on all VLANs on the switch. Once I enable the mirror port it starts sending all the multicast traffic even if the only port being monitored is down. And if I disable the mirror, the traffic doesn't stop. I have to reload the switch. This means, until I reload the stack, anything on that switch port receives all the IPv6 RAs, etc, on every VLAN on the switch.

The traffic is definitely not generated by the capture workstation. It is VRRP, CARP, RAs, etc from other VLANs on the switch stack.

I would love to find a solution because as it is, mirror/monitor is about useless to me.

Code level: FCXR08030u.bin
 

virulent

New Member
Jul 3, 2019
17
24
3
Received my ICX6450-24p today -- everything seems to check out!

Judging by the hostname it was in a barn environment.. opened it up and cleaned out that dust..

Before: After:
Fans are surprisingly not destroyed by the dust.. Idle ~40 dBa (basically same as spec sheet's 39 dBa) and boot ~56 dBa.
For me it is effectively silent next to my T320s! No issues with PoE it looks like either.

I do have a pair of Delta FFB0412HN-5T20 I picked up to compare with, but not sure if I will attempt to swap the pins around tonight or some other time. It's already effectively quiet but I'm more concerned about the dust now. Might as well while I wait for licenses :)
 

virulent

New Member
Jul 3, 2019
17
24
3
Okay, swapping fan pins was a lot simpler than I thought (the pins come right out when pushing on the tab with a flathead.)

The acoustics are definitely better. The full speed noise is significantly less than stock. I attempted to measure it, but you can only go so far with a phone app & rough measurements but you get the idea.

ICX6450-24P

Stock fans (Nidec UltraFlo W40S12BMA5-52):
Idle: ~40.6 dBa
Boot: ~56.7 dBa

Delta FFB0412HN-5T20:
(In room with rack)
Idle: ~silent (NB I couldn't even hear the fans, so too quiet to register above server fans with my fake db meter?)
Boot: ~47.9 dBa

(In much quieter room)
Idle: ~40.8 dBa (even though this measures about the same as Nidec, they sound much less 'loud', so I think it's just my phone app's issue or I was incorrectly not the same distance away.)
Boot: ~49.2 dBa

(NB background noise taken out with this calculator)

In the quieter room I agree with @adman_c's mention that the harmonics might not be great to listen to all day. If your room already has other fans going the noise is non-existent.
Thanks for the great recommendation, I had grabbed the last pair off of amazon.com -- even with shipping to Canada it was well worth it.
 

koifish59

Member
Sep 30, 2020
34
9
8
I'm looking to pick up this ICX 6610 switch but a few posts above says they had issues with vLAN and MTU working properly? This has me worried.

I plan to hook up a ESXi host with vSAN, vMotion, and LAN traffic running through a single 40GbE port on different vLANs on a mellanox connectX-4 NIC, and trunking it to the 40GbE port on this switch. Will this be of any issues?