Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

fohdeesha

Kaini Industries
Nov 20, 2016
1,955
1,780
113
29
fohdeesha.com
do you have the LAG tagged in both those vlans on the brocade side? aside from that, you need to figure out why one of the links is only coming up at 100mbps
 

fohdeesha

Kaini Industries
Nov 20, 2016
1,955
1,780
113
29
fohdeesha.com
as a last resort, just reboot both the switch stack and the pfsense box, I've had issues in freebsd before where newly configured LACP interfaces did not come up properly until a reboot - it's a long shot, but probably faster than anything else at this point
 

PGlover

Active Member
Nov 8, 2014
468
55
28
54
do you have the LAG tagged in both those vlans on the brocade side? aside from that, you need to figure out why one of the links is only coming up at 100mbps
Yes.. Vlan 30 and Vlan 2000 is tagged in both vlan on interface 1/1/1 and 2/1/1
 

fohdeesha

Kaini Industries
Nov 20, 2016
1,955
1,780
113
29
fohdeesha.com
I just realized you have two different LACP bonds, to two different pfsense boxes, and it looks like you have issues with both.

In the config you sent me, the LACP bond to pfsense master is port 1/1/1 and 1/2/1

From your show interface output, none of the ports in the first LACP group are up at all:

Code:
Port Link State Dupl Speed Trunk Tag Pvid Pri MAC Name

1/1/1 Down None None None 1 Yes N/A 0 cc4e.24b8.5c4c To pfSense Mast
2/1/1 Down None None None 1 Yes N/A 0 cc4e.24b8.5c4c

Then the LACP bond for the second pfsense box, which is ports 1/1/2 and 2/1/2, one is up at 1G but the other is stuck at 100mbps:

Code:
1/1/2 Up Blocked Full 100M 2 Yes N/A 0 cc4e.24b8.5c4c To pfSense Back
2/1/2 Up Blocked Full 1G 2 Yes N/A 0 cc4e.24b8.5c4c
I would triple check you actually have the proper port pairs going from the switch to the correct pfsense boxes, because something here is screwed up
 

PGlover

Active Member
Nov 8, 2014
468
55
28
54
I just realized you have two different LACP bonds, to two different pfsense boxes, and it looks like you have issues with both.

In the config you sent me, the LACP bond to pfsense master is port 1/1/1 and 1/2/1

From your show interface output, none of the ports in the first LACP group are up at all:

Code:
Port Link State Dupl Speed Trunk Tag Pvid Pri MAC Name

1/1/1 Down None None None 1 Yes N/A 0 cc4e.24b8.5c4c To pfSense Mast
2/1/1 Down None None None 1 Yes N/A 0 cc4e.24b8.5c4c

Then the LACP bond for the second pfsense box, which is ports 1/1/2 and 2/1/2, one is up at 1G but the other is stuck at 100mbps:

Code:
1/1/2 Up Blocked Full 100M 2 Yes N/A 0 cc4e.24b8.5c4c To pfSense Back
2/1/2 Up Blocked Full 1G 2 Yes N/A 0 cc4e.24b8.5c4c
I would triple check you actually have the proper port pairs going from the switch to the correct pfsense boxes, because something here is screwed up
I 'm troubleshooting the pfSense Backup box first. The interfaces are 1/1/2 and 2/1/2. In pfSense the speed and duplex is set to autoselect. On the ICX6610 each interface is set to auto. They should match...
 

PGlover

Active Member
Nov 8, 2014
468
55
28
54
I just realized you have two different LACP bonds, to two different pfsense boxes, and it looks like you have issues with both.

In the config you sent me, the LACP bond to pfsense master is port 1/1/1 and 1/2/1

From your show interface output, none of the ports in the first LACP group are up at all:

Code:
Port Link State Dupl Speed Trunk Tag Pvid Pri MAC Name

1/1/1 Down None None None 1 Yes N/A 0 cc4e.24b8.5c4c To pfSense Mast
2/1/1 Down None None None 1 Yes N/A 0 cc4e.24b8.5c4c

Then the LACP bond for the second pfsense box, which is ports 1/1/2 and 2/1/2, one is up at 1G but the other is stuck at 100mbps:

Code:
1/1/2 Up Blocked Full 100M 2 Yes N/A 0 cc4e.24b8.5c4c To pfSense Back
2/1/2 Up Blocked Full 1G 2 Yes N/A 0 cc4e.24b8.5c4c
I would triple check you actually have the proper port pairs going from the switch to the correct pfsense boxes, because something here is screwed up
There is something going on with the LAG setup on the pfSense_Backup lag.. For some reason, 1 port is registering as 1gbps and the other is registering as 100mbps in the auto speed-duplex setting... This is weird... Maybe I need to recreate the pfSense_Backup lag...
 

PGlover

Active Member
Nov 8, 2014
468
55
28
54
You would not believe this, but I had a bad CAT6 cable... Working now...

=== LAG "pfSense_Backup" ID 2 (dynamic Deployed) ===
LAG Configuration:
Ports: e 1/1/2 e 2/1/2
Port Count: 2
Primary Port: 1/1/2
Trunk Type: hash-based
LACP Key: 20002
Deployment: HW Trunk ID 3
Port Link State Dupl Speed Trunk Tag Pvid Pri MAC Name
1/1/2 Up Forward Full 1G 2 Yes N/A 0 cc4e.24b8.5c4c To pfSense Back
2/1/2 Up Forward Full 1G 2 Yes N/A 0 cc4e.24b8.5c4c
 

PGlover

Active Member
Nov 8, 2014
468
55
28
54
=== LAG "pfSense_Master" ID 1 (dynamic Deployed) ===
LAG Configuration:
Ports: e 1/1/1 e 2/1/1
Port Count: 2
Primary Port: 1/1/1
Trunk Type: hash-based
LACP Key: 20001
Deployment: HW Trunk ID 4
Port Link State Dupl Speed Trunk Tag Pvid Pri MAC Name
1/1/1 Up Forward Full 1G 1 Yes N/A 0 cc4e.24b8.5c4c To pfSense Mast
2/1/1 Up Forward Full 1G 1 Yes N/A 0 cc4e.24b8.5c4c
 

arglebargle

H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈
Jul 15, 2018
656
233
43
It's even worse. The fans in the power supplies are controlled by the switch over i2c as well. If you take apart one of the power supplies (and I did), the fan/power led of the PSU connects to a header inside the PSU. If you remove this wire, the PSU won't turn on... :)

If you replace the fan IN the PSU with a slower one, the PSU will turn on, but the switch won't boot...'cause..well Brocade said FU when it came to silencing this switch.

That said, in all fairness, the 6610 is 1U switch with almost o.5tbps of switching bandwidth. That's a lot. The 6610 does get seriously hot, and probably needs these screamers, unless they change the chassis design. After letting the switch run for a few hours, open it up, and you can barely touch the heatsinks for more than a few seconds. They are that hot and that's WITH the screamers. If you even manage to replace the fans with something slower, there's a better than even chance, you'll cook the switch.
Thanks a lot for this, it gives me somewhere to start thinking about how to attack the problem. I'm not shooting for anything unreasonable like a silent icx 6100 -- just a bit of a noise reduction on a 6450 so I could use it near where I sleep without irritating my SO too much (my home office is in the bedroom thanks to SF bay area rents.)

I picked up a second 6450-24 to work on, 40Gb would have been fun to play with but energy costs here are absurd and quieting a 6610 to use 10' from where I sleep sounds like more of a challenge than I want right now.

I've been learning Fusion360 just for designing shrouds/ducts. PM me, I'm interested on discussing some designs.

Like others said, the PSUs have data lines to monitor/check for these things. It's a PITA. Without putting together a PCB with a MCU that spoofs those signals, you won't get anywhere sadly. Like others said too, the thermal of the 6610 is already tight enough with the stock fans. I can't imagine making it work well with slower/less CFM fans.

I just got a bunch of Sunon and Delta units to test if anyone is interested.
Yeah, it sounds like we're thinking about this in similar ways. I haven't done any controls programming in a while but I think an arduino should work for this, no? The last time I did anything like this was two jobs ago using labview. It's a good excuse to play around with programming controls again anyway.

I have a second 6450-24 arriving early next week, I'll start investigating what the signals look like on the fan headers when I have time. Assuming I have some success there I'm thinking shroud and duct one or two quiet oversized blowers (or larger quiet fans with decent CFM) off of the exhaust port and disconnect the stock fans.

Depending on how the switch is reading sensor data from the fans it might be as simple as feeding the fan controller appropriate responses on the sense wire, if it's trickier I might need dummy loads to simulate the fans at different speeds (I'd need to talk to my more experienced with electronics friends here.) That should be enough to keep the switch thinking that the fans are in an OK state and that more or less frees me up to replace existing fans with whatever I want.

I haven't used fusion360 before but I'll check it out and ping you, I usually cheat and ask friends for help with parts like this but I should probably learn some basic modelling so I don't have to bug them all of the time.
 
Last edited:

fohdeesha

Kaini Industries
Nov 20, 2016
1,955
1,780
113
29
fohdeesha.com
if you're talking about the 6450, none of that is required. it doesn't have a hardware watchdog controller like the larger 6610. The 6450 is simple enough I'm pretty sure you can just unplug the built in fan entirely and be fine - it's simple 3 pin header, just like in a computer case fan - with the 3rd pin being the tach output from the fan. It doesn't even read/warn on RPM I'm pretty sure, so you could just stick in a MUCH slower fan. I've operated one overnight with NO fan and it was fine, so even a barely moving fan should be fine. see my post here - https://forums.servethehome.com/ind...-icx6450-icx6610-etc.21107/page-3#post-197040

you can see the fan connector(s) on the far right in the picture here - https://forums.servethehome.com/index.php?threads/brocade-icx6450-icx6610-etc.21107/#post-196450
 
Last edited:
  • Like
Reactions: arglebargle

fohdeesha

Kaini Industries
Nov 20, 2016
1,955
1,780
113
29
fohdeesha.com
Yup, just double checked - unplugged the fan completely in one of my lab 6450's - it boots up perfectly fine, doesn't even give an error in console or the log. The only indication anything is changed, is if you run "show chassis" it says "fan failed" in one of the lines, but it doesn't care and continues running fine. This means you can stick pretty much any fan in here you want (or none at all) and it won't care

It just needs to be a 40mm 12v 3pin (power + tach) fan. The included model is W40S12BMA5 - some info here - [W40S12BMA5-5*]| Nidec Corporation

The non poe models have 1 of them, the PoE models have 2 of the fans. I've had it running with the fan disconnected for a while now and the temp has only slowly crept up, still not even half of warning level - so I'd imagine you could grab a fan with half the CFM of the stock unit and be totally fine
 

PGlover

Active Member
Nov 8, 2014
468
55
28
54
Late night... Got most of the cables moved from my 4 switches to the 2 ICX6610 switches. Will finish up on Saturday night. Thanks for your help tonight.
 
  • Like
Reactions: fohdeesha

fohdeesha

Kaini Industries
Nov 20, 2016
1,955
1,780
113
29
fohdeesha.com
after letting it sit and cook without a fan until the temp reached equilibrium (stopped rising), these are the results:

With the stock 10CFM fan:
Code:
Fan ok, speed (auto): [[1]]<->2
Fan speed switching temperature thresholds:
        1 -> 2 @ 71 deg-C
        1 <- 2 @ 66 deg-C

Sensor B Temperature Readings:
        Current temperature : 39.0 deg-C
Sensor A Temperature Readings:
        Current temperature : 43.5 deg-C
        Warning level.......: 73.0 deg-C
        Shutdown level......: 83.0 deg-C

With the fan unplugged (ZERO COOLING):
Code:
Fan failed
Sensor B Temperature Readings:
        Current temperature : 47.5 deg-C
Sensor A Temperature Readings:
        Current temperature : 51.0 deg-C
        Warning level.......: 73.0 deg-C
        Shutdown level......: 83.0 deg-C

Conclusion: you could stick a barely moving 2CFM fan in the ICX6450 and be perfectly fine. If you note the first output, the code does not even bother speeding up the fan until it reaches 71C, and with no fan it doesn't even get near that

EDIT: some pics of the fan location/headers in the 6450 - Imgur
 
Last edited:

arglebargle

H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈
Jul 15, 2018
656
233
43
Yup, just double checked - unplugged the fan completely in one of my lab 6450's - it boots up perfectly fine, doesn't even give an error in console or the log. The only indication anything is changed, is if you run "show chassis" it says "fan failed" in one of the lines, but it doesn't care and continues running fine. This means you can stick pretty much any fan in here you want (or none at all) and it won't care

It just needs to be a 40mm 12v 3pin (power + tach) fan. The included model is W40S12BMA5 - some info here - [W40S12BMA5-5*]| Nidec Corporation

The non poe models have 1 of them, the PoE models have 2 of the fans. I've had it running with the fan disconnected for a while now and the temp has only slowly crept up, still not even half of warning level - so I'd imagine you could grab a fan with half the CFM of the stock unit and be totally fine
That's certainly easier than attacking the hardware watchdog, thanks for checking!

I guess I'll have to pick up a 6610 at some point, I was looking forward to a week of hacking on this.
 
  • Like
Reactions: fohdeesha

tommybackeast

Active Member
Jun 10, 2018
251
82
28
no special switch commands, it'll work fine

QoS for voice is important when you're on a limited WAN connection and have a lot of other competing traffic, eg business environments where you need to guarantee bandwidth for the phone system. In a home environment with 50mbps of WAN available (or even more these days) QoS'ing 0.001mbps worth of VOIP traffic isn't going to get you a whole lot, even when pegging the connection with netflix instances etc most home routers these days have smart enough buffers/management to not squash a tiny 8kbps voip connection. This becomes more important when you start to have a larger overall phone system (like a PBX) where you need guaranteed bandwidth to support several simultaneous calls

If you do want to implement QoS, you would need to do it where the WAN connection is (eg your router). Since the switch will do full line rate on every port (tens of thousands of times more bandwidth than a VOIP call) there's not much to be gained by assigning different QoS flows/etc on the switch to voip traffic in a home situation
Thank you for your note re VOIP for home usage; and that today, QOS is not needed for one VOIP line.

The last time I had VOIP phones, was a very long time ago, ISP speeds were vastly slower; and QOS settings were needed for VOIP.
 

PGlover

Active Member
Nov 8, 2014
468
55
28
54
Well.. I consolidated 4 switches into 2 Brocade ICX6610 running in a stack configuration. All the configuration from the 4 switches has been consolidated and translated to work on the ICX6610. Thanks for this thread and fohdessha for answering all my questions.

Only thing left is to figure why port 1/2/2 is not active on my breakout cable when the switch is running in a stack configuration.

Thanks again.
 
  • Like
Reactions: fohdeesha

PGlover

Active Member
Nov 8, 2014
468
55
28
54
Well... It looks like my new setup broke my Sonos setup.. In my attempt to setup each Sonos component again, when it tries to connect to my wireless network and get a DHCP address, it is getting a 169.xxx.xxx.xxx address. It should be getting a 192.168.1.x address assigned to the component. I'm not having this issue with other wireless devices on my network. They all are getting a DHCP address.

Here is the DHCP code in my configuration file....

ip dhcp-server enable
!
ip dhcp-server pool 192.168.1.0/24
dhcp-default-router 192.168.1.2
dns-server 75.75.75.75 75.75.76.76 8.8.8.8
excluded-address 192.168.1.1 192.168.1.99
lease 1 0 0
network 192.168.1.0 255.255.255.0
deploy