Drag to reposition cover

Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

linuxsrc

New Member
Oct 1, 2018
24
4
3
Brownsburg, IN
The only thing you need to worry about is the difference between the I and E models. The E models exhaust front to back which is quieter than the I models (back to front). Otherwise, the differences (32/48/56 port) and Advanced license only is relevant to licensing. You can use the guide to enable all features for free. So, you'll be able to utilize all 48 SFP+ ports, both breakout QSFP+ ports (8 more 10G ports), and all four 40G QSFP+ ports, and also use the advanced L3 features. So, any ICX6550 will give you 56 10G ports and four 40G ports.

The other difference you may see is that some switches may not come with two power supplies and two fans. On the ICX6610, there are 3 revisions of the power supply - A, B, and C. The A revision is significantly louder, even when mixed with a B or C. I'm not sure if this applies to the ICX6650. Anyone know?

To understand your statement correctly, the 32/48/56 is the number of aggregate ports or SFP+ ports only?
The following description is what is confusing me:

ICX6650-48-I-ADV 56 Port
The description says 48 in it be but s described as having 56 ports.
Thanks again.


Licensing, I get it now!

Thanks, I had to read better!
 
Last edited:

Chewza

New Member
Dec 31, 2016
5
1
3
39
@fohdeesha have a weird issue with a new to me ICX6610-24f.

As a starting point... It's been fully upgraded per your guide. (Thanks a ton for this btw!)

However I've got a Rev B PSU that refuses to behave.
The system boots, loud fans as one expects.
Loads config, PSU is quiet.
About a minute later I can hear the PSU fan spin up for seemingly no reason at all and just stays there. The switch doesn't have any activity, and no ethernet ports are connected.

Code:
SSH@brocade6610>show chassis
The stack unit 1 chassis info:

Power supply 1 not present
Power supply 2 (AC - Regular) present, status ok
     Model Number:    23-0000144-01
    Serial Number:    0XS     
    Firmware Ver:      B
Power supply 2 Fan Air Flow Direction:  Front to Back

Fan 1 not present
Fan 2 ok, speed (auto): [[1]]<->2

Fan controlled temperature: 57.5 deg-C

Fan speed switching temperature thresholds:
        Speed 1: NM<----->80       deg-C
        Speed 2:       75<-----> 87 deg-C (shutdown)

Fan 2 Air Flow Direction:  Front to Back
MAC 1 Temperature Readings:
    Current temperature : 55.5 deg-C
CPU Temperature Readings:
    Current temperature : 51.5 deg-C
sensor A Temperature Readings:                                   
    Current temperature : 35.0 deg-C
sensor B Temperature Readings:
    Current temperature : 48.5 deg-C
sensor C Temperature Readings:
    Current temperature : 35.5 deg-C
sensor D Temperature Readings:
    Current temperature : 31.0 deg-C
stacking card Temperature Readings:
    Current temperature : 57.5 deg-C
    Warning level.......: 84.0 deg-C
    Shutdown level......: 87.0 deg-C
Nothing from the chassis info seems out of whack, unless I'm missing something.
I have yet to perform testing with a Rev A PSU that was also included with the system, but have shied away from per the guidance that it's considerably louder.
I've also tried seating the PSU in each of the slots, along with moving the fan (fan is definitely NOT the culprit here)

Any suggestions would be greatly appreciated!
 

jasonwc

Member
Dec 31, 2018
40
15
8
@fohdeesha have a weird issue with a new to me ICX6610-24f.

As a starting point... It's been fully upgraded per your guide. (Thanks a ton for this btw!)

However I've got a Rev B PSU that refuses to behave.
The system boots, loud fans as one expects.
Loads config, PSU is quiet.
About a minute later I can hear the PSU fan spin up for seemingly no reason at all and just stays there. The switch doesn't have any activity, and no ethernet ports are connected.

Code:
SSH@brocade6610>show chassis
The stack unit 1 chassis info:

Power supply 1 not present
Power supply 2 (AC - Regular) present, status ok
     Model Number:    23-0000144-01
    Serial Number:    0XS   
    Firmware Ver:      B
Power supply 2 Fan Air Flow Direction:  Front to Back

Fan 1 not present
Fan 2 ok, speed (auto): [[1]]<->2

Fan controlled temperature: 57.5 deg-C

Fan speed switching temperature thresholds:
        Speed 1: NM<----->80       deg-C
        Speed 2:       75<-----> 87 deg-C (shutdown)

Fan 2 Air Flow Direction:  Front to Back
MAC 1 Temperature Readings:
    Current temperature : 55.5 deg-C
CPU Temperature Readings:
    Current temperature : 51.5 deg-C
sensor A Temperature Readings:                                 
    Current temperature : 35.0 deg-C
sensor B Temperature Readings:
    Current temperature : 48.5 deg-C
sensor C Temperature Readings:
    Current temperature : 35.5 deg-C
sensor D Temperature Readings:
    Current temperature : 31.0 deg-C
stacking card Temperature Readings:
    Current temperature : 57.5 deg-C
    Warning level.......: 84.0 deg-C
    Shutdown level......: 87.0 deg-C
Nothing from the chassis info seems out of whack, unless I'm missing something.
I have yet to perform testing with a Rev A PSU that was also included with the system, but have shied away from per the guidance that it's considerably louder.
I've also tried seating the PSU in each of the slots, along with moving the fan (fan is definitely NOT the culprit here)

Any suggestions would be greatly appreciated!
The guide indicates that if you have only one PSU and one fan, to put both in the leftmost position, when the switch is viewed from the rear (qsfp ports). You mentioned trying different positions for the power supply but the fan position will also change the noise level.

The switch will run quietist with two power supplies and two fans, but I found I needed to reload the switch after connecting the second power supply to see the lower noise level. However, this is only true with B/C power supplies. With your A and B combo, you’re best off only using the B.
 

adman_c

Active Member
Feb 14, 2016
145
63
28
Chicago
I recently picked up a couple 7250s and transplanted my aftermarket fans (I haven't been able to find a record of which specific model, but I believe it's one of the Sunon maglev models discussed early in the thread) from my 6450s. My 6450s and 7250s are running the latest firmware on @fohdeesha 's site and don't throw any errors.

To my ear, the 6450s were still borderline too loud with the new fans. When I moved the fans into my 7250s, the temps are quite high and send the fans into the "speed 2" regime every 5 minutes or so. This is with ~20W of POE and 6 10G devices in each switch (half DAC, half RJ45 transceivers).

While I still have both my 6450s and 7250s I may make a separate thread about fans with noise samples since it seems like a frequent topic.
I found that the Sunon maglev fans made a horrible resonance when installed in my 6450, so I bought a bunch of other fans to try. I ended up settling on the first ones I tried after the Sunons. They were quieter than the stock fans and waaaaay less annoying than the Sunons. Check out rootwyrm's fan post of doom for some other suggestions, as the Deltas that I bought are OOS at the link I posted in my earlier post.
 

Chewza

New Member
Dec 31, 2016
5
1
3
39
The guide indicates that if you have only one PSU and one fan, to put both in the leftmost position, when the switch is viewed from the rear (qsfp ports). You mentioned trying different positions for the power supply but the fan position will also change the noise level.

The switch will run quietist with two power supplies and two fans, but I found I needed to reload the switch after connecting the second power supply to see the lower noise level. However, this is only true with B/C power supplies. With your A and B combo, you’re best off only using the B.
This didn't seem to be the case.
With either PSU and Fan in either of the left or right slots, the system would refuse to lower the fan speeds after a full boot.
For shits and giggles, I put the B PSU and fan in the left slots, and the A PSU in the right.
The system has yet to ramp it's fans up.

So I guess that’s my bad thinking I could get away with only 1 PSU.
I’ll keep an eye out for a Rev B PSU to match with my other.

Thanks!
 

jasonwc

Member
Dec 31, 2018
40
15
8
This didn't seem to be the case.
With either PSU and Fan in either of the left or right slots, the system would refuse to lower the fan speeds after a full boot.
For shits and giggles, I put the B PSU and fan in the left slots, and the A PSU in the right.
The system has yet to ramp it's fans up.

So I guess that’s my bad thinking I could get away with only 1 PSU.
I’ll keep an eye out for a Rev B PSU to match with my other.

Thanks!
That’s odd. When you say the switch refused to lower the fan speeds, do you mean it audibly didn’t lower, or that “show chassis” continued to show the fan in high mode?

I have an ICX6610-48P with dual C power supplies. It definitely ramps down with either one or two power supplies and dual fans. However, with dual PSUs, the fans clearly run at a lower RPM and are noticeably quieter (49dB versus 55-57dB). Running with a single PSU and dual fans also runs cooler, presumably due to the higher fan RPM.

I also tested with a single PSU and single fan in the left slots. It ramped down as well but it was too hot for my preference. The second fan really drops the temperatures. I’m also using two 2.5W 40GBase-LR4 Lite optics and seven mostly 10G-LR optics, so the extra airflow helps.
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,525
2,698
113
31
fohdeesha.com
This didn't seem to be the case.
With either PSU and Fan in either of the left or right slots, the system would refuse to lower the fan speeds after a full boot.
For shits and giggles, I put the B PSU and fan in the left slots, and the A PSU in the right.
The system has yet to ramp it's fans up.

So I guess that’s my bad thinking I could get away with only 1 PSU.
I’ll keep an eye out for a Rev B PSU to match with my other.

Thanks!
pretty sure it just runs the fan annoyingly higher on the 24F version, as it presumably has 24x power hungry SFP transceivers to cool
 

Chewza

New Member
Dec 31, 2016
5
1
3
39
pretty sure it just runs the fan annoyingly higher on the 24F version, as it presumably has 24x power hungry SFP transceivers to cool
sorry, my bad, it's not the 24F, it's just the straight 24 (8x SFP+ in the front with 24x 1gbe RJ45, and the stacking module)
 

Chewza

New Member
Dec 31, 2016
5
1
3
39
correction: pretty sure I'm wrong and have no idea
Regardless of whether you may or may not be wrong haha, I'm happy with the noise from the B+A PSU's for the time being.
Again, appreciate everything you've done for us with the documentation and testing!
 
  • Like
Reactions: fohdeesha

covfefe

New Member
Jan 9, 2022
4
0
1
How did you guys install an ASIC fan in the ICX7250? I can't find good mounting points. Did you make holes in the top case, and fastened it there?
 

Freebsd1976

Active Member
Feb 23, 2018
327
55
28
How did you guys install an ASIC fan in the ICX7250? I can't find good mounting points. Did you make holes in the top case, and fastened it there?
no need make holes , for mounting low speed 6010 or 6015 fan , tapping Screw is enough ,
 

cap

New Member
Sep 20, 2021
12
14
3
California
How did you guys install an ASIC fan in the ICX7250? I can't find good mounting points. Did you make holes in the top case, and fastened it there?
I used a dab of hot-melt glue on each corner of the fan. Hope I can remove the fan in case I ever need to, but I don't really know.
 

rocketpanda40

Member
Dec 12, 2019
36
24
8

covfefe

New Member
Jan 9, 2022
4
0
1
I used a dab of hot-melt glue on each corner of the fan. Hope I can remove the fan in case I ever need to, but I don't really know.
hot glue to the top case, I’m assuming.
Lots of good ideas, thanks guys. I’m leaning towards the twist ties to the ASIC radiator. It sounds easier to avoid ripping the cable out when removing the top case later on
 

SuperMiguel

New Member
Jun 17, 2021
12
1
3
PFSense handles all routing with the exception of traffic between two VLANs with 10G clients. The IoT VLAN (VLAN 2) that contains my Phillips Hue hub is routed by PFSense. It's accessed primarily by my trusted WPA3-Enterprise WLAN network (VLAN 11) and limited WLAN network (VLAN 9). Even with 10G traffic, PFSense had no issues routing traffic at wirespeed using jumbo packets at 10G (~10% CPU utilization). Given that it's easier to manage all the firewall rules in one place in PFSense, I'm planning to move all my 10G and 40G clients to one VLAN for simplicity, as shown below.

PFSense Hardware:
Xeon E5-1220v2 3.1Ghz
8GB DDR3
Intel X520-DA2 with two SFP+ ports
FS QSFP+ to 4x SFP+ breakout cable connects from the X520-DA2 to my ICX6610-48p for WAN and LAN
LAN contains untagged traffic for the trusted LAN and 14 tagged VLANs
How do you control, which routing PFSense handles vs which VLANs traffic gets handled by your switch?
 

jasonwc

Member
Dec 31, 2018
40
15
8
How do you control, which routing PFSense handles vs which VLANs traffic gets handled by your switch?
PFSense is the default gateway (0.0.0.0/0). Static routes are setup to handle inter-VLAN traffic. Say you have VLAN2 on subnet 192.168.2.0/24 with a VE IP of 192.168.2.250 and VLAN3 on 192.168.3.0/24 with a VE IP of 192.168.3.250. If you want a client on VLAN2 to route traffic via the switch, you setup a static route on the VLAN2 client to route all traffic destined for 192.168.3.0/24 via 192.168.2.250. On the VLAN3 clients, you setup a static route to route all traffic destined for 192.168.2.0/24 via 192.168.3.250. Internet traffic continues to go through the default gateway (pfsense router).
 

cap

New Member
Sep 20, 2021
12
14
3
California
hot glue to the top case, I’m assuming.
Lots of good ideas, thanks guys. I’m leaning towards the twist ties to the ASIC radiator. It sounds easier to avoid ripping the cable out when removing the top case later on
To clarify, I hot glued the fan to the ASIC heat sink, not to the top case. I imagine that if you glued the fan to the top case, there would be no path for air to pass between the fan and the top case.
 

SuperMiguel

New Member
Jun 17, 2021
12
1
3
PFSense is the default gateway (0.0.0.0/0). Static routes are setup to handle inter-VLAN traffic. Say you have VLAN2 on subnet 192.168.2.0/24 with a VE IP of 192.168.2.250 and VLAN3 on 192.168.3.0/24 with a VE IP of 192.168.3.250. If you want a client on VLAN2 to route traffic via the switch, you setup a static route on the VLAN2 client to route all traffic destined for 192.168.3.0/24 via 192.168.2.250. On the VLAN3 clients, you setup a static route to route all traffic destined for 192.168.2.0/24 via 192.168.3.250. Internet traffic continues to go through the default gateway (pfsense router).
Mind sharing your switch config? Love to see how all this is configured. Thanks
 
  • Like
Reactions: adman_c

RoachedCoach

Member
Feb 4, 2020
29
36
13
hot glue to the top case, I’m assuming.
Lots of good ideas, thanks guys. I’m leaning towards the twist ties to the ASIC radiator. It sounds easier to avoid ripping the cable out when removing the top case later on
Make sure the fan is attached to the ASIC heat sink, not the case.

I used screws and just screwed right into the fins of the heat sink. Been great for almost 2 years now.