Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[jasonwc]

This didn't seem to be the case.
With either PSU and Fan in either of the left or right slots, the system would refuse to lower the fan speeds after a full boot.
For shits and giggles, I put the B PSU and fan in the left slots, and the A PSU in the right.
The system has yet to ramp it's fans up.

So I guess that’s my bad thinking I could get away with only 1 PSU.
I’ll keep an eye out for a Rev B PSU to match with my other.

Thanks!

That’s odd. When you say the switch refused to lower the fan speeds, do you mean it audibly didn’t lower, or that “show chassis” continued to show the fan in high mode?

I have an ICX6610-48P with dual C power supplies. It definitely ramps down with either one or two power supplies and dual fans. However, with dual PSUs, the fans clearly run at a lower RPM and are noticeably quieter (49dB versus 55-57dB). Running with a single PSU and dual fans also runs cooler, presumably due to the higher fan RPM.

I also tested with a single PSU and single fan in the left slots. It ramped down as well but it was too hot for my preference. The second fan really drops the temperatures. I’m also using two 2.5W 40GBase-LR4 Lite optics and seven mostly 10G-LR optics, so the extra airflow helps.

 

[fohdeesha]

This didn't seem to be the case.
With either PSU and Fan in either of the left or right slots, the system would refuse to lower the fan speeds after a full boot.
For shits and giggles, I put the B PSU and fan in the left slots, and the A PSU in the right.
The system has yet to ramp it's fans up.

So I guess that’s my bad thinking I could get away with only 1 PSU.
I’ll keep an eye out for a Rev B PSU to match with my other.

Thanks!

pretty sure it just runs the fan annoyingly higher on the 24F version, as it presumably has 24x power hungry SFP transceivers to cool

 

[Chewza]

pretty sure it just runs the fan annoyingly higher on the 24F version, as it presumably has 24x power hungry SFP transceivers to cool

sorry, my bad, it's not the 24F, it's just the straight 24 (8x SFP+ in the front with 24x 1gbe RJ45, and the stacking module)

 

[fohdeesha]

sorry, my bad, it's not the 24F, it's just the straight 24 (8x SFP+ in the front with 24x 1gbe RJ45, and the stacking module)

correction: pretty sure I'm wrong and have no idea

 

[Chewza]

correction: pretty sure I'm wrong and have no idea

Regardless of whether you may or may not be wrong haha, I'm happy with the noise from the B+A PSU's for the time being.
Again, appreciate everything you've done for us with the documentation and testing!

 

[covfefe]

How did you guys install an ASIC fan in the ICX7250? I can't find good mounting points. Did you make holes in the top case, and fastened it there?

 

[Freebsd1976]

How did you guys install an ASIC fan in the ICX7250? I can't find good mounting points. Did you make holes in the top case, and fastened it there?

no need make holes , for mounting low speed 6010 or 6015 fan , tapping Screw is enough ,

 

[cap]

How did you guys install an ASIC fan in the ICX7250? I can't find good mounting points. Did you make holes in the top case, and fastened it there?

I used a dab of hot-melt glue on each corner of the fan. Hope I can remove the fan in case I ever need to, but I don't really know.

 

[rocketpanda40]

How did you guys install an ASIC fan in the ICX7250? I can't find good mounting points. Did you make holes in the top case, and fastened it there?

Well to add another idea, I just used twist ties to secure it to the Poe board above it which held it loosely in place. Didn't want to make any permanent modifications and it still worked excellently.

This was mine: https://forums.servethehome.com/ind...be-40gbe-switching.21107/page-255#post-296191

 

[covfefe]

I used a dab of hot-melt glue on each corner of the fan. Hope I can remove the fan in case I ever need to, but I don't really know.

hot glue to the top case, I’m assuming.
Lots of good ideas, thanks guys. I’m leaning towards the twist ties to the ASIC radiator. It sounds easier to avoid ripping the cable out when removing the top case later on

 

[SuperMiguel]

PFSense handles all routing with the exception of traffic between two VLANs with 10G clients. The IoT VLAN (VLAN 2) that contains my Phillips Hue hub is routed by PFSense. It's accessed primarily by my trusted WPA3-Enterprise WLAN network (VLAN 11) and limited WLAN network (VLAN 9). Even with 10G traffic, PFSense had no issues routing traffic at wirespeed using jumbo packets at 10G (~10% CPU utilization). Given that it's easier to manage all the firewall rules in one place in PFSense, I'm planning to move all my 10G and 40G clients to one VLAN for simplicity, as shown below.

PFSense Hardware:
Xeon E5-1220v2 3.1Ghz
8GB DDR3
Intel X520-DA2 with two SFP+ ports
FS QSFP+ to 4x SFP+ breakout cable connects from the X520-DA2 to my ICX6610-48p for WAN and LAN
LAN contains untagged traffic for the trusted LAN and 14 tagged VLANs

How do you control, which routing PFSense handles vs which VLANs traffic gets handled by your switch?

 

[jasonwc]

How do you control, which routing PFSense handles vs which VLANs traffic gets handled by your switch?

PFSense is the default gateway (0.0.0.0/0). Static routes are setup to handle inter-VLAN traffic. Say you have VLAN2 on subnet 192.168.2.0/24 with a VE IP of 192.168.2.250 and VLAN3 on 192.168.3.0/24 with a VE IP of 192.168.3.250. If you want a client on VLAN2 to route traffic via the switch, you setup a static route on the VLAN2 client to route all traffic destined for 192.168.3.0/24 via 192.168.2.250. On the VLAN3 clients, you setup a static route to route all traffic destined for 192.168.2.0/24 via 192.168.3.250. Internet traffic continues to go through the default gateway (pfsense router).

 

[cap]

hot glue to the top case, I’m assuming.
Lots of good ideas, thanks guys. I’m leaning towards the twist ties to the ASIC radiator. It sounds easier to avoid ripping the cable out when removing the top case later on

To clarify, I hot glued the fan to the ASIC heat sink, not to the top case. I imagine that if you glued the fan to the top case, there would be no path for air to pass between the fan and the top case.

 

[SuperMiguel]

PFSense is the default gateway (0.0.0.0/0). Static routes are setup to handle inter-VLAN traffic. Say you have VLAN2 on subnet 192.168.2.0/24 with a VE IP of 192.168.2.250 and VLAN3 on 192.168.3.0/24 with a VE IP of 192.168.3.250. If you want a client on VLAN2 to route traffic via the switch, you setup a static route on the VLAN2 client to route all traffic destined for 192.168.3.0/24 via 192.168.2.250. On the VLAN3 clients, you setup a static route to route all traffic destined for 192.168.2.0/24 via 192.168.3.250. Internet traffic continues to go through the default gateway (pfsense router).

Mind sharing your switch config? Love to see how all this is configured. Thanks

 

[RoachedCoach]

hot glue to the top case, I’m assuming.
Lots of good ideas, thanks guys. I’m leaning towards the twist ties to the ASIC radiator. It sounds easier to avoid ripping the cable out when removing the top case later on

Make sure the fan is attached to the ASIC heat sink, not the case.

I used screws and just screwed right into the fins of the heat sink. Been great for almost 2 years now.

 

[Wolfcastle]

I finally had some time to try to get my 6610 up and running... I updated the main FW without issue but trying to get the POE board updated yields the following (see log below). It just stays stuck in this loop forever. I found another thread about this but in his case, the POE board had a blown IC on it, which is not the case for my board. I took a quick look and everything seems ok as far as I can tell. I reseated the connectors between the POE board and the main board and it did not improve.

ICX6610-24P Router#inline power install-firmware stack-unit 1 tftp 192.168.1.32 ICX6610-FCX/fcx_poeplus_02.1.0.b004.fw
ICX6610-24P Router#Flash Memory Write (8192 bytes per dot) .........
 tftp download successful file name = poe-fw
Sending PoE Firmware to Unit 1.
Firmware Update failed
PoE Info: Resetting module in slot 1....completed.
PoE Error: Device 0 failed to start on PoE module.
PoE Error: Device 1 failed to start on PoE module.
Resetting module in slot 1 again to recover from dev fault
PoE Info: Hard Resetting in slot 1....
PoE Info: Resetting module in slot 1....completed.
PoE Error: Device 0 failed to start on PoE module.
PoE Error: Device 1 failed to start on PoE module.
Resetting module in slot 1 again to recover from dev fault
PoE Info: Hard Resetting in slot 1....

Interestingly I initially tried to connect my laptop to RJ45 ports 1 couldn't connect (no status light), tried port 2 the same, finally I tried port 8 which works. I've only checked quickly but ports 1-4 appear totally dead. This feels like a hardware issue, I will see if I can find any damage near ports 1-4. Does anyone have any idea to what device 0 and device 1 correspond, maybe this will give me some hint as to the issue?

 

[Wolfcastle]

Good news/bad news if anyone is interested, I took another look at the board today and I found the following:

• Looks like possibly some sort of liquid damage near the power connector on the POE board
• R27 and R201 are 0 ohm resistors but appeared burnt, measured as open so they are definitely toast
• F45, F20 measure open (they're fuses so should measure 0)
• F51 near PoE controller U6 measures open
• Confirmed that ports 1-4 are dead, I'm not sure I want to take the board off the chassis and investigate if there is any damage in there. I think I'll pass

Aside from this, my unit has 2x Rev A power supplies and 2x Fan modules. This switch is way louder than I was hoping even with no POE load, I can hear it through the ceiling of my mechanical room into the living room above it (my house has admittedly little/no sound insulation).

 

[dswartz]

Not to be a wise guy, but isn't a 0 ohm resistor... a piece of wire?

 

[NablaSquaredG]

Not to be a wise guy, but isn't a 0 ohm resistor... a piece of wire?

Well, kind of.

A 0ohm resistor can be used to let traces cross (basically a bridge where one trace goes through two others which are connected by the 0ohm resistor)

 

[kpfleming]

Yes, it's a piece of wire, except it can be picked and placed by an SMT machine putting the rest of the parts on the board