Drag to reposition cover

Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

kousuke

New Member
Jul 18, 2017
19
0
1
37
If you're lucky, you might find a set of fans for the 7750 for $600-$650 (for 4); I often see them listed at $150/ea. I passed up several opportunities for a cheap 7750 because when I added up the cost of finding PSUs and fans, it often equaled or exceeded just getting it with everything from a different seller. At this point, I've about given up hope of seeing a -48F for a reasonable price and am contemplating a -26Q and doing passive breakouts to a fiber patch, or 40G MPO SR or LR to a splitter for the breakout. Only reason I'm eyeing the 7750 over a 6610 is that I want to use the campus fabric feature and turn my 7150's into port extenders and have a single management interface.
I'll try using the RPS9 power supply for broacade net-iron - the connector appears to be the same but can't seems to find affordable fans refurbished or new anywhere...

I was hoping the 6610 fans would work but not they don't
 

fohdeesha

Kaini Industries
Nov 20, 2016
1,994
1,812
113
29
fohdeesha.com
First post here - love this community already and lots of helpful posts on modding for less noise. Have to run a small lab for VMware certs out of my home office and replacing a stack of 3750s with a 7250-48p and 8x licensed 10Gb ports. Already that is quieter, but not enough for me to be happy yet. I've read all of the fan mod threads multiple times and essentially it seems that there's a comedy of issues with the fan mods leading to people using wood screws to wedge fans on top of the ASIC heatsink to cool it down after putting in slower fans - otherwise the system might ramp up to speed 2 indefinitely or might switch between speed 1 and speed 2 relentlessly since slower fans move less air and cause the ASIC to go to speed 2. Top this with the fact that it sounds like the board itself requires a minimum RPM to boot and you can either put in fans that meet this RPM or you can spoof the signal - which is also quite tedious.

Annoyed slightly by this issue, I started thinking about re-applying thermal paste - because I didn't quite understand how my switch could idle at 58C on the ASIC with 68-70F ambient temps when I literally only had a serial cable plugged into it and nothing else. I think the thing deterring people from doing this is that it's very difficult to do without damaging the switch. The heat sink is mounted with push pins that typically require you to have access to the other side of the board to get them out. Once the pins are in, they're not coming out without some serious persuasion.

So I tried to take the circuit board out only to find that it's literally riveted to the switch chassis and practically impossible. What followed was almost 60 minutes of me trying to get these things out without damaging the circuit board. I think I may have been successful overall. I had to remove everything down to the board to work in the space. Small warning- the fins arekind of edgy - I sliced my finger tips twice getting this out proceeded by dousing them in alcohol when I was cleaning the junk off the heatsink/asic.

View attachment 15993


And what I found is in my opinion utter stupidity. If you've applied thermal compound to a heat sink before, you know that the best practice is a very thin layer of compound to aid the transmission of heat. Too thick and you lose effectiveness. CPUs call for a compound the size of a grain of rice. What I found left me dumbfounded that this is how this equipment was designed. Perhaps I'm missing some of the finer details of enterprise switching...?

Pre-cleaning
View attachment 15994

Post Cleaning
View attachment 15995


Board after the fact - you can see a little scratching around the holes, but that should be fine given they're no circuitry there.
View attachment 15996

So I'm in a spot where I'd like to put a better heatsink on here - preferably one with a small fan integrated. The problem is that the heatsink literally doesn't have a part number and I cannot find any reference for the size / compatibility. I measure it to be approx 65mmx70mm 10mm fins and 2 push-pin mounts. There's nothing out there that fits the bill ... everything on digi and a few other sites are mostly square.

So I might just have to reapply paste and mount the heatsink. I think I have to replace the push-pins as I kind of mangled them getting them out - not really an issue as they're cheap I believe 3MM, and I can put slightly stronger springs on them this time around if I have to go that direction to improve contact with the ASIC.

Anyone know where I can get specs on this heatsink? I imagine re-applying good paste to the heatsink (like MX4 thermal compound or ceramique) could go a long ways here to improving temperatures given the previous situations.
hmm, I've removed the boards without much issue, nothing should be riveted - you have to unscrew and remove the silver standoffs, they're holding the board down
 

DMFDMinister

New Member
Oct 4, 2020
3
2
3
hmm, I've removed the boards without much issue, nothing should be riveted - you have to unscrew and remove the silver standoffs, they're holding the board down
Interesting - maybe I just need more torque - They just seemed quite resistant and when I looked at the bottom of the chassis, there were rivets beneath each and the soldering around it on the board gave me pause. So you're saying that these pegs can be removed?
Screenshot_1.jpg
 

fohdeesha

Kaini Industries
Nov 20, 2016
1,994
1,812
113
29
fohdeesha.com
absolutely - what's riveted to the chassis is the nut those things are screwing into - they're the same as motherboard standoffs, just longer. the solder around all the circles is so the PCB makes good electrical contact with the standoffs (and therefore chassis ground), it's not actually bonded to the standoff
 

DMFDMinister

New Member
Oct 4, 2020
3
2
3
absolutely - what's riveted to the chassis is the nut those things are screwing into - they're the same as motherboard standoffs, just longer. the solder around all the circles is so the PCB makes good electrical contact with the standoffs (and therefore chassis ground), it's not actually bonded to the standoff
Well that's helpful - and leaves me feeling a bit sheepish. Now just a matter of figuring out how I'm going to mod this. I'm curious if better compound with slightly stronger push-pin mount with 9-10K RPM fans will keep the fans operating in speed 1 while still satisfying the tach requirements. Still going to do some digging on the heatsink and if there's something better I can install.
 
  • Love
Reactions: virulent

rootwyrm

Member
Mar 25, 2017
43
48
18
www.rootwyrm.com
Annoyed slightly by this issue, I started thinking about re-applying thermal paste - because I didn't quite understand how my switch could idle at 58C on the ASIC with 68-70F ambient temps when I literally only had a serial cable plugged into it and nothing else. I think the thing deterring people from doing this is that it's very difficult to do without damaging the switch. The heat sink is mounted with push pins that typically require you to have access to the other side of the board to get them out. Once the pins are in, they're not coming out without some serious persuasion.
Because it's designed to operate at those temperatures. 24x7x365. For years. These are not consumer junk. The max die on these processors is 105C+, all of the capacitors are 85C rated or above, and it is designed to run at these temperatures and higher without failing. They get thrown into the top of a rack with terrible airflow design at best, with 30U+ of heat source directly below, and next to no cold air supply. And that's the least severe duty they do.
And believe me, I would know. I've been engineering the systems that rack below these for decades. Brocade's one of the few I'm not aware of having successfully cooked out when located in the same rack as one of my heat exchanger equipped systems. Even when basically starved of cold air supply and sitting in a rack with more than 8kW draw, it was fine.

Anyone know where I can get specs on this heatsink? I imagine re-applying good paste to the heatsink (like MX4 thermal compound or ceramique) could go a long ways here to improving temperatures given the previous situations.
That's not thermal paste. That's a perfectly good reusable thermal pad, which you ruined. They get used in this sort of application because they do not wear out or dry out. (No, that was not dried out. It was perfectly fine till you separated it from the woven layer.) And they're more tolerant of severely unlevel IHS and CPU mounting. That IHS gets slapped on there with some glue to protect the die and give them somewhere for a pretty part label, not be a critical heat transfer element.
And no, you can't just buy replacement. The particular compound they used there is familiar to me, and only sold in bulk roll. And no, you can't just slap some AS5 or GC12 on and 'fix' it; the woven compound is impregnated.
Thermal compound is not going to improve performance meaningfully at all. These are very, very low TDP parts. All the heat load is in the POE board. You could strap an Alpha cast copper slug on there and it wouldn't net you anything significant. Which is why they're just fine with a cheap, loose coupled, cast aluminum part that isn't even in direct airflow but go to pains to route over the POE board. So basically anything that matches the pin span (which is measured center to center with a caliper) and fits within the keepout zone indicated by the outer white box will work fine.
No, really. Whatever. Doesn't matter. It'll be fine. Effectively coupling will be a whole other story, but, it won't break anything.
 

pokeimon

New Member
Oct 13, 2019
5
1
3
USA
I'm a relatively new to this and I have run up against a wall of what to do next.
I'm trying to configure the switch as layer 3 while routing outgoing traffic to OPNsense (More or less PFsense).

Port 1/2/1 which is connected to OPNsense with a Gateway created with the IP set to 10.0.2.1 and a Route set to that gateway with Network set to 10.0.10.0/24 to encompass the VLAN 10 network. OPNsense's IP is 10.0.2.2.

I did try to follow a guide though its for a CISCO router but I'm stuck.
I am currently just trying to get VLAN 10 to connect to OPNsense/Internet. Currently I created VLAN 2 to be used as the Transit VLAN and VLAN 10 to be a 'trusted' network.

Any help would be much appreciated.

For the switch I have the following config:
Code:
ICX6450-48P Router(config)#show run
Current configuration:
!
ver 08.0.30tT313
!
stack unit 1
  module 1 icx6450-48p-poe-port-management-module
  module 2 icx6450-sfp-plus-4port-40g-module
!
!
!
!
vlan 1 name DEFAULT-VLAN by port
!
vlan 2 name Transit by port
 untagged ethe 1/1/1 ethe 1/2/1
 router-interface ve 2
!
vlan 10 name Trusted by port
 untagged ethe 1/1/3
 router-interface ve 10
!
!
!
!
!
aaa authentication web-server default local
aaa authentication enable default local
aaa authentication login default local
ip dhcp-client disable
ip dhcp-server enable
!
ip dhcp-server pool pool10
 dhcp-default-router 10.0.10.1
 dns-server 1.1.1.1
 excluded-address 10.0.10.1
 excluded-address 10.0.10.2 10.0.10.99
 lease 1 0 0
 network 10.0.10.0 255.255.255.0
 deploy
!
ip route 0.0.0.0/0 10.0.2.2
!
no telnet server
username root password .....
!
!
!
!
!
interface ve 2
 ip address 10.0.2.1 255.255.255.0
!
interface ve 10
 ip address 10.0.10.1 255.255.255.0
!
!
!
!
!
!
!
!
!
end
 

BobTB

Member
Jul 19, 2019
36
4
8
did you run the "ip mtu 9000" command on the ve assigned to that vlan? should work fine. if you're trying to do it just on a per port basis I believe you'll need to run the layer2 only firmware
Ok, I think I found out what is wrong. This is bad. Very bad. I am using mikrotik sfp+ ports into rj45 10gbE copper adapters ( S+RJ10 )

These do not support jumbo frames? Check the forum posts below:


This is the only thing that could be wrong with my setup, I tried everything to make it work, and it does not :( work

Anyone else using these sucesfully with jumbo frames?
 
Last edited:

BobTB

Member
Jul 19, 2019
36
4
8
ok, there are two versions rev.1 and rev.2 of Mikrotik S+RJ10. The R2 supports jumbo, R1 does not. I don't even need to go to look on mine which rev. they are....
 
  • Like
Reactions: fohdeesha

virulent

New Member
Jul 3, 2019
17
24
3
I'm a relatively new to this and I have run up against a wall of what to do next.
I'm trying to configure the switch as layer 3 while routing outgoing traffic to OPNsense (More or less PFsense).

Port 1/2/1 which is connected to OPNsense with a Gateway created with the IP set to 10.0.2.1 and a Route set to that gateway with Network set to 10.0.10.0/24 to encompass the VLAN 10 network. OPNsense's IP is 10.0.2.2.

I did try to follow a guide though its for a CISCO router but I'm stuck.
I am currently just trying to get VLAN 10 to connect to OPNsense/Internet. Currently I created VLAN 2 to be used as the Transit VLAN and VLAN 10 to be a 'trusted' network.

Any help would be much appreciated.

For the switch I have the following config:
Code:
ICX6450-48P Router(config)#show run
Current configuration:
!
ver 08.0.30tT313
!
stack unit 1
  module 1 icx6450-48p-poe-port-management-module
  module 2 icx6450-sfp-plus-4port-40g-module
!
!
!
!
vlan 1 name DEFAULT-VLAN by port
!
vlan 2 name Transit by port
untagged ethe 1/1/1 ethe 1/2/1
router-interface ve 2
!
vlan 10 name Trusted by port
untagged ethe 1/1/3
router-interface ve 10
!
!
!
!
!
aaa authentication web-server default local
aaa authentication enable default local
aaa authentication login default local
ip dhcp-client disable
ip dhcp-server enable
!
ip dhcp-server pool pool10
dhcp-default-router 10.0.10.1
dns-server 1.1.1.1
excluded-address 10.0.10.1
excluded-address 10.0.10.2 10.0.10.99
lease 1 0 0
network 10.0.10.0 255.255.255.0
deploy
!
ip route 0.0.0.0/0 10.0.2.2
!
no telnet server
username root password .....
!
!
!
!
!
interface ve 2
ip address 10.0.2.1 255.255.255.0
!
interface ve 10
ip address 10.0.10.1 255.255.255.0
!
!
!
!
!
!
!
!
!
end
your switch config looks ok (I think?)
from a device on vlan10 (not opnsense) can you ping the switch and opnsense? 10.0.10.1 (will definitely work), 10.0.2.1 (should), 10.0.2.2 (should)

if you can ping 10.0.2.2 then you have an issue on OPNsense, so check your NAT rules that they are OK. Also check the live firewall view for any blocked packets from 10.0.10.0/24.

edit: tied to above, does it work with a device on vlan2?
 

infoMatt

Active Member
Apr 16, 2019
181
76
28
ip helper-address: this is your DHCP server. The IP is the native host IP in its native VLAN! This is essentially the DHCP relay.
Good writing, but just to clarify... the switch itself in this case is the DHCP Relay, the helper address points to the server that receives the packet sourcing from the VE IP address on behalf of the client asking for an address. ;) The server can choose the right subnet/pool based of the source address (ie. the VE one).
 
  • Like
Reactions: virulent

pokeimon

New Member
Oct 13, 2019
5
1
3
USA
your switch config looks ok (I think?)
from a device on vlan10 (not opnsense) can you ping the switch and opnsense? 10.0.10.1 (will definitely work), 10.0.2.1 (should), 10.0.2.2 (should)

if you can ping 10.0.2.2 then you have an issue on OPNsense, so check your NAT rules that they are OK. Also check the live firewall view for any blocked packets from 10.0.10.0/24.

edit: tied to above, does it work with a device on vlan2?
Thanks for the info!
It was my NAT that was incorrectly set up (or I should say forgotten to set up)
 
  • Like
Reactions: virulent

koifish59

Member
Sep 30, 2020
33
9
8
I just received a ICX 6610!

I don't have a console cable yet. While I wait for one, if I do a factory reset, will the switch be ready to use on my network as a plug-and-play unmanaged switch?
 

virulent

New Member
Jul 3, 2019
17
24
3
I just received a ICX 6610!

I don't have a console cable yet. While I wait for one, if I do a factory reset, will the switch be ready to use on my network as a plug-and-play unmanaged switch?
More or less, except your switch won't have a management IP (or does it default to DHCP on the mgmt port with telnet enabled? ...no idea). It will only route VLAN 1 but will otherwise act as unmanaged. PoE (if equipped) is off by default so if needed you will need to wait for a console cable.
 
  • Like
Reactions: koifish59