Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)
- Thread starter fohdeesha
- Start date
Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
I'm running the updates on my 6610 and while running
it will transfer for a bit (5mins-30mins) and then I get
Is there something I'm missing ?
Code:
copy tftp flash 192.168.1.110 FCXR08030t.bin primary
Code:
TFTP: received error request -- code 0Thanks. I got him to check again, and they're Rev B. Photo shows two fans but I think only 1 PSU, and he's confirmed they're PoE rear exhaust with the rails included, so I've just taken one for £150; I suspect I could have gone lower, given how quickly he accepted. There are another 34 available at the moment...
Brocade ICX 6610-48P ICX6610-48P-E 48 Ports Network Switch | eBay
Brocade ICX 6610-48P ICX6610-48P-E 48 Ports Network Switch, we have 35 units. dusty and could do with a clean, they will have marks and scratches from use. The front rails/ears are present but rails have been cut shorter than they originally would have been.
www.ebay.co.uk
Does anyone have a working icx7750?
I snag one on ebay but it doesn't come with fans or power supply so I'm on the hunt for these parts.
About the power supply - the part number is RPS9+E (rear exhaust) or RPS9+I (side-port)
Will the RPS9 Net Iron power supply works the same?
Also, compared to the icx6610 - the fan headers are missing... did the fan header broke off?
@fohdeesha
I snag one on ebay but it doesn't come with fans or power supply so I'm on the hunt for these parts.
About the power supply - the part number is RPS9+E (rear exhaust) or RPS9+I (side-port)
Will the RPS9 Net Iron power supply works the same?
Also, compared to the icx6610 - the fan headers are missing... did the fan header broke off?
@fohdeesha
make sure you can ping your tftp server, just run "ping 192.168.1.110" from the same bootloader prompt. if you're on windows I would turn off the windows firewall, it always gets in the way of tftp
your switch is fine, the 7750 doesn't use big fan connectors like you're used to seeing, it uses the PCB, think of a PCI connector. that big blank looking tab in the middle has contacts on the bottom of it. as for the PSU, I have no clue. If it's the same exact model number and formfactor, then yes it should work
I cant make anything larger than 1500 MTU to work, I gave up, sadly
Perhaps I have something wrong, but this seems that just doesn't work.
did you run the "ip mtu 9000" command on the ve assigned to that vlan? should work fine. if you're trying to do it just on a per port basis I believe you'll need to run the layer2 only firmwareI cant make anything larger than 1500 MTU to work, I gave up, sadly
If you're lucky, you might find a set of fans for the 7750 for $600-$650 (for 4); I often see them listed at $150/ea. I passed up several opportunities for a cheap 7750 because when I added up the cost of finding PSUs and fans, it often equaled or exceeded just getting it with everything from a different seller. At this point, I've about given up hope of seeing a -48F for a reasonable price and am contemplating a -26Q and doing passive breakouts to a fiber patch, or 40G MPO SR or LR to a splitter for the breakout. Only reason I'm eyeing the 7750 over a 6610 is that I want to use the campus fabric feature and turn my 7150's into port extenders and have a single management interface.
double checking this on mine, just running "jumbo" globally on the layer 3 routing firmware and rebooting, ports are now set to 10200 and I can pass jumbo frames between them. if you're using VEs for routing, running "ip mtu" on the VE only affects layer3 traffic getting routed through that VE (eg the layer2 ports themselves will still have an MTU of 10200 or whatever). was there a specific issue or use case you were trying to do that didn't work?I cant make anything larger than 1500 MTU to work, I gave up, sadly
All I did was run 'jubmo and reload on layer 3 routing fw too, all ports are now set to 10200. I dont have any VE, I just tried with a vlan with 2 ports, connecting two MTU9000 esxi vkernel vswitches. I can't ping from esxi (vmkping) with large frames to other one. If I connect them directly with a cable it works.
Last edited:
strange, what frame size are you trying? After setting jumbo and reloading I can pass 9000byte frames
I have MTU set to 9000 on both esxi, interestingly when connected with direct cable, I get 840Mb/s NFS disk transfer speed, with MTU 1500 it drops to 340Mb/s or so, therefore I want to use this. I will go and recheck again...
btw, is there a quick way to set all of the rest interfaces to MTU1500, or do I have to do it one by one?
btw, is there a quick way to set all of the rest interfaces to MTU1500, or do I have to do it one by one?
You cannot set per-interface MTU on anything that is not VRF capable, period. The documentation is just wrong about that.
For passing VLANs where frames can hit 1536, you need to set both jumbo and aggregated-vlan, reload (each requires a reload but can be combined into one reload.) And that's really it. If it's still not working, the distant end may be attempting PMTUD, which Brocade does wrong.
I haven't got enough space to set up a test here, but based on other test results and my vSphere knowledge, there's a couple possibilities. To confirm you'll need to deploy a dvSwitch, set the MTU to 9216 (NOT 9000, 9216,) set multicast to IGMP/MLD snoop, enable LLDP in bidirectional, and enable the VLAN and MTU health check. Use a tagged or dual-mode port with a tagged VLAN. If jumbo frames are working correctly in LLDP negotiation mode, VLAN and MTU check should pass at 9216 as the MRU is <10200.
First post here - love this community already and lots of helpful posts on modding for less noise. Have to run a small lab for VMware certs out of my home office and replacing a stack of 3750s with a 7250-48p and 8x licensed 10Gb ports. Already that is quieter, but not enough for me to be happy yet. I've read all of the fan mod threads multiple times and essentially it seems that there's a comedy of issues with the fan mods leading to people using wood screws to wedge fans on top of the ASIC heatsink to cool it down after putting in slower fans - otherwise the system might ramp up to speed 2 indefinitely or might switch between speed 1 and speed 2 relentlessly since slower fans move less air and cause the ASIC to go to speed 2. Top this with the fact that it sounds like the board itself requires a minimum RPM to boot and you can either put in fans that meet this RPM or you can spoof the signal - which is also quite tedious.
Annoyed slightly by this issue, I started thinking about re-applying thermal paste - because I didn't quite understand how my switch could idle at 58C on the ASIC with 68-70F ambient temps when I literally only had a serial cable plugged into it and nothing else. I think the thing deterring people from doing this is that it's very difficult to do without damaging the switch. The heat sink is mounted with push pins that typically require you to have access to the other side of the board to get them out. Once the pins are in, they're not coming out without some serious persuasion.
So I tried to take the circuit board out only to find that it's literally riveted to the switch chassis and practically impossible. What followed was almost 60 minutes of me trying to get these things out without damaging the circuit board. I think I may have been successful overall. I had to remove everything down to the board to work in the space. Small warning- the fins arekind of edgy - I sliced my finger tips twice getting this out proceeded by dousing them in alcohol when I was cleaning the junk off the heatsink/asic.
And what I found is in my opinion utter stupidity. If you've applied thermal compound to a heat sink before, you know that the best practice is a very thin layer of compound to aid the transmission of heat. Too thick and you lose effectiveness. CPUs call for a compound the size of a grain of rice. What I found left me dumbfounded that this is how this equipment was designed. Perhaps I'm missing some of the finer details of enterprise switching...?
Pre-cleaning
Post Cleaning
Board after the fact - you can see a little scratching around the holes, but that should be fine given they're no circuitry there.
So I'm in a spot where I'd like to put a better heatsink on here - preferably one with a small fan integrated. The problem is that the heatsink literally doesn't have a part number and I cannot find any reference for the size / compatibility. I measure it to be approx 65mmx70mm 10mm fins and 2 push-pin mounts. There's nothing out there that fits the bill ... everything on digi and a few other sites are mostly square.
So I might just have to reapply paste and mount the heatsink. I think I have to replace the push-pins as I kind of mangled them getting them out - not really an issue as they're cheap I believe 3MM, and I can put slightly stronger springs on them this time around if I have to go that direction to improve contact with the ASIC.
Anyone know where I can get specs on this heatsink? I imagine re-applying good paste to the heatsink (like MX4 thermal compound or ceramique) could go a long ways here to improving temperatures given the previous situations.
Annoyed slightly by this issue, I started thinking about re-applying thermal paste - because I didn't quite understand how my switch could idle at 58C on the ASIC with 68-70F ambient temps when I literally only had a serial cable plugged into it and nothing else. I think the thing deterring people from doing this is that it's very difficult to do without damaging the switch. The heat sink is mounted with push pins that typically require you to have access to the other side of the board to get them out. Once the pins are in, they're not coming out without some serious persuasion.
So I tried to take the circuit board out only to find that it's literally riveted to the switch chassis and practically impossible. What followed was almost 60 minutes of me trying to get these things out without damaging the circuit board. I think I may have been successful overall. I had to remove everything down to the board to work in the space. Small warning- the fins arekind of edgy - I sliced my finger tips twice getting this out proceeded by dousing them in alcohol when I was cleaning the junk off the heatsink/asic.
And what I found is in my opinion utter stupidity. If you've applied thermal compound to a heat sink before, you know that the best practice is a very thin layer of compound to aid the transmission of heat. Too thick and you lose effectiveness. CPUs call for a compound the size of a grain of rice. What I found left me dumbfounded that this is how this equipment was designed. Perhaps I'm missing some of the finer details of enterprise switching...?
Pre-cleaning
Post Cleaning
Board after the fact - you can see a little scratching around the holes, but that should be fine given they're no circuitry there.
So I'm in a spot where I'd like to put a better heatsink on here - preferably one with a small fan integrated. The problem is that the heatsink literally doesn't have a part number and I cannot find any reference for the size / compatibility. I measure it to be approx 65mmx70mm 10mm fins and 2 push-pin mounts. There's nothing out there that fits the bill ... everything on digi and a few other sites are mostly square.
So I might just have to reapply paste and mount the heatsink. I think I have to replace the push-pins as I kind of mangled them getting them out - not really an issue as they're cheap I believe 3MM, and I can put slightly stronger springs on them this time around if I have to go that direction to improve contact with the ASIC.
Anyone know where I can get specs on this heatsink? I imagine re-applying good paste to the heatsink (like MX4 thermal compound or ceramique) could go a long ways here to improving temperatures given the previous situations.
I moved the L3 routing duties from my untangle vm to the ICX6450. It was a lot easier than expected, but just wanted to post what I did in one post since I ended up taking info from different posts here & externally to make one complete post. now my internet gw is just a firewall and not a does everything box at long last!
1. Move your DHCP outside of your firewall. I've been meaning to do this for such a long time but finally have.
It's convenient, it works, you know it all. But the more I can make my firewall just a dumb box that does the one function it's supposed to (you know.. connecting to the internet) the better. You also avoid issues with the crap firewall vendors stick you with, easing configuration when you get the itch to try the next greatest firewall.
For some different reasons you should avoid using the FastIron DHCP server as well.
My poison of choice is dnsmasq for zero effort DHCP+DNS integration, but you can do the exact same thing with ISC, Kea, Windows Server, whatever.
Your dnsmasq or equivalent configuration should look similar to this:
VLAN1 will work automatically and use the right range. (really, I think it just uses the range with the same subnet as your DHCP host.)
At this point everything should be working as it was before.
2. Setup VLAN11 on the switch first. This will be the validation test that your config is OK.
N.B. I already had the tagged interfaces setup.... if you don't and are going to be using both VLAN 1 (untagged) and tagged VLANs on the same port, connect via serial and enable dual-mode on vlan 1 for those ports. Otherwise you will stop getting VLAN 1.
Using dual-mode is really the same as combining trunk+access (Cisco). You can trunk X vlans with an untagged native VLAN.
In my situation, 1/2/1 to 1/2/4 are connected to my proxmox hosts. They speak vlan1 but have trunks for a few tagged VLANs. This requires dual-mode.
At this point you should have a working VLAN 11 with DHCP. Try it. Everything else should be working as it is.
3. Change the internet gw to another IP. I changed mine from 10.10.1.1/24 to 10.10.1.4/24.
This is optional if you want to juggle around different IPs but I kept it consistent... .1- ICX6450, .4- Untangle.
Switch to the serial console at this point unless you live by the seat of your pants because you will probably lose connectivity unless you don't change the IP address.
4. Patch up VLAN 1 on the switch and fix the default routes.
Your addresses will of course be different.
5. Lastly, add a static route on the firewall for vlan11 (and any new future VLANs) to point at your switch. This will allow vlan11 to access the internet.
For mine, I created the route 10.11.1.0/24 via 10.10.1.1.
Yes, could just do the whole /8, but it prevents internet connectivity for new VLANs until I link it up which is a nice side effect.
At this point everything should work and your old firewall should no longer be doing inter-VLAN routing. Test DHCP, routing (between subnets and to the internet), etc.
(10.10.1.1 is now my switch, 10.10.1.4 (the firewall) is no longer involved!)
(full duplex working! this is inside random containers, so the speed is what it is)
1. Move your DHCP outside of your firewall. I've been meaning to do this for such a long time but finally have.
It's convenient, it works, you know it all. But the more I can make my firewall just a dumb box that does the one function it's supposed to (you know.. connecting to the internet) the better. You also avoid issues with the crap firewall vendors stick you with, easing configuration when you get the itch to try the next greatest firewall.
For some different reasons you should avoid using the FastIron DHCP server as well.
My poison of choice is dnsmasq for zero effort DHCP+DNS integration, but you can do the exact same thing with ISC, Kea, Windows Server, whatever.
Your dnsmasq or equivalent configuration should look similar to this:
Code:
# Untagged VLAN DHCP
# The tag names (home,vlan11) are purely visual for you.
# The dhcp range/options used will be based on whether the ICX switch relayed a DHCP request or not
# (in which case it embeds the source VLAN's subnet! that is where the magic happens)
dhcp-range=home,10.10.1.100,10.10.1.240,255.255.255.0,12h
dhcp-option=home,option:router,10.10.1.1
dhcp-option=home,option:dns-server,10.10.1.8
dhcp-option=home,option:domain-search,BLAH.ca
# A single VLAN (11)
# Note you do NOT 'configure' the VLAN, you just configure the DHCP range and it will sort itself out
dhcp-range=vlan11,10.11.1.100,10.11.1.240,255.255.255.0,12h
dhcp-option=vlan11,option:router,10.11.1.1
dhcp-option=vlan11,option:dns-server,10.10.1.8
dhcp-option=vlan11,option:domain-search,BLAH.ca- router: the default route/next-hop. I changed addressing so that the ICX6450 is .1 and my internet gw is .4 just to keep things consistent. Do as you do. In either situation, the router option should be set to the IP of the ICX device in the same subnet.
- dns server: your dns server. For me this is my dhcp server as well. Yours may be different.
- search domain: optional, only if you need it do you know you need it.
VLAN1 will work automatically and use the right range. (really, I think it just uses the range with the same subnet as your DHCP host.)
At this point everything should be working as it was before.
2. Setup VLAN11 on the switch first. This will be the validation test that your config is OK.
N.B. I already had the tagged interfaces setup.... if you don't and are going to be using both VLAN 1 (untagged) and tagged VLANs on the same port, connect via serial and enable dual-mode on vlan 1 for those ports. Otherwise you will stop getting VLAN 1.
Using dual-mode is really the same as combining trunk+access (Cisco). You can trunk X vlans with an untagged native VLAN.
In my situation, 1/2/1 to 1/2/4 are connected to my proxmox hosts. They speak vlan1 but have trunks for a few tagged VLANs. This requires dual-mode.
Code:
# If you have NOT configured any VLANs before and need to enable dual-mode
# (because you want to send untagged vlan1 or another vlan as WELL as tagged traffic)
# you need to configure each interface with dual-mode, e.g.:
interface eth 1/2/4
dual-mode
# done
# Create VLAN 11 and tag interfaces
vlan 11 vlantest
tagged ethernet 1/2/1 to 1/2/4
router-interface ve 11
# Assign ve11 as the default route
interface ve 11
ip address 10.11.1.1/24
ip helper-address 1 10.10.1.8- ip helper-address: This is the DHCP relay option. The dest is your DHCP server (any VLAN, so long as the switch can access it). The VE's address is used by your DHCP server to pick the same subnet.
- ip address: the IP address the switch will take in the subnet (& mask). This is what you will use for next-hop/default route on devices
At this point you should have a working VLAN 11 with DHCP. Try it. Everything else should be working as it is.
3. Change the internet gw to another IP. I changed mine from 10.10.1.1/24 to 10.10.1.4/24.
This is optional if you want to juggle around different IPs but I kept it consistent... .1- ICX6450, .4- Untangle.
Switch to the serial console at this point unless you live by the seat of your pants because you will probably lose connectivity unless you don't change the IP address.
4. Patch up VLAN 1 on the switch and fix the default routes.
Your addresses will of course be different.
Code:
# Change ve1's IP and default route
interface ve 1
no ip address 10.10.1.19/24
ip address 10.10.1.1/24
# (You need this if your DHCP server is on a different VLAN)
# (If it's the same VLAN, don't sweat it and you can omit this.)
ip helper-address 1 10.10.1.8
# Change the default route to the new firewall
exit
no ip route 0.0.0.0/0 10.10.1.1
ip route 0.0.0.0/0 10.10.1.4For mine, I created the route 10.11.1.0/24 via 10.10.1.1.
Yes, could just do the whole /8, but it prevents internet connectivity for new VLANs until I link it up which is a nice side effect.
At this point everything should work and your old firewall should no longer be doing inter-VLAN routing. Test DHCP, routing (between subnets and to the internet), etc.
Code:
$ traceroute 10.11.1.223
traceroute to 10.11.1.223 (10.11.1.223), 64 hops max
1 10.10.1.1 1.764ms 1.757ms 0.762ms
2 10.11.1.223 1.025ms 0.222ms 0.119ms
Code:
# ./iperf3 -c 10.10.1.12 --bidir
Connecting to host 10.10.1.12, port 5201
[ 6] local 10.11.1.223 port 51624 connected to 10.10.1.12 port 5201
[ 8] local 10.11.1.223 port 51626 connected to 10.10.1.12 port 5201
[ ID][Role] Interval Transfer Bitrate Retr Cwnd
[ 6][TX-C] 0.00-1.00 sec 1.01 GBytes 8.67 Gbits/sec 301 413 KBytes
[ 8][RX-C] 0.00-1.00 sec 1001 MBytes 8.39 Gbits/sec
[ 6][TX-C] 1.00-2.00 sec 954 MBytes 8.01 Gbits/sec 0 721 KBytes
[ 8][RX-C] 1.00-2.00 sec 962 MBytes 8.07 Gbits/sec
[ 6][TX-C] 2.00-3.00 sec 1.05 GBytes 9.02 Gbits/sec 26 758 KBytes
[ 8][RX-C] 2.00-3.00 sec 1.04 GBytes 8.93 Gbits/sec
[...]
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID][Role] Interval Transfer Bitrate Retr
[ 6][TX-C] 0.00-10.00 sec 10.1 GBytes 8.64 Gbits/sec 1026 sender
[ 6][TX-C] 0.00-10.00 sec 10.1 GBytes 8.63 Gbits/sec receiver
[ 8][RX-C] 0.00-10.00 sec 10.1 GBytes 8.67 Gbits/sec 0 sender
[ 8][RX-C] 0.00-10.00 sec 10.1 GBytes 8.67 Gbits/sec receiver
Last edited:
I'll try using the RPS9 power supply for broacade net-iron - the connector appears to be the same but can't seems to find affordable fans refurbished or new anywhere...
I was hoping the 6610 fans would work but not they don't
hmm, I've removed the boards without much issue, nothing should be riveted - you have to unscrew and remove the silver standoffs, they're holding the board down
Interesting - maybe I just need more torque - They just seemed quite resistant and when I looked at the bottom of the chassis, there were rivets beneath each and the soldering around it on the board gave me pause. So you're saying that these pegs can be removed?
absolutely - what's riveted to the chassis is the nut those things are screwing into - they're the same as motherboard standoffs, just longer. the solder around all the circles is so the PCB makes good electrical contact with the standoffs (and therefore chassis ground), it's not actually bonded to the standoff