Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[squadfer]

New member saying Hi!

I wanted to first thank @fohdeesha for sharing his knowledge of these switches and putting this thread and accompanying guides/documents together.

I've been reading through this thread for the past month as I am in the process of upgrading from a cisco SG300-28p switch which is my core switch. Up until I stumbled upon this thread I was leaning towards going the Ubiquiti route though I was hesitant as I wanted the switch to be layer3 and handle the vlan ACL's instead of going up to the router. I have three Supermicro servers and a desktop that have dual 10gbE ports and figured it was time to utilize them. Only 1 server will be utilizing both ports while the others will be just 1 for now as I am aware that my storage layer won't be able to saturate these links.

One of my main concerns with my systems is noise. My rack is not located in a spot that is closed/separate from the rest of my house. It is the opposite, located in the middle of an open floor/hallway. Think game room setup that has three walls for the room next to stairs. With that, I have made all attempts to keep my systems quiet. Currently, the loudest component of my setup is the exhaust fan located in the attic but piped to my rack that pulls the hot air out of the back which is producing 51db a foot from the front of my rack. As much as I would of loved to get the 6610 for future expansions to 10gb, I just couldn't accept the noise level without some serious alterations to the fan setup, which @dodgy route did an excellent execution on with his take (here).

So, I have acquired a 7250-48p as 8 ports of 10gb will hold me over for the foreseeable future. Upon delivery I plugged the switch in to get a reference to how loud the stock fans are. Prior to doing this I was aware that boot process with the fans at full tilt would be loud. But, even after the fans spun down to idle it was still too loud for my taste. I utilized the fans that @RoachedCoach mentioned on post 3,179 (here). The way I have mine wired up is (header on pcb) FanA is controlling both the Sunon fan on the ASIC heatsink via the 12v and ground wire and also one of the mechatronics fans with the tach wire hooked up to the mechatronics fan. Fan B/C are going to the other mechatronics fan respectively. Again, the boot process with fans at full tilt is louder than I would like but it is only temporary so its bearable. When the fans spin back down to idle I can not hear them from my rack so this is working perfectly for me.

I have not been able to get to the CLI of this device yet but that is due to me waiting on the shipment of a mini-usb to rj45 Brocade console cable. Once that arrives I'll be able to begin getting this device setup as my core switch.

 

[psc]

I've found a 6610 for what seems like reasonable money, but since it's the 48-port and I really only need 24 (and don't need the extra power consumption...) I'm only really interested if it's Rev B or C PSUs. The vendor has responded that the power supplies are "MREV 04" but I'm not sure what they tells me... To avoid going back and asking the same question again, and potentially still not buying, does anyone know how that translates?

 

[richrichgreen]

I've found a 6610 for what seems like reasonable money, but since it's the 48-port and I really only need 24 (and don't need the extra power consumption...) I'm only really interested if it's Rev B or C PSUs. The vendor has responded that the power supplies are "MREV 04" but I'm not sure what they tells me... To avoid going back and asking the same question again, and potentially still not buying, does anyone know how that translates?

Here is a picture of the REV A and B psu that I have. The REV B is labeled "MREV 04" and the A is "MREV 03". Its still possible that the MREV isn't directly tied to the revision (Could be batch or inspection date or manufacture year or just about anything) You should still ask just to be safe

 

[hmw]

I've found a 6610 for what seems like reasonable money, but since it's the 48-port and I really only need 24 (and don't need the extra power consumption...) I'm only really interested if it's Rev B or C PSUs. The vendor has responded that the power supplies are "MREV 04" but I'm not sure what they tells me... To avoid going back and asking the same question again, and potentially still not buying, does anyone know how that translates?

I don't think you can rely 100% on the MREV number (although I have three Rev B's and one Rev C and all the Rev B's are MREV 04)

Why take a risk? Ask the seller to send a pic of the PSU or the show chassis output

(edit) - found an eBay listing where a Rev A PSU is MREV 08 (Brocade 1000W Power Supply RPS16-E 23-0000142-02 Delta AWF-2DC-1000W-E 881119179654 | eBay), so perhaps the MREV doesn't increment linearly with the REV ?

 

[koifish59]

I asked about the two PSUs in a unit from a seller and got this convoluted answer: "One is Revision AC and the other is Rev 01". Whatever that means. I'm going to assume AC means A, and rev 01 also means A. Unless someone else can decipher this?

 

[richrichgreen]

I'm running the updates on my 6610 and while running

copy tftp flash 192.168.1.110 FCXR08030t.bin primary

it will transfer for a bit (5mins-30mins) and then I get

TFTP: received error request -- code 0

Is there something I'm missing ?

 

[psc]

Here is a picture of the REV A and B psu that I have. The REV B is labeled "MREV 04" and the A is "MREV 03". Its still possible that the MREV isn't directly tied to the revision (Could be batch or inspection date or manufacture year or just about anything) You should still ask just to be safe

Thanks. I got him to check again, and they're Rev B. Photo shows two fans but I think only 1 PSU, and he's confirmed they're PoE rear exhaust with the rails included, so I've just taken one for £150; I suspect I could have gone lower, given how quickly he accepted. There are another 34 available at the moment...

Brocade ICX 6610-48P ICX6610-48P-E 48 Ports Network Switch | eBay

Brocade ICX 6610-48P ICX6610-48P-E 48 Ports Network Switch, we have 35 units. dusty and could do with a clean, they will have marks and scratches from use. The front rails/ears are present but rails have been cut shorter than they originally would have been.

www.ebay.co.uk

 

[kousuke]

Does anyone have a working icx7750?

I snag one on ebay but it doesn't come with fans or power supply so I'm on the hunt for these parts.

About the power supply - the part number is RPS9+E (rear exhaust) or RPS9+I (side-port)

Will the RPS9 Net Iron power supply works the same?

Also, compared to the icx6610 - the fan headers are missing... did the fan header broke off?

@fohdeesha

 

[fohdeesha]

I'm running the updates on my 6610 and while running

copy tftp flash 192.168.1.110 FCXR08030t.bin primary

it will transfer for a bit (5mins-30mins) and then I get

TFTP: received error request -- code 0

Is there something I'm missing ?

make sure you can ping your tftp server, just run "ping 192.168.1.110" from the same bootloader prompt. if you're on windows I would turn off the windows firewall, it always gets in the way of tftp

 

[fohdeesha]

Does anyone have a working icx7750?

I snag one on ebay but it doesn't come with fans or power supply so I'm on the hunt for these parts.

About the power supply - the part number is RPS9+E (rear exhaust) or RPS9+I (side-port)

Will the RPS9 Net Iron power supply works the same?

Also, compared to the icx6610 - the fan headers are missing... did the fan header broke off?

@fohdeesha

View attachment 15987

your switch is fine, the 7750 doesn't use big fan connectors like you're used to seeing, it uses the PCB, think of a PCI connector. that big blank looking tab in the middle has contacts on the bottom of it. as for the PSU, I have no clue. If it's the same exact model number and formfactor, then yes it should work

 

[BobTB]

I'm looking to pick up this ICX 6610 switch but a few posts above says they had issues with vLAN and MTU working properly? This has me worried.

I plan to hook up a ESXi host with vSAN, vMotion, and LAN traffic running through a single 40GbE port on different vLANs on a mellanox connectX-4 NIC, and trunking it to the 40GbE port on this switch. Will this be of any issues?

I cant make anything larger than 1500 MTU to work, I gave up, sadly Perhaps I have something wrong, but this seems that just doesn't work.

 

[fohdeesha]

I cant make anything larger than 1500 MTU to work, I gave up, sadly Perhaps I have something wrong, but this seems that just doesn't work.

did you run the "ip mtu 9000" command on the ve assigned to that vlan? should work fine. if you're trying to do it just on a per port basis I believe you'll need to run the layer2 only firmware

 

[LodeRunner]

If you're lucky, you might find a set of fans for the 7750 for $600-$650 (for 4); I often see them listed at $150/ea. I passed up several opportunities for a cheap 7750 because when I added up the cost of finding PSUs and fans, it often equaled or exceeded just getting it with everything from a different seller. At this point, I've about given up hope of seeing a -48F for a reasonable price and am contemplating a -26Q and doing passive breakouts to a fiber patch, or 40G MPO SR or LR to a splitter for the breakout. Only reason I'm eyeing the 7750 over a 6610 is that I want to use the campus fabric feature and turn my 7150's into port extenders and have a single management interface.

 

[fohdeesha]

I cant make anything larger than 1500 MTU to work, I gave up, sadly Perhaps I have something wrong, but this seems that just doesn't work.

double checking this on mine, just running "jumbo" globally on the layer 3 routing firmware and rebooting, ports are now set to 10200 and I can pass jumbo frames between them. if you're using VEs for routing, running "ip mtu" on the VE only affects layer3 traffic getting routed through that VE (eg the layer2 ports themselves will still have an MTU of 10200 or whatever). was there a specific issue or use case you were trying to do that didn't work?

 

[BobTB]

double checking this on mine, just running "jumbo" globally on the layer 3 routing firmware and rebooting, ports are now set to 10200 and I can pass jumbo frames between them.

All I did was run 'jubmo and reload on layer 3 routing fw too, all ports are now set to 10200. I dont have any VE, I just tried with a vlan with 2 ports, connecting two MTU9000 esxi vkernel vswitches. I can't ping from esxi (vmkping) with large frames to other one. If I connect them directly with a cable it works.

 

[fohdeesha]

All I did was run 'jubmo and reload on layer 3 routing fw too, all ports are now set to 10200. I dont have any VE, I just tried with a vlan with 2 ports, connecting two MTU9000 esxi vkernel vswitches. I can't ping from esxi (vmkping) with large frames to other one If I connect them directly with a cable it works.

strange, what frame size are you trying? After setting jumbo and reloading I can pass 9000byte frames

 

[BobTB]

I have MTU set to 9000 on both esxi, interestingly when connected with direct cable, I get 840Mb/s NFS disk transfer speed, with MTU 1500 it drops to 340Mb/s or so, therefore I want to use this. I will go and recheck again...

btw, is there a quick way to set all of the rest interfaces to MTU1500, or do I have to do it one by one?

 

[rootwyrm]

I have MTU set to 9000 on both esxi, interestingly when connected with direct cable, I get 840Mb/s NFS disk transfer speed, with MTU 1500 it drops to 340Mb/s or so, therefore I want to use this. I will go and recheck again...

btw, is there a quick way to set all of the rest interfaces to MTU1500, or do I have to do it one by one?

You cannot set per-interface MTU on anything that is not VRF capable, period. The documentation is just wrong about that.

For passing VLANs where frames can hit 1536, you need to set both jumbo and aggregated-vlan, reload (each requires a reload but can be combined into one reload.) And that's really it. If it's still not working, the distant end may be attempting PMTUD, which Brocade does wrong.

I haven't got enough space to set up a test here, but based on other test results and my vSphere knowledge, there's a couple possibilities. To confirm you'll need to deploy a dvSwitch, set the MTU to 9216 (NOT 9000, 9216,) set multicast to IGMP/MLD snoop, enable LLDP in bidirectional, and enable the VLAN and MTU health check. Use a tagged or dual-mode port with a tagged VLAN. If jumbo frames are working correctly in LLDP negotiation mode, VLAN and MTU check should pass at 9216 as the MRU is <10200.

 

[DMFDMinister]

First post here - love this community already and lots of helpful posts on modding for less noise. Have to run a small lab for VMware certs out of my home office and replacing a stack of 3750s with a 7250-48p and 8x licensed 10Gb ports. Already that is quieter, but not enough for me to be happy yet. I've read all of the fan mod threads multiple times and essentially it seems that there's a comedy of issues with the fan mods leading to people using wood screws to wedge fans on top of the ASIC heatsink to cool it down after putting in slower fans - otherwise the system might ramp up to speed 2 indefinitely or might switch between speed 1 and speed 2 relentlessly since slower fans move less air and cause the ASIC to go to speed 2. Top this with the fact that it sounds like the board itself requires a minimum RPM to boot and you can either put in fans that meet this RPM or you can spoof the signal - which is also quite tedious.

Annoyed slightly by this issue, I started thinking about re-applying thermal paste - because I didn't quite understand how my switch could idle at 58C on the ASIC with 68-70F ambient temps when I literally only had a serial cable plugged into it and nothing else. I think the thing deterring people from doing this is that it's very difficult to do without damaging the switch. The heat sink is mounted with push pins that typically require you to have access to the other side of the board to get them out. Once the pins are in, they're not coming out without some serious persuasion.

So I tried to take the circuit board out only to find that it's literally riveted to the switch chassis and practically impossible. What followed was almost 60 minutes of me trying to get these things out without damaging the circuit board. I think I may have been successful overall. I had to remove everything down to the board to work in the space. Small warning- the fins arekind of edgy - I sliced my finger tips twice getting this out proceeded by dousing them in alcohol when I was cleaning the junk off the heatsink/asic.




And what I found is in my opinion utter stupidity. If you've applied thermal compound to a heat sink before, you know that the best practice is a very thin layer of compound to aid the transmission of heat. Too thick and you lose effectiveness. CPUs call for a compound the size of a grain of rice. What I found left me dumbfounded that this is how this equipment was designed. Perhaps I'm missing some of the finer details of enterprise switching...?

Pre-cleaning


Post Cleaning



Board after the fact - you can see a little scratching around the holes, but that should be fine given they're no circuitry there.


So I'm in a spot where I'd like to put a better heatsink on here - preferably one with a small fan integrated. The problem is that the heatsink literally doesn't have a part number and I cannot find any reference for the size / compatibility. I measure it to be approx 65mmx70mm 10mm fins and 2 push-pin mounts. There's nothing out there that fits the bill ... everything on digi and a few other sites are mostly square.

So I might just have to reapply paste and mount the heatsink. I think I have to replace the push-pins as I kind of mangled them getting them out - not really an issue as they're cheap I believe 3MM, and I can put slightly stronger springs on them this time around if I have to go that direction to improve contact with the ASIC.

Anyone know where I can get specs on this heatsink? I imagine re-applying good paste to the heatsink (like MX4 thermal compound or ceramique) could go a long ways here to improving temperatures given the previous situations.

 

[virulent]

I moved the L3 routing duties from my untangle vm to the ICX6450. It was a lot easier than expected, but just wanted to post what I did in one post since I ended up taking info from different posts here & externally to make one complete post. now my internet gw is just a firewall and not a does everything box at long last!

1. Move your DHCP outside of your firewall. I've been meaning to do this for such a long time but finally have.

It's convenient, it works, you know it all. But the more I can make my firewall just a dumb box that does the one function it's supposed to (you know.. connecting to the internet) the better. You also avoid issues with the crap firewall vendors stick you with, easing configuration when you get the itch to try the next greatest firewall.

For some different reasons you should avoid using the FastIron DHCP server as well.

My poison of choice is dnsmasq for zero effort DHCP+DNS integration, but you can do the exact same thing with ISC, Kea, Windows Server, whatever.

Your dnsmasq or equivalent configuration should look similar to this:

# Untagged VLAN DHCP
# The tag names (home,vlan11) are purely visual for you.
# The dhcp range/options used will be based on whether the ICX switch relayed a DHCP request or not
# (in which case it embeds the source VLAN's subnet! that is where the magic happens)
dhcp-range=home,10.10.1.100,10.10.1.240,255.255.255.0,12h
dhcp-option=home,option:router,10.10.1.1
dhcp-option=home,option:dns-server,10.10.1.8
dhcp-option=home,option:domain-search,BLAH.ca

# A single VLAN (11)
# Note you do NOT 'configure' the VLAN, you just configure the DHCP range and it will sort itself out
dhcp-range=vlan11,10.11.1.100,10.11.1.240,255.255.255.0,12h
dhcp-option=vlan11,option:router,10.11.1.1
dhcp-option=vlan11,option:dns-server,10.10.1.8
dhcp-option=vlan11,option:domain-search,BLAH.ca

• router: the default route/next-hop. I changed addressing so that the ICX6450 is .1 and my internet gw is .4 just to keep things consistent. Do as you do. In either situation, the router option should be set to the IP of the ICX device in the same subnet.
• dns server: your dns server. For me this is my dhcp server as well. Yours may be different.
• search domain: optional, only if you need it do you know you need it.

Then spend the next 20 mins copying over DHCP reservations to your config file and you are good.
VLAN1 will work automatically and use the right range. (really, I think it just uses the range with the same subnet as your DHCP host.)

At this point everything should be working as it was before.

2. Setup VLAN11 on the switch first. This will be the validation test that your config is OK.

N.B. I already had the tagged interfaces setup.... if you don't and are going to be using both VLAN 1 (untagged) and tagged VLANs on the same port, connect via serial and enable dual-mode on vlan 1 for those ports. Otherwise you will stop getting VLAN 1.

Using dual-mode is really the same as combining trunk+access (Cisco). You can trunk X vlans with an untagged native VLAN.
In my situation, 1/2/1 to 1/2/4 are connected to my proxmox hosts. They speak vlan1 but have trunks for a few tagged VLANs. This requires dual-mode.

# If you have NOT configured any VLANs before and need to enable dual-mode
# (because you want to send untagged vlan1 or another vlan as WELL as tagged traffic)
# you need to configure each interface with dual-mode, e.g.:
interface eth 1/2/4
dual-mode
# done

# Create VLAN 11 and tag interfaces
vlan 11 vlantest
tagged ethernet 1/2/1 to 1/2/4
router-interface ve 11

# Assign ve11 as the default route
interface ve 11
ip address 10.11.1.1/24
ip helper-address 1 10.10.1.8

• ip helper-address: This is the DHCP relay option. The dest is your DHCP server (any VLAN, so long as the switch can access it). The VE's address is used by your DHCP server to pick the same subnet.
• ip address: the IP address the switch will take in the subnet (& mask). This is what you will use for next-hop/default route on devices

At this point you should have a working VLAN 11 with DHCP. Try it. Everything else should be working as it is.

3. Change the internet gw to another IP. I changed mine from 10.10.1.1/24 to 10.10.1.4/24.
This is optional if you want to juggle around different IPs but I kept it consistent... .1- ICX6450, .4- Untangle.

Switch to the serial console at this point unless you live by the seat of your pants because you will probably lose connectivity unless you don't change the IP address.

4. Patch up VLAN 1 on the switch and fix the default routes.
Your addresses will of course be different.

# Change ve1's IP and default route
interface ve 1
no ip address 10.10.1.19/24
ip address 10.10.1.1/24

# (You need this if your DHCP server is on a different VLAN)
# (If it's the same VLAN, don't sweat it and you can omit this.)
ip helper-address 1 10.10.1.8

# Change the default route to the new firewall
exit
no ip route 0.0.0.0/0 10.10.1.1
ip route 0.0.0.0/0 10.10.1.4

5. Lastly, add a static route on the firewall for vlan11 (and any new future VLANs) to point at your switch. This will allow vlan11 to access the internet.
For mine, I created the route 10.11.1.0/24 via 10.10.1.1.
Yes, could just do the whole /8, but it prevents internet connectivity for new VLANs until I link it up which is a nice side effect.

At this point everything should work and your old firewall should no longer be doing inter-VLAN routing. Test DHCP, routing (between subnets and to the internet), etc.

$ traceroute 10.11.1.223
traceroute to 10.11.1.223 (10.11.1.223), 64 hops max
  1   10.10.1.1  1.764ms  1.757ms  0.762ms
  2   10.11.1.223  1.025ms  0.222ms  0.119ms

(10.10.1.1 is now my switch, 10.10.1.4 (the firewall) is no longer involved!)

# ./iperf3 -c 10.10.1.12 --bidir
Connecting to host 10.10.1.12, port 5201
[  6] local 10.11.1.223 port 51624 connected to 10.10.1.12 port 5201
[  8] local 10.11.1.223 port 51626 connected to 10.10.1.12 port 5201
[ ID][Role] Interval           Transfer     Bitrate         Retr  Cwnd
[  6][TX-C]   0.00-1.00   sec  1.01 GBytes  8.67 Gbits/sec  301    413 KBytes
[  8][RX-C]   0.00-1.00   sec  1001 MBytes  8.39 Gbits/sec
[  6][TX-C]   1.00-2.00   sec   954 MBytes  8.01 Gbits/sec    0    721 KBytes
[  8][RX-C]   1.00-2.00   sec   962 MBytes  8.07 Gbits/sec
[  6][TX-C]   2.00-3.00   sec  1.05 GBytes  9.02 Gbits/sec   26    758 KBytes
[  8][RX-C]   2.00-3.00   sec  1.04 GBytes  8.93 Gbits/sec
[...]
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID][Role] Interval           Transfer     Bitrate         Retr
[  6][TX-C]   0.00-10.00  sec  10.1 GBytes  8.64 Gbits/sec  1026             sender
[  6][TX-C]   0.00-10.00  sec  10.1 GBytes  8.63 Gbits/sec                  receiver
[  8][RX-C]   0.00-10.00  sec  10.1 GBytes  8.67 Gbits/sec    0             sender
[  8][RX-C]   0.00-10.00  sec  10.1 GBytes  8.67 Gbits/sec                  receiver

(full duplex working! this is inside random containers, so the speed is what it is)