Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

Churchill

Admiral
Jan 6, 2016
788
185
43
30gbps iperf is good. you're not going to do much better than that without newer CPUs and some kernel tweaking. 45MB/s file shares sounds like NFS with sync=enabled or something

I am doing an rsync between two systems to replicate the data.

These are both 2620 Xeon V2 so relatively new. I'd love to know what kind of kernel tweaking i can do.

The 40GB ports are all in their own VLAN seperated out from any network and traffic. fully isolated. I have 10GB ports I can hookup as well.
 

infoMatt

Active Member
Apr 16, 2019
146
53
28
I am doing an rsync between two systems to replicate the data.

These are both 2620 Xeon V2 so relatively new. I'd love to know what kind of kernel tweaking i can do.

The 40GB ports are all in their own VLAN seperated out from any network and traffic. fully isolated. I have 10GB ports I can hookup as well.
You could test if the bottleneck is in the switch making a direct connection between the two systems... if both NICs are at 40Gbe...
 

Whitecitadel

New Member
Feb 13, 2018
1
0
1
100
Looks like the guide needs an update as 6650 is no longer anywhere close to $300!

Is there anything between 6610 and 6650 with more QSFP ports than 4?

Churchill - try more iPerf threads to saturate the link?
 
Last edited:

pir2

New Member
Aug 3, 2011
4
0
1
NOTE: All Brocade switches are compatible with the Mikrotik S+RJ10 SFP+ 10gbase-T copper module, if you want to cheaply turn some of the sfp+ ports into rj45 10gbE copper ports: Mikrotik 6-Speed Rj-45 Module Up To 10Gbps Speeds
Might be helpful to others...

The Mikrotik S+RJ10 does not work with the ICX-7150-C12P. It shows up as an SFP Module intsead of SFP+. However, FS.com ones seem to work.

SW version: 08080d

Code:
show media ethernet 1/3/1
Port   1/3/1: Type  : 1GE M-SX(SFP)
             Vendor: MikroTik           Version: 2.11
             Part# : S+RJ10             Serial#:
icx(config)#show media ethernet 1/3/2
Port   1/3/2: Type  : 10GE Passive Twinax  1m (SFP+)
             Vendor: OEM                Version: D
             Part# : SFP-10G-T          Serial#:
 

juey

Member
Oct 1, 2018
56
14
8
Germany
Is it somehow possible to grab transceiver stats like rx power, temps,... unlike by the CLI? The SNMP MIB does not contain any entries for that, it seems that this feature is absent in all fastiron powered devices.
 

yukaia

New Member
Aug 8, 2017
4
0
1
Is it somehow possible to grab transceiver stats like rx power, temps,... unlike by the CLI? The SNMP MIB does not contain any entries for that, it seems that this feature is absent in all fastiron powered devices.
You have to run Brocade optics to get those metrics.
 

BoGs

New Member
Feb 18, 2019
2
0
1
I was looking on Ebay for some of these switches and prices sure have gone up ;) I guess I will keep my nortel and arista :)
 

fohdeesha

Kaini Industries
Nov 20, 2016
1,851
1,644
113
29
fohdeesha.com
Is it somehow possible to grab transceiver stats like rx power, temps,... unlike by the CLI? The SNMP MIB does not contain any entries for that, it seems that this feature is absent in all fastiron powered devices.
the MIB absolutely contains entries for that bro

 

juey

Member
Oct 1, 2018
56
14
8
Germany
You have to run Brocade optics to get those metrics.
I know, i have Brocade optics and i do get transceiver stats via CLI.

the MIB absolutely contains entries for that bro
Ok, i did not express myself well, the MIB contains the objects, but they are not available on the ICX6610/6450:

Code:
juey@influxdb:~# snmpwalk -v2c -c public <IP> 1.3.6.1.4.1.1991.1.1.3.3.6.1.1
FOUNDRY-SN-SWITCH-GROUP-MIB::snIfOpticalMonitoringTemperature = No Such Object available on this agent at this OID
ifIndex to optical parameters table
If the table provides an ifIndex, the ifIndex to optical parameters table returns the optical parameters for the ifIndex.
NOTE
The following objects provide information for POS and Ethernet optical monitoring. They are equivalent to the output of the show
optics command.
NOTE
The following table objects are not supported on the Brocade FastIron series devices.
Found on page 259 from ipmib-aug2016-reference.pdf
 

fohdeesha

Kaini Industries
Nov 20, 2016
1,851
1,644
113
29
fohdeesha.com
it definitely works, at least on the icx7xxx series stuff I have access to test. there's a new table that takes into account lanes as well in case of multilane optics like 40g etc which is the table you need to use, the old one was removed:



everything in that table works to pull info off of normal 10gbe optics:



I guess it's possible it was never added to the 6xxx series, not sure
 

blinkenlights

Active Member
May 24, 2019
133
48
28
First, many thanks to @fohdeesha for all of his online and offline help. I've been playing around with Brocade's ICX7xxx switches for the past month and arrived at some conclusions that may be helpful for other members.

My servers and network gear are located behind a door in an unfinished, unconditioned 250 sq. ft. utility area of the basement.. not ideal, but not unlike most homelabs. I purchased an ICX7250-48 (non-PoE) in decent condition a few weeks ago, but was disappointed by the thermal characteristics and stock fan noise. About a week later, I purchased a late-model ICX7450-48 (non-PoE) data center denizen and migrated my home network.

Here are my findings for people deciding between the ICX7250 and ICX7450:
1) The main ASIC in the ICX7250 (BCM56344) runs hot, especially in a room without an AC unit/direct vent. On cold boot at 22 deg-C (72 deg-F) ambient, the temperature sits around 55 deg-C and quickly rises to the low 70's (158+ deg-F). My ICX7450 (BCM56548) sits around 48 deg-C fully loaded and the fan noise is nearly the same to my old ears.

The ICX7250 temperatures were not unexpected by Brocade, though: warning at 100.0 deg-C and shuts down at 105.0 deg-C. I chickened out around 80 deg-C and fully disassembled the switch, cleaned it up, and applied a quality ceramic thermal paste to the main ASIC. I then swapped the original Foxconn fans for the infamous "blew itself off the table" "sliced open my finger" Delta ultra-high speed fans (FFB0412SHN-F00). Yes, these fans move a ton of air at full voltage.. but the switch typically runs them around 30% of their maximum speed. The switch is much quieter - downright tolerable at 56dB a few feet away - and idles at 68 deg-C under the same environmental conditions. Still planning to post it for sale, as I don't need two core switches for home.

Long story short, if you can tolerate fan noise and/or have a way to isolate the noise, the ICX7450 (and ICX6610, I imagine) are better performers for the money.

Regarding the ICX7450:
2) Find a switch with Rev. C power supplies - buy them separately if the price makes sense. I don't have hands-on experience with the older power supplies, but trust when people say they are much louder.
3) You might want to pick up spare parts if you are in this for the long haul. I plan to buy one cold spare Rev C. power supply and one fan.

General advice, aka everything of this world has a price:
4) Unless you plan to use 24 or 48 ports with PoE, go for the non-PoE version. Heat generated by the additional circuit board(s) and larger power supplies will not be helping you. If I need to run PoE for a security camera or two, I can afford to buy an injector.
5) Nothing unnecessary in the slots generating more heat or blocking airflow. If you don't need multiple QSFP ports for stacking, don't install them! In my case, I have a single ICX7400-4x10GC module up front but nothing in the rear slots. When I remove the module, the temperature drops from 48 deg-C to around 40 deg-C.

Hope all of this helps someone out there thinking of buying one of these impressive switches.
 
  • Like
Reactions: fohdeesha

Necrotyr

Active Member
Jun 25, 2017
179
46
28
Denmark
Does any one know why Brocade though 5mm holes in their rack ears were a good idea?

I have to drill out the holes to make my M6 bolts and cagenuts fit...
 

itronin

Active Member
Nov 24, 2018
376
232
43
Denver, Colorado
I'm guessing to sell more 4 post kits? (jk)
ICX6610-RMK-4P
I'm not sure why all relay and square hole racks and can't be one thread type. though I guess with square hole you can probably get cage nuts with 10-32 or 12-24 and avoid drilling out the ears...

Thanks for the info on this - I had not tried m6 on ears and *was* thinking about consolidating all my gear into 1 four poster. I guess if I do I'll need a mix of cage nuts - probably paint 'em red or something so I know by looking at them what they are.
 

Necrotyr

Active Member
Jun 25, 2017
179
46
28
Denmark
M5 bolts and cagenuts would do the trick, but I don't have any of those, so I went and found my round file and voilá!

Bear in mind, that I'm speaking about the ICX6450 ;)
 

magi

New Member
Aug 22, 2019
9
3
3
United States
Anyone doing a PSU fan switch? I disassembled one of my PSUs and the fan is FFB0412UHN. Wonder if there are recommended alternatives to it?
 

fohdeesha

Kaini Industries
Nov 20, 2016
1,851
1,644
113
29
fohdeesha.com
Anyone doing a PSU fan switch? I disassembled one of my PSUs and the fan is FFB0412UHN. Wonder if there are recommended alternatives to it?
what switch, 6610? it monitors the rpm of the psu fans, if you replace it needs to be roughly the same rpm ~23k full speed, or it will refuse to boot. only other way is to build or buy something to trick the RPM signal the PSUs see from the fan. read: pain in the ass
 
  • Like
Reactions: magi

magi

New Member
Aug 22, 2019
9
3
3
United States
what switch, 6610? it monitors the rpm of the psu fans, if you replace it needs to be roughly the same rpm ~23k full speed, or it will refuse to boot. only other way is to build or buy something to trick the RPM signal the PSUs see from the fan. read: pain in the ass
Can try put a small PPM generator in so can trick the monitor. Anyway it has two PSUs so can work on one and still run the router.

Will post if succeeded ;o
 

juey

Member
Oct 1, 2018
56
14
8
Germany
What side effects do i have to face, if the uplink connection of stacking members will exceed the 100m limit ? Losing stack members, lost packets ? I want to stack one member using a 140m multimode cable, otherwise i just can stack the cores and use regular LAGs.