Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[arglebargle]

What kind of circuitry are they using to control the fans on these? I've been considering pulling a 6450 apart (or a 6610) and replacing the fans with a 3d printed duct and quiet blower to use under my desk in my bedroom. Will the switch still function if it senses fan failure or do I need to look into building something that speaks i2c to keep the switch happy?

 

[kapone]

i2c...already been down this road. And even then, the switch is "hard coded" for the RPMs of the stock fans. Ask me how I know... Hacked apart one of the fan assemblies and put in slower fans...switch won't boot.

 

[PGlover]

Got 2 ICX6610 on the way.. Now I need to convert all of my Juniper EX3300 configuration to ICX6610. Time to learn the CLI commands for the ICX6610 unit.

 

[arglebargle]

i2c...already been down this road. And even then, the switch is "hard coded" for the RPMs of the stock fans. Ask me how I know... Hacked apart one of the fan assemblies and put in slower fans...switch won't boot.

Oh dear. Alright, that's what I was afraid of. I'll look into this more when I have more time but if there's somewhere to grab 5v or 12v from the power supply it might be viable to build an i2c dummy out of an arduino and plug that into the fan headers to fake the switch out. I was hoping it'd be simpler.

 

[nthu9280]

Most 2U servers & this switch don't belong in your home office Should be located in a sufficiently cooled closet away from the living areas

 

[kapone]

Oh dear. Alright, that's what I was afraid of. I'll look into this more when I have more time but if there's somewhere to grab 5v or 12v from the power supply it might be viable to build an i2c dummy out of an arduino and plug that into the fan headers to fake the switch out. I was hoping it'd be simpler.

It's even worse. The fans in the power supplies are controlled by the switch over i2c as well. If you take apart one of the power supplies (and I did), the fan/power led of the PSU connects to a header inside the PSU. If you remove this wire, the PSU won't turn on...

If you replace the fan IN the PSU with a slower one, the PSU will turn on, but the switch won't boot...'cause..well Brocade said FU when it came to silencing this switch.

That said, in all fairness, the 6610 is 1U switch with almost o.5tbps of switching bandwidth. That's a lot. The 6610 does get seriously hot, and probably needs these screamers, unless they change the chassis design. After letting the switch run for a few hours, open it up, and you can barely touch the heatsinks for more than a few seconds. They are that hot and that's WITH the screamers. If you even manage to replace the fans with something slower, there's a better than even chance, you'll cook the switch.

I went a slightly different route to silence mine. After realizing I can't change the fans, can't remove them, can't slow them down...well..it's all about airflow and temps, so give the switch more airflow and lower the temps WITH all stock fans. That will make the switch happy, and it'll slow down the fans internally.

So...I cut apart the top cover to mount THREE 120mm fans pretty much covering the front (or back, depending on how you look at it) 3rd of the width. These are nothing special (Yate Loon fans I had lying around) and standard 3 pin 12v fans. Now came the question of how to power them from the switch.

The PSU I had hacked earlier...hacked it even more and soldered wires internally on the big connector (that mates to the switch) to get 12v and GND out. Then routed these two wires carefully out the back of the PSU (next to the power LED), so that I can connect them to the fan header for the 3 fans.

It's pretty "hacky", but I had to get the noise down, my home office is about 10ft from where the rack is and even with a wall in between, in stock configuration, I could hear it...a lot...even on the first floor.

 

[paidhima]

I bought a 6450-48 (non-PoE) early last year after using an FWS648 (with the old Foundry logo) for a while, salvaged from a customer. I enjoyed the 648 so much that I looked for another Brocade when I needed 10Gb support. I paid $400 for mine which, at the time, was the best price I could find for that switch from a reputable seller. It was well worth the money to me and has been a real joy to work with. The only annoyance is that it uses an old encryption method for SSH and any Linux machine I use refuses to connect unless I edit settings or use PuTTY. I had no idea the price had dropped this much. I'm consider picking up a 6610 now just to mess around with.

 

[buckminst]

I'm considering picking up an ICX6610 because of this thread, but while looking in the manual I saw something that made me want to get clarification:

The manual states that the 4x10GB breakout QSFPs on the back are not for data, they can only be used for stacking. Is this inaccurate, or is there something special that must be done in order to use those breakout connections to connect to my servers?

 

[fohdeesha]

The manual states that the 4x10GB breakout QSFPs on the back are not for data, they can only be used for stacking.

It's a marketing lie so people wouldn't realize it was half the price and had just as much 40gbE connectivity as their flagship SX800 chassis router at the time. all the rear QSFP ports (including the non-breakout 40gbE) work great for general use, just try not to publicize it too much

 

[PGlover]

On the ICX6610, what is the interface numbering scheme for the 8 10gbe ports on the first of the switch. Is it 1/2/1 - 1/2/8? Please advise.

 

[PGlover]

On the ICX6610, what is the interface numbering scheme for the 8 10gbe ports on the first of the switch. Is it 1/2/1 - 1/2/8? Please advise.

I found the answer... 1/3/1 - 1/3/8

 

[anomaly]

What kind of circuitry are they using to control the fans on these? I've been considering pulling a 6450 apart (or a 6610) and replacing the fans with a 3d printed duct and quiet blower to use under my desk in my bedroom. Will the switch still function if it senses fan failure or do I need to look into building something that speaks i2c to keep the switch happy?

I've been learning Fusion360 just for designing shrouds/ducts. PM me, I'm interested on discussing some designs.

Like others said, the PSUs have data lines to monitor/check for these things. It's a PITA. Without putting together a PCB with a MCU that spoofs those signals, you won't get anywhere sadly. Like others said too, the thermal of the 6610 is already tight enough with the stock fans. I can't imagine making it work well with slower/less CFM fans.

I just got a bunch of Sunon and Delta units to test if anyone is interested.

 

[PGlover]

What is the default routing protocol on the ICX6610?

My plan is to run 2 ICX6610 in a stack configuration. The switches will be connected to my pfSense application upstream. There will be no downstream switches.

Is running OSPF as my routing protocol overkill for my setup? If yes, what routing protocol should I setup on the ICX6610?

 

[tommybackeast]

fohdeesha : somewhere in this thread you mentioned the $60 MikroTik S+RJ10 transceivers (10GBe RJ45) worked fine in the Brocade 6450.

MikroTik

I had been planning to find the time to do a Fiber run from 2nd floor office (Win7 Workstation) to Basement (Brocade Switch); but given the many twists and turns to run cable through the attic and down; since you mentioned these MikroTik RJ45 10GBe transceivers, I am thinking of just using Cat6 Shielded (there's a lot of 12/2 Electrical lines to cross over and ran parallel to. (NAS is in the basement and connected to Brocade via 10GBe connection)

Just wanted to confirm that for a cable run of 75-100 feet; using two MikroTik RJ45 10GBe transceivers + Shielded Cat6 cable will give me the same file transfer speeds as using Fiber and two SFP+ 10GBe transceivers.

Thanks

 

[fohdeesha]

Just wanted to confirm that for a cable run of 75-100 feet; using two MikroTik RJ45 10GBe transceivers + Shielded Cat6 cable will give me the same file transfer speeds as using Fiber and two SFP+ 10GBe transceivers.

Thanks

it depends so I can't say for sure, especially at those distances. 10gbps over copper is much more finicky - there's a thread somewhere on STH about that transceiver and distance tests, some people still got 10gbps at 100ft, others didn't. If you're worried about the "routability" of fiber, most multimode fiber is more flexible/easier to run than shielded cat6/cat7 in my experience (the shielding makes it less flexible)

 

[fohdeesha]

What is the default routing protocol on the ICX6610?

My plan is to run 2 ICX6610 in a stack configuration. The switches will be connected to my pfSense application upstream. There will be no downstream switches.

Is running OSPF as my routing protocol overkill for my setup? If yes, what routing protocol should I setup on the ICX6610?

What do you mean default routing protocol? It doesn't come with any routing protocols enabled by default (that would make it a nightmare to plug in new switches into an existing network)

I see zero reason for OSPF in that setup, OSPF is nice to automatically propagate new routes to a bunch of switches without having to manually add them to each router/l3 switch - but I don't see new routes ever getting added in your system, let alone regularly enough to warrant needing a protocol to do it for you

As for what routing protocol to recommend, what makes you think you need one at all? Just LACP from both switches to your pfsense appliance. If you want to have seperate vlan's on the brocade's and have them perform inter-vlan routing (and not pfsense), you would just use static routes, not a routing protocl

 

[PGlover]

What do you mean default routing protocol? It doesn't come with any routing protocols enabled by default (that would make it a nightmare to plug in new switches into an existing network)

I see zero reason for OSPF in that setup, OSPF is nice to automatically propagate new routes to a bunch of switches without having to manually add them to each router/l3 switch - but I don't see new routes ever getting added in your system, let alone regularly enough to warrant needing a protocol to do it for you

As for what routing protocol to recommend, what makes you think you need one at all? Just LACP from both switches to your pfsense appliance. If you want to have seperate vlan's on the brocade's and have them perform inter-vlan routing (and not pfsense), you would just use static routes, not a routing protocol

Ok.. I will have Vlans and plan to setup router interfaces for inter-vlan routing on the switch and not pfSense. Do I need to setup the static routes myself for inter-vlan routing or will the switch create the static routes in the routing table automatically?

As far the connection to the pfSense appliance, I will setup LACP from both switches to the pfSense appliance and create a static route to the pfSense appliance ( ip route 0.0.0.0/0 10.1.1.6).

 

[fohdeesha]

it will automatically create static routes for vlan interfaces as you add the VE interfaces to vlans

dont forget that pfsense will also need a route to know how to reach all the vlans, see the diagram here: https://forums.servethehome.com/ind...ware-trouble-routing-vlans.21113/#post-196527

 

[PGlover]

it will automatically create static routes for vlan interfaces as you add the VE interfaces to vlans

dont forget that pfsense will also need a route to know how to reach all the vlans, see the diagram here: https://forums.servethehome.com/ind...ware-trouble-routing-vlans.21113/#post-196527

Exactly.. I have a transit vlan (vlan 2000) link with a /29 subnet mask from the switch to my pfSense appliance. Within pfSense I have setup routes to reach all my vlans.

Thanks for all the great knowledge.

 

[dec]

Anyone managed to pick up any of the DC PSUs for the 6610? They're RPS16DC-{I,E}.