Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[fohdeesha]

there's nothing you can change or do on the 6610, trust me. If there's one point I've made in this thread that I wish would actually stick, it's this one. They designed the 6610 on the absolute thermal limit - as you noticed, totally stock, it runs HOT. they specced it with the least airflow/noise possible in 1Ru. If you change the fans/reduce airflow by even 5%, it's going to cook. It's a half a terabit router in 1RU, every other switch in this class has an idle equivalent to the 6610's on full blast. The fact that 6610's EVER run succesfully on fan speed 1 is a miracle, especially with 80f ambient intake

I sincerely doubt changing the thermal compound will net you more than half a degree, again, given this was a $10k switch with quite a lot of engineering put into thermals, if the engineers could have shaved off an entire 5 degrees by speccing 50 cent more expensive thermal compound, they would have done it.

The ONLY solution that has any chance of working is what @kapone did, I'm sure he'll chime in. You can't modify the existing fans and monitoring for a lot of reasons, so he basically cut the top half of his switch off, and stuck some case fans over it to augment the existing cooling. That has kept his switch on fan speed 1 24/7. Obviously, this involves cutting the chassis lid up, and will make it take up much more than 1RU.

It's possible a second fan tray will move just enough extra air to keep it under threshold, but you'll then be contending with the extra noise of a second fan tray. no free lunch

 

[mixmansc]

Ah - did not read back through the whole thread so did not catch that this models thresholds cannot be changed. No worries. Not going to cut the top up but I do have a fan in there that directs some cooler air into the closet. I'll have to aim it a bit different. I can also re-route the bulk of the cables and mount it lower in the rack. Right now its at the very top which is obviously the hottest place for it to be. Just did a quick peek on eBay and the dang fan modules are pretty pricey. That might be enough though. On low you cannot hear it in there at all.

Just need to do some adjustments to get a little more air to it. Thanks man!

 

[tommybackeast]

Brocade 6450 - if one mounts this in a server cabinet at the rear of the rack (backwards from normal);
is there any way via the CLI to "reverse" the fans' direction ?

 

[PithyChats]

Brocade 6450 - if one mounts this in a server cabinet at the rear of the rack (backwards from normal);
is there any way via the CLI to "reverse" the fans' direction ?

Sadly no, the fans are actually physically reversed in the two different orientations. You could probably open up the chassis and the power supply and just swap the fan orientations, which would give you the desired results.

 

[ljvb]

I'm unfamiliar with SFP modules. for the 6610, to stack, what modules would I need to run a 100f stacking cable (1 switch will be in my server rack in the laundry room where my rack is (its noisy.. blame my wife ). The patch panel from the drops in the house will be in the utility room on the second floor, where the second 6610 will be going.

 

[PithyChats]

I'm unfamiliar with SFP modules. for the 6610, to stack, what modules would I need to run a 100f stacking cable (1 switch will be in my server rack in the laundry room where my rack is (its noisy.. blame my wife ). The patch panel from the drops in the house will be in the utility room on the second floor, where the second 6610 will be going.

For 100' it depends if you want a 10G or 40G. For 10G, you'll want 10G-SR SFP+ modules. Any will do, but the major brands to look for are Finisar, Avago, or just get some from FS.com. Then you'll need an OM3 or OM4 LC\LC cable.

EDIT - Turns out only the 6450 supports 10G stacking. If you want to stack the 6610 you'll need to go 40G.

If instead you want to do 40G, you'll want 40G-SR4 QSFP modules and an MTP or MPO trunk cable. You'll need a type B female to female trunk cable (up to up) like this one suggested by fohdeesha:

MTP Female 50M Interconnect Cable Assembly, Flat Ribbon, Plenum Rated, 12-Fibers | eBay

EDIT - This cable is key up to key down. It'll need some modification. See the post below

@fohdeesha mentioned this QSFP module will work, for a 4 x 10G breakout, but not sure if it will work for stacking (I'm fairly certain it will, but maybe he can confirm)

Brocade XBR-000232 QSFP 57-1000267-01 4*16Gb SWL 100m for DCX8510 EMC | eBay

 

[fohdeesha]

that optic will definitely work for stacking

However you need key up to key up MTP/MPO, and pretty much every MTP/MPO cable on ebay (and elsewhere) is key up to key down (including the one pithy linked). The easy/cheap way around this is to buy one of the cheap key up to key down MTP or MPO cables and swap the key direction on one of the ends. I outline how to do this here: https://forums.servethehome.com/ind...icx6450-icx6610-etc.21107/page-23#post-201367

Not terribly difficult, @Gamble did exactly this to stack his two 6610's long distance.

Regarding stacking via 10gbe ports instead, I'm not actually sure if it supports this. I know you can specify the 10gbE ports for stacking when connecting them to an ICX6450 in a mixed stack, but never tried doing 10gbe port stacking between two 6610's. The official docu can't be trusted on this, because it still says things like "the rear stacking ports cannot be used for data". So even if it explicitly says the front 10gbe ports can't be used for stacking, there's still a chance they will just fine

edit: yeah not possible, 40gbe stacking only:

ICX1(config-unit-1)#stack-port 1/3/5
Error! 1/3/5 must be one of the stacking ports: 1/2/1 1/2/6.

 

[u238]

Arrrgghhhh. I just received my second 6450 badly damaged. A rack ear is totally destroyed, and the whole chassis is bent and dented. I'm not having great luck. Of course there are no more reasonably priced switches on ebay...

 

[MikeWebb]

slightly off topic
RE: Brocade XBR-000232 QSFP 57-1000267-01 4*16Gb SWL 100m
Since these are 4*16G, can I assume they are FDR and I can use them for 56Gb links with my Mellanox IB switch?

 

[juey]

Hi there,

after reading trough a the first 15 pages my first post so...are there any good sources to get a 6610-24P-E that ships that beast over to Europe ?
I can get a bunch of 6610 with 2Y Warranty, beginning at 700 $ (incl. VAT/Shipping)

The 40G Ports on the back can be used without having installed any additional licenses (e.g. with 4x 10G Breakout) right ?

 

[ljvb]

Since I do not require redundancy, do I need 2 cables still to stack the switches. My purpose is just to manage them both from 1 device. The images I have found, and instructions (which you say not to completely believe) seem to show two cables.

that optic will definitely work for stacking

However you need key up to key up MTP/MPO, and pretty much every MTP/MPO cable on ebay (and elsewhere) is key up to key down (including the one pithy linked). The easy/cheap way around this is to buy one of the cheap key up to key down MTP or MPO cables and swap the key direction on one of the ends. I outline how to do this here: https://forums.servethehome.com/ind...icx6450-icx6610-etc.21107/page-23#post-201367

Not terribly difficult, @Gamble did exactly this to stack his two 6610's long distance.

Regarding stacking via 10gbe ports instead, I'm not actually sure if it supports this. I know you can specify the 10gbE ports for stacking when connecting them to an ICX6450 in a mixed stack, but never tried doing 10gbe port stacking between two 6610's. The official docu can't be trusted on this, because it still says things like "the rear stacking ports cannot be used for data". So even if it explicitly says the front 10gbe ports can't be used for stacking, there's still a chance they will just fine

edit: yeah not possible, 40gbe stacking only:

ICX1(config-unit-1)#stack-port 1/3/5
Error! 1/3/5 must be one of the stacking ports: 1/2/1 1/2/6.

 

[fohdeesha]

you can definitely stack with just one cable, a few people in here are doing this to use the rest of the stacking ports for normal use

 

[Gamble]

For 100' it depends if you want a 10G or 40G. For 10G, you'll want 10G-SR SFP+ modules. Any will do, but the major brands to look for are Finisar, Avago, or just get some from FS.com. Then you'll need an OM3 or OM4 LC\LC cable.

EDIT - Turns out only the 6450 supports 10G stacking. If you want to stack the 6610 you'll need to go 40G.

If instead you want to do 40G, you'll want 40G-SR4 QSFP modules and an MTP or MPO trunk cable. You'll need a type B female to female trunk cable (up to up) like this one suggested by fohdeesha:

MTP Female 50M Interconnect Cable Assembly, Flat Ribbon, Plenum Rated, 12-Fibers | eBay

EDIT - This cable is key up to key down. It'll need some modification. See the post below

@fohdeesha mentioned this QSFP module will work, for a 4 x 10G breakout, but not sure if it will work for stacking (I'm fairly certain it will, but maybe he can confirm)

Brocade XBR-000232 QSFP 57-1000267-01 4*16Gb SWL 100m for DCX8510 EMC | eBay

Since I do not require redundancy, do I need 2 cables still to stack the switches. My purpose is just to manage them both from 1 device. The images I have found, and instructions (which you say not to completely believe) seem to show two cables.

I used that exact cable and those brocade modules in my setup, 2x6610 spread about 150ft apart and it worked perfectly! Doing the flip to key-up wasn't bad, follow the video guide and you'll be fine. The trickiest part was getting the end back on. Takes a little wiggling and using a paper clip or pin to guide the end into place. Just remember to pull that cord tight when you do it.

 

[tommybackeast]

Sadly no, the fans are actually physically reversed in the two different orientations. You could probably open up the chassis and the power supply and just swap the fan orientations, which would give you the desired results.

I have not yet opened up the switch chasis, and can appreciate that the fans inside only blow air in one direction.

I was not even thinking of the fan inside the PSU, for that would need to reverse also.

Might you know, is it simply powering down the switch; and then lifting up the fans and reversing their direction; and then re-sitting them "backwards" ?

 

[fohdeesha]

he was thinking of the 6610 with hot swap power supplies, the 6450 doesn't have any power supply fans

You can just flip them yeah, you'll have to untuck the wires from the little plastic tab on the fan so they're not coming out the front where the back of the case will be after you flip them

 

[tommybackeast]

he was thinking of the 6610 with hot swap power supplies, the 6450 doesn't have any power supply fans

You can just flip them yeah, you'll have to untuck the wires from the little plastic tab on the fan so they're not coming out the front where the back of the case will be after you flip them

thank you, i imagine your paragraph will make a great deal more sense when the 6450 chasis is open, which will not be tonight.

no rush, but see PM about you getting a free PDU to play with (if you want it)

 

[kapone]

I have not tried physically reversing the fans...but...I do know that the fan tray that connects (the two "tandem" fans each), to the switch has resistors in the wire/connector (on the fan side). That's how it knows if you put in a fan tray that's "front to back" or "back to front". You'll have to hack that as well.

 

[ljvb]

I picked up 2 6610-24p for $500 (hoping that is a reasonable price). They wanted $295 each, but as the seller was local, I offered 500 for both.

Currently going to go through the update.. as they are horribly out of date.

This is the current version.

ICX6610-24P Switch#show version
  Copyright (c) 1996-2011 Brocade Communications Systems, Inc.
    UNIT 1: compiled on Apr 25 2012 at 16:41:32 labeled as FCXS07300c
(5325231 bytes) from Primary /foundry/FGS/os/FCXS07300c.bin
        SW: Version 07.3.00cT7f1
  Boot-Monitor Image size = 370555, Version:07.3.02T7f5 (grz07302)
  HW: Stackable ICX6610-24-HPOE
==========================================================================
UNIT 1: SL 1: ICX6610-24P POE 24-port Management Module
  Serial  #: xxxxxxxxxxxxxxxx
  License: BASE_SOFT_PACKAGE   (LID: dzoHKHKlFgH)
  P-ENGINE  0: type E02B, rev 01
==========================================================================
UNIT 1: SL 2: ICX6610-QSFP 10-port 160G Module
==========================================================================
UNIT 1: SL 3: ICX6610-8-port Dual Mode(SFP/SFP+) Module
==========================================================================
  800 MHz Power PC processor 8544E (version 0021/0023) 400 MHz bus
65536 KB flash memory
  512 MB DRAM

 

[fohdeesha]

I have not tried physically reversing the fans...but...I do know that the fan tray that connects (the two "tandem" fans each), to the switch has resistors in the wire/connector (on the fan side). That's how it knows if you put in a fan tray that's "front to back" or "back to front". You'll have to hack that as well.

check the model he has in the 3 posts before yours

 

[kapone]

check the model he has in the 3 posts before yours

I need more coffee again...dammit. Too many things, too little time.