Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[kiteboarder]

Hi friends,

My 6610 started spinning up to fan speed 2. It would spin up for 10-30 seconds and slow back down to speed 1 for awhile. Rinse / repeat. It did this for a couple days and I finally got tired of it, so I took off the lid. It had been running for over 300 days in a garage rack, so some amount of dust had built up on the faceplate, but it wasn't blocked or anything terrible.

I opened the lid and found this:



Yeah, the spring loaded heatsink retention pin had broken due to age/heat/thermal stress. Luckily the metal spring landed on top of the heatsink and didn't short anything. And yes, it had been running like this for multiple days.

I couldn't get the remnants of the push pin out of the MB hole. So in a bind for time, I scientifically cut a piece of foam to squish between the top of the heatsink and the case lid, to push the heatsink back down. It's now been running like this for multiple days, staying at fan speed 1. Done.

Thanks again to fohdeesha for enabling such a great switch to be the backbone of my (and now several friends) home network.

Cheers

 

[clcorbin]

You can indeed breakout the switch qsfp ports to 4x 10gbe, but you can't do the same from the nic - nics do not support qsfp breakout, at least none I've heard of. You can use a qsfp > sfp+ adapter (search eBay for 655874-B21) and get a single 10gb connection out of the nic though

That kind of sucks! I was hopping to use the two 40Gb ports (1/2/1 and 1/2/6) for stacking (with the way my house is wired, it is MUCH better to have a stacked switch in the garage and another one in my server rack on the other side of the house) and use the other two 40Gb ports to connect my two servers to the switch. I know they had to operate as 4x10Gb, but I was hoping it was possible to use the "right" MPO cable so they appeared as 4 independent 10Gb ports on the switch and on the server. But with only a single MPO cable connecting them. That would work well in ESXI for my use case.

 

[aaroneaton]

Hmm mine does the same thing, wonder if its the same issue..

Hi friends,

My 6610 started spinning up to fan speed 2. It would spin up for 10-30 seconds and slow back down to speed 1 for awhile. Rinse / repeat. It did this for a couple days and I finally got tired of it, so I took off the lid. It had been running for over 300 days in a garage rack, so some amount of dust had built up on the faceplate, but it wasn't blocked or anything terrible.

I opened the lid and found this:

View attachment 19652

Yeah, the spring loaded heatsink retention pin had broken due to age/heat/thermal stress. Luckily the metal spring landed on top of the heatsink and didn't short anything. And yes, it had been running like this for multiple days.

I couldn't get the remnants of the push pin out of the MB hole. So in a bind for time, I scientifically cut a piece of foam to squish between the top of the heatsink and the case lid, to push the heatsink back down. It's now been running like this for multiple days, staying at fan speed 1. Done.

Thanks again to fohdeesha for enabling such a great switch to be the backbone of my (and now several friends) home network.

Cheers

 

[clcorbin]

Hmm mine does the same thing, wonder if its the same issue..

During testing on mine, it did the same thing as well. Perhaps a little snooping is in order. Better to fix this now than after I put it into service.

 

[aaroneaton]

Ya, have a new one coming.. well a new to me coming today. swapping it out then will take it apart and see.. hoping i can fix it and then stack them

During testing on mine, it did the same thing as well. Perhaps a little snooping is in order. Better to fix this now than after I put it into service.

 

[leon_pro]

not good. They draw more power than the ICX6610 with WAY less 10gbe ports, they run NOS which is nothing like fastiron (I do not like it), they're now owned by Extreme which has no public firmware updates, the VDX series has almost no L3 features, they are extremely vendor locked (non-brocade optics will not even link up).

The VDX product philosophy has always been the total opposite of the fastiron stuff. Fastiron heritage is a lot of engineers from Foundry etc, while VDX has always been a totally separate group that shares more heritage with Brocade's fibrechannel product lines than it does with Foundry's ethernet stuff

Is it REALLY so bad? I see some good lots on ebay, and I am thinking about buying one.
What I need is: 8 or more 10GBase-T ethernet ports, SFP+ is optional, 40Gbit uplink port would be a big plus. I don't think I need L3 features, just need to connect several hosts with 10G/40G ethernet. I was thinking on getting Mikrotik CRS312-4C+8XG, it lacks QSFP+, but still good enough.

Maybe VDX 6740t is good enough for me? What should I ask seller before buying one (licenses, etc.)?

 

[Sean Ho]

That kind of sucks! I was hopping to use the two 40Gb ports (1/2/1 and 1/2/6) for stacking (with the way my house is wired, it is MUCH better to have a stacked switch in the garage and another one in my server rack on the other side of the house) and use the other two 40Gb ports to connect my two servers to the switch. I know they had to operate as 4x10Gb, but I was hoping it was possible to use the "right" MPO cable so they appeared as 4 independent 10Gb ports on the switch and on the server. But with only a single MPO cable connecting them. That would work well in ESXI for my use case.

I might be misunderstanding your desired layout, but could you not use the breakout ports (e.g., 1/2/2) for stacking with MPO, and the 40Gbps ports (e.g., 1/2/1 and 1/2/6) to servers with cheap NetApp QSFP+ DACs and CX354?

 

[clcorbin]

My understanding is you can't use the breakout port for stacking without also using the 1/2/1 or 1/2/6 ports in the stack. If I am confused, I would LOVE to do just that.

 

[fohdeesha]

Is it REALLY so bad? I see some good lots on ebay, and I am thinking about buying one.
What I need is: 8 or more 10GBase-T ethernet ports, SFP+ is optional, 40Gbit uplink port would be a big plus. I don't think I need L3 features, just need to connect several hosts with 10G/40G ethernet. I was thinking on getting Mikrotik CRS312-4C+8XG, it lacks QSFP+, but still good enough.

Maybe VDX 6740t is good enough for me? What should I ask seller before buying one (licenses, etc.)?

I can license them, just be aware the copper version (the vdx6740t) draws significant power, it's first gen 10gbase-T PHY, so like a few watts per port. I think they idle around 180w but I don't have any numbers written down from when I had one. also not quiet

 

[klui]

The 6740T idles at 192W with only the console port connected. For comparison, a more modern Arista 7280T idles at 155W with nothing connected. 48-port 10GbaseT switches just require a lot of power.

 

[sean]

The VDX 6740 data sheet says the 6740T has a maximum power usage of 460 W while the SFP+ version, the 6740, has a maximum power usage of 110 W. The nominal airflow of the 6740T is higher than the maximum airflow of the 6740. I don't personally dislike them; it was someone else's budget that purchased them and I don't have to touch the config much.

they're now owned by Extreme which has no public firmware updates

I can offer a story about this. I needed to get a matching firmware for the switches to form a VCS fabric. The support person was great and quickly agreed to get me the right firmware to match what was already on one of the switches. The quickest way to do that was for me to share control of my screen, him log in to his account on Extreme's site on my computer, and point me to the right firmware. I started the download and the support call ended with me still logged in. I had access to a lot of files and considered downloading more. This happened before the existence of this thread or I may have, uh, asked around a bit about it.

 

[clcorbin]

What it took to figure this out:

​

I see you are using a J-Link on that. Are you also able to get the J-Link to work work on a 6610? I ask because I have a J-Link EDU and would like to use it for digging around in the flash on it. Of course, I think I've used the J-Link twice in the last five years, so not my normal go to tool. But if it will work...

 

[Sean Ho]

My understanding is you can't use the breakout port for stacking without also using the 1/2/1 or 1/2/6 ports in the stack. If I am confused, I would LOVE to do just that.

I don't have multiple ICX yet for stacking, but I think each of the four QSFP+ on the back can be independently enabled/disabled for stacking. Stock config after factory reset is that all four are stack, but you can use just the two breakout ports to make a stack ring (if only two switches in a stack, then just one breakout port should be sufficient). I haven't tried this out myself, though.

 

[leon_pro]

I can offer a story about this. I needed to get a matching firmware for the switches to form a VCS fabric. The support person was great and quickly agreed to get me the right firmware to match what was already on one of the switches. The quickest way to do that was for me to share control of my screen, him log in to his account on Extreme's site on my computer, and point me to the right firmware. I started the download and the support call ended with me still logged in. I had access to a lot of files and considered downloading more. This happened before the existence of this thread or I may have, uh, asked around a bit about it.

If you wouldn't mind sharing firmware, I can host it so everyone can download it.

 

[nickf1227]

I can license them, just be aware the copper version (the vdx6740t) draws significant power, it's first gen 10gbase-T PHY, so like a few watts per port. I think they idle around 180w but I don't have any numbers written down from when I had one. also not quiet

I would not buy the copper version for the reason fohdeesha stated. The reason that it draws so much more power is because of an inherent design choice made by Brocade. They took more or less the same board from the SFP+ version and then integrated a daughter board that has 48 SFP+ to RJ45 trancievers on it...each of these will add 1-3 watts of power depending on load. I could be wrong but I don't think it supports EEE

Buy the SFP+ version and only as many copper ports as you need.

 

[leon_pro]

I would not buy the copper version for the reason fohdeesha stated.

Well, I just bought one...
Buying SFP+ version plus trancievers/cables would be more expensive.

If it'll be too loud/hot for me I'll donate it to my friend.

 

[fohdeesha]

I see you are using a J-Link on that. Are you also able to get the J-Link to work work on a 6610? I ask because I have a J-Link EDU and would like to use it for digging around in the flash on it. Of course, I think I've used the J-Link twice in the last five years, so not my normal go to tool. But if it will work...

the 6610 is a totally different architecture (PPC) and requires a PowerPC capable JTAG unit, like the bdi2000

 

[fohdeesha]

If you wouldn't mind sharing firmware, I can host it so everyone can download it.

you *will* get a cease and desist if you publicly host Extreme firmware, ask me how I know. Extreme might be the most obnoxious and litigious of all the network vendors, even when buying half a million dollars in gear, you still do not get blanket firmware access - only access to files for the exact SN / P/N you've registered. they're almost as bad as Ciena. The story from the other user about an engineer providing files without a support contract is literally one in a million

 

[leon_pro]

Extreme might be the most obnoxious and litigious of all the network vendors

That's sad. I thought HP is worst... I was wrong.

Spoiler

Once, on one retro hardware forum people just posted torrent hash. So technically forum's rules were not violated: it's not a link, it's a hex string

 

[dswartz]

That's sad. I thought HP is worst... I was wrong.

Spoiler

Once, on one retro hardware forum people just posted torrent hash. So technically forum's rules were not violated: it's not a link, it's a hex string

Oracle sucks bag too.