Running out of 10g ports on Brocade ICX 6610, what to do?

[kapone]

Well, guys, this lil switch has served me well for a while, but I'm in a quandary at this point.

This is the Brocade ICX 6610-48-E (non POE). It has eight 1g/10g SFP+ ports (and they are all licensed for 10g) and 48 1G RJ45 ports. It also has four 40g QSFP+ ports on the back, but these are supposedly only for stacking, and I haven't used them until now, nor do I foresee using them. The switch also has the "Premium" license for Layer 3, which allows it to do all the fancy stuff, including static routes and what not.

The problem is that I'm running out of the 10g ports. I need to connect more stuff to it than there are ports. Up until now, this was THE perfect switch for my home/lab, as it had enough 10G and 1G ports for everything in one box, that's not too loud or too power inefficient. I'm actually not even using that many RJ45s on it, I just need more 10G ports.

What would you do?

- Get a different "core" switch and hang this one off of that,
- get a different "edge" switch that has more 10g ports and hang it off of this one,
- sell this one, get a different switch altogether which will satisfy the growing needs?

 

[pyro_]

I would just pick up an additional 10G switch and hang it off of that one if it is doing everything that you want and you are happy with it

 

[fohdeesha]

you could always buy an lb6m from unixplus using the STH discount code, flash it to brocade, and have the exact same CLI/OS you're already familiar with since they're both brocade you could also do things like stick them in an MRP ring for very fast l2 failover

 

[kapone]

you could always buy an lb6m from unixplus using the STH discount code, flash it to brocade, and have the exact same CLI/OS you're already familiar with since they're both brocade you could also do things like stick them in an MRP ring for very fast l2 failover

That was the FIRST thought (minus the MRP ring part...which is an interesting thought...) I had in the last few days, especially after seeing your work on the cross-flash.

What's stopping me is the LB6M's power consumption... it is just too high for 24x7 use.

 

[fohdeesha]

Have you measured your ICX's draw? I'm not familiar with the 6610 but I know the older ICX's were around 90 watts, which is only 20 watts lower than I've measured on my lb6m. if you could totally replace it you might not come out too bad, who knows

 

[kapone]

Have you measured your ICX's draw? I'm not familiar with the 6610 but I know the older ICX's were around 90 watts, which is only 20 watts lower than I've measured on my lb6m. if you could totally replace it you might not come out too bad, who knows

I think the last time I measured the 6610's power draw, it was ~80w, but up until now, this was the only switch in my rack, and handled both 10g and 1g.

If I add a LB6M to this...that's gonna be ~200w just for the switches!

Edit: Meant to imply that the LB6M can't replace this switch...I still need a few 1G ports, and the LB6M doesn't have that many of them.

 

[fohdeesha]

how many 1gbe ports do you need? it has 4x copper built in, then you can use the sfp+ ports as 1gbe copper ports with rj45 sfp's, if you don't need too many of them then you would only need the 1 switch

 

[kapone]

how many 1gbe ports do you need? it has 4x copper built in, then you can use the sfp+ ports as 1gbe copper ports with rj45 sfp's, if you don't need too many of them then you would only need the 1 switch

I'd estimate I need ~24 1G ports. I have about 16 in use right now, but I expect that number to go up. (web host cluster, which will use 1G on the web side, and 10G on the storage/DB side)

 

[fohdeesha]

oooh, yeah it looks like you're gunna need to either throw down some cash or build a nuke plant in your backyard

 

[kapone]

oooh, yeah it looks like you're gunna need to either throw down some cash or build a nuke plant in your backyard

Hmm...I must have missed this news....

Portable Backyard Nuclear Reactors Ready to be Installed by 2013

 

[StammesOpfer]

Have you looked into using those 40G QSPF ports with breakout cables for 4x 10G each. Don't know if they prevent this from working on these switches.

 

[kapone]

Have you looked into using those 40G QSPF ports with breakout cables for 4x 10G each. Don't know if they prevent this from working on these switches.

I think I mentioned this earlier. On the 6610s, the 40g ports in the back are "dedicated" for stacking. Interestingly, the stacking ports are divided into two groups. Two of those ports DO support breaking into 4x 10g ports (the other two don't), but they can't be used as "data" ports. The switching software will not use them as data ports.

If I were able to do this... I could have another eight 10g ports...and that would solve the problem! I haven't found a way to make them work with breakout cables AND connect a device to them using those breakout cables. If anybody knows how...I'll buy them a beer!

 

[fohdeesha]

yeah, they're software locked for stacking only. An engineer friend has a 6610 and he's had a side project of disassembling the fastiron image to remove the stacking limit on all 40gbe ports, and he's made some progress, but it's not something I would count on. The image is encrypted and last time I talked to him the only way he said he's able to get the key is to catch it over the wire in flight on boot by sniffing the powerpc bus, which he hasn't done succesfully yet. such a stupid decision to arbitrarily restrict them

 

[pyro_]

What kind of budget do you have and how many 10g ports do you need

 

[kapone]

What kind of budget do you have and how many 10g ports do you need

It's not so much the budget, but about additional switch/replace switch decision.

This switch could easily fetch $1K or more used (remember, 8 licenses for the 10g ports, and the premium layer 3 license), which could fund a replacement if need be.

 

[fractal]

Why not just get another of what you have and stack them?

 

[kapone]

Why not just get another of what you have and stack them?

I have considered that, but these are relatively hard to find with the 10G licenses installed. Without the licenses, it's not worth it, because the resellers sell the license(s) for ~$1,500.

 

[pyro_]

How many 10g ports total do you need at the end of the day

 

[kapone]

How many 10g ports total do you need at the end of the day

I'd say ~16 should get me by for the foreseeable future. More is good, but if that means twice the power consumption...uh oh.

I've thought about these new Mikrotik 10G switches as well, and while they are great from a power consumption perspective, the CPU on these things is slower than the one in my washing machine. For any layer 3 stuff, they just don't do well.

 

[fohdeesha]

I have considered that, but these are relatively hard to find with the 10G licenses installed. Without the licenses, it's not worth it, because the resellers sell the license(s) for ~$1,500.

it would also add as much power draw as just chucking on an lb6m