Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[ViciousXUSMC]

I guess I'm lucky I bought 2 6450-24s this week for $99/ea

Jealous, thats the model I wanted and less than I paid

The seller I bought the ICX 6011 from emailed me and tried to throw in the 6050-48P @175 w/ Free Ship.
I talked them down to $150 based on the price on the first post here so figured it was at least not a bad deal.

I did want the 24P simply due to it using less power. For 24/7 operation it adds up and I do not need more than 24 ports.

 

[dlaffin1012]

Well I would like 48 ports at least but don't really need more than that. So is it better to get two 6610-24p's and stack them or get one 6610-48p? I don't use all 48 ports but I have some more wiring to do so I may surpass the 24 ports. The benefit to two 24's is I get more 10G ports and redundancy. Maybe i will go with two 24's, sounds like I get the redundancy and 48 ports.

When you stack switches, how do you guys determine which switch to plug into. Other than LACP setups which would be both, do you start with the master switch and then proceed to the other one as the first gets full?

 

[ViciousXUSMC]

For pfSense to be the DHCP server for "anything", it needs to be an interface in pfSense.

The problem is, if you're using a layer 3 switch then your VLANs are defined at the switch level and you want the switch to do the routing, not pfSense.

And if you define the interface at the pfSense level, your layer 3 switch essentially becomes layer 2 for that VLAN. Not ideal.

And pfSense DHCP is not "VLAN aware", so you can't use the "ip-helper" command at the switch to point it to the pfSense IP. Not ideal.

The logical solution is to use a dedicated DHCP server (that is VLAN aware), that is outside pfSense and the switch, and then use the ip-helper commands to point to it.

So just wanted to key in on my "hybrid" solution.

Rather than do what was mentioned a page or so back from this post quoted above where you use the switch for inter-vlan routing and have a separate subnet going to PFSense where PFSense can not see any of your production VLANs only that one segment.

Or the alternative of having a trunk to PFSense and having it do all the routing thus "waisting" a layer 3 switch as somebody else mentioned.

My hybrid approach is this.

VLAN1 - Production Data with DHCP
VLAN10 - Production Data w/o DHCP

VLAN 1 has all my client devices (phones, streaming devices, etc) and VLAN10 has my servers.
I do not want Servers on DHCP anyways so PFSense is actually the default gateway for the switch but in the subnet of VLAN1 so it CAN be the DHCP server for that subnet, the gateway for the devices in VLAN1 is a virtual interface on the switch and same for VLAN10 so all the devices use the switch for the default gateway and the switch for routing, but then the switch itself sends everything to PFSense only for WAN access as the gateway of last resort.

So all local communication is via the switch, PFSense is only use for WAN/Internet but still gets to be my DHCP and DNS server.

I just like doing all my static DHCP leases from PFSense too much to lose that, and I hate wasting a fast Layer 3 switch having PFSense do my routing.

Expounded by the fact my VLAN10 is Jumbo Frames, I trust the switch to handle the segmentation easily while PFSense may be a bit slower.

 

[Zervun]

Well I would like 48 ports at least but don't really need more than that. So is it better to get two 6610-24p's and stack them or get one 6610-48p? I don't use all 48 ports but I have some more wiring to do so I may surpass the 24 ports. The benefit to two 24's is I get more 10G ports and redundancy. Maybe i will go with two 24's, sounds like I get the redundancy and 48 ports.

When you stack switches, how do you guys determine which switch to plug into. Other than LACP setups which would be both, do you start with the master switch and then proceed to the other one as the first gets full?

You configure the stack ports within the OS, you can pick which of the 40g ports to use on the back.

The 6610-24p take a little less juice than the 48p but if you have 2x power supplies in them you are going to have the same noise. I supposed you could run just one PS in each of the two. 2x 24p vs. 1x 48p are going to take more watts though.

You can always get 2x 6610s and keep one as a cold spare/powered off to swap in if the primary goes down instead of having it on all of the time. You won't have as many 10g/40g ports as 2x 24p's though. These are enterprise grade switches which are designed to run 24/7 so the odds of one going out is probably fairly slim. They have redundant power supplies and fans which can be swapped out if they go bad.

If it isn't mission critical I wouldn't worry about dual homing, LACP or any of that. If you have a spare 6610-48p (or whatever) powered off it is only going to take you a little bit of time to swap cables and power it on. The 2x 24p's would give you more redundancy assuming you dual home everything, but then again you are doubling your port requirements than single homing.

 

[exvarkin]

So I decided to join in on the fun and ordered a ICX 6610-24-P. First real experience for me with higher end switch gear other then a old gigabit dell. The plan for me is to network my FreeNAS and Proxmox box via the 40gbE ports. In my understanding, I will require a license to be able accomplish this, correct? (need to PM fohdeesha and hope for the best) For now back to search for some NICs and cables.

 

[Zervun]

So I decided to join in on the fun and ordered a ICX 6610-24-P. First real experience for me with higher end switch gear other then a old gigabit dell. The plan for me is to network my FreeNAS and Proxmox box via the 40gbE ports. In my understanding, I will require a license to be able accomplish this, correct? (need to PM fohdeesha and hope for the best) For now back to search for some NICs and cables.

Yes you will need a license for the ports, they are not licensed by default. For 2x 40gb connections you can remove the stack from them and use those. HP infiband cards can be flashed to Mellanox connect-3's (another thread on here) for 40g. Then get 40gb DAC cables. The stack can still be configured on the 40gb breakout ports if needed.

 

[exvarkin]

Yes you will need a license for the ports, they are not licensed by default. For 2x 40gb connections you can remove the stack from them and use those. HP infiband cards can be flashed to Mellanox connect-3's (another thread on here) for 40g. Then get 40gb DAC cables. The stack can still be configured on the 40gb breakout ports if needed.

I've been reading up on the Mellanox connect-3 option and its what my current plan is. Thanks for the confirmation on the license aspect.

Might be a bit of a beginner question, but is there any benefit of DAC over fiber other then cost saving if it all in close proximity.

 

[Zervun]

I've been reading up on the Mellanox connect-3 option and its what my current plan is. Thanks for the confirmation on the license aspect.

Might be a bit of a beginner question, but is there any benefit of DAC over fiber other then cost saving if it all in close proximity.

Ease of use/cost if it is short distance. Fiber of course has no issue with crosstalk etc. Can find an abundance of DAC cables around if you are going only a short difference. I was going to go the fiber route but ended up going DAC (everything is in the same rack) due to the simplicity for the 40gb connections anyway. Might due fiber for the 10g connection as the SFPs are really cheap for them.

 

[tjk]

Anyone know if you can do hitless OS stack upgrades, assuming all connections are LACP'd across different stack members?

I read the upgrade docs, but it made it sound like a code upgrade was an out of service event.

Thanks

 

[fohdeesha]

Anyone know if you can do hitless OS stack upgrades, assuming all connections are LACP'd across different stack members?

I read the upgrade docs, but it made it sound like a code upgrade was an out of service event.

Thanks

on the icx6xxx series no, on the icx7xxx series yes

 

[jmacsob]

Wondering if the ICX7250 fans are easy to swap for quieter ones like the ICX6450's are? Or if they will refuse to boot like the 6610, anyone have any experience with this?

 

[fohdeesha]

Wondering if the ICX7250 fans are easy to swap for quieter ones like the ICX6450's are? Or if they will refuse to boot like the 6610, anyone have any experience with this?

they can be replaced without it complaining

 

[jmacsob]

Thanks, that opens up some options for me then.

 

[tjk]

on the icx6xxx series no, on the icx7xxx series yes

As always, thanks @fohdeesha.

Another questions, can I change the hash type on the lags? I don't see it in the docs nor as an options in the config.

The remote side of this lag is a dell, and I have the following hash options on the dell side, I want to make sure I match the dell side to the 6610 side.

#show interfaces port-channel 48

Channel   Ports                         Hash Algorithm Type
-------   ----------------------------- -------------------
ch48      Active: 1/g35, 1/g36,         3
          1/g37, 1/g38, 2/g35, 2/g36,   
          2/g37, 2/g38                 

Hash Algorithm Type
1 - Source MAC, VLAN, EtherType, source module and port Id
2 - Destination MAC, VLAN, EtherType, source module and port Id
3 - Source IP and source TCP/UDP port
4 - Destination IP and destination TCP/UDP port
5 - Source/Destination MAC, VLAN, EtherType, source MODID/port
6 - Source/Destination IP and source/destination TCP/UDP port

 

[tjk]

Man, these things draw a boat load of power...

I've got a stack of 6 of these running in my dc, nothing plugged into them (yet) on the ethernet side, and they are drawing a total of 6 amps on a 208v/30a circuit. That's over 210+ watts idle per unit.

Ouch.

 

[fohdeesha]

Man, these things draw a boat load of power...

I've got a stack of 6 of these running in my dc, nothing plugged into them (yet) on the ethernet side, and they are drawing a total of 6 amps on a 208v/30a circuit. That's over 210+ watts idle per unit.

Ouch.

there's something seriously wrong with your metering or your switches, I've never seen them draw more than ~120w even under heavy load (assuming we're talking about 6610's)

 

[tjk]

Yes, these are the 6610's, and I don't think there is anything wrong with the APC reporting #'s nor what the DC is monitoring, they both agree on draw...just did a snapshot, 2.5A on each circuit being pulled, 208v/30a circuit, nothing but 6 x 6610's plugged in, 6 per circuit.

5a * 208v = 1040 watts / 6 = 173 watts each unit, unless my math is wrong somewhere? I've seen as high as 3 amps on each circuit being pulled, and again, nothing else is plugged into these apc's yet.

 

[fohdeesha]

It's possible they just have a poor power factor, eg high amps at low voltage, but I can 100% promise they're not drawing anywhere near 200 watts (granted that doesn't help you when you're getting billed based on current draw). I've metered a few on a 0.5% accuracy wattnode before making this thread (some pics on page uhh 5 if I remember right), and the highest I ever saw was 115w with PoE on

 

[tjk]

Good point on the power factor, didn't take that into the equation.

Since I have your attention - any idea on the hash settings for lags?

 

[juey]

Yes you will need a license for the ports, they are not licensed by default. For 2x 40gb connections you can remove the stack from them and use those. HP infiband cards can be flashed to Mellanox connect-3's (another thread on here) for 40g. Then get 40gb DAC cables. The stack can still be configured on the 40gb breakout ports if needed.

The QSFP+ ports in the back do not require any license, the front SFP+ ports need 2 licences in order to use them with 10G.