Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[eduncan911]

FWIW, the best option for that type of setup (e.g., VyOS, pfsense) is DHCP option 121. You can either explicitly add a route to one different VLAN subnet via your router (e.g. 192.168.1.0/24 via the ve on your Brocade) or to an entire range, depending on the address ranges you’re using (e.g., 192.168.0.0/16 via the ICX).

With your internet-facing router set as the default gateway and on the same subnet, your devices will know it can reach the gateway directly, avoiding any asymmetrical routing issues.

I'm trying to figure this out, could you give more information/details/steps to DHCP option 121? I'm trying to find more information about it.

Alternatively, which may not be the best idea as I haven't done it yet, I was thinking I could tag all the VLANs on a trunk that I want DHCP for within the switch, and connect it to the router on another dedicated port. On the router, I could have the DHCP server(s) bind to an IP/bridge within those VLANs which are tagged on that interface.

The idea keeps InterVLAN routing on the switch by keeping the gateway on the switch's `ve`, but would route those DHCP helper-ip requests to the actual DHCP server IP address - on the same VLAN (or perhaps the DHCP server could actually reach the broadcasts of that VLAN).

I don't know if that would work, because the router would have a different default route.

 

[jjacobs]

Default gateway: were do I send my packets
Default route: were do I send some other guys packets and my own packets

So, DHCP option 121 tells some device what to do with other guys packets

*I* like the way this guy explains things, ymmv.

 

[Rttg]

I'm trying to figure this out, could you give more information/details/steps to DHCP option 121? I'm trying to find more information about it.

The gist of it can be found here (searching for ‘dhcp option 121‘ or ‘classless static route’ should provide some good add’l detail).

The specific configuration syntax varies by the router running your DHCP server, but the end result is that DHCP clients get pushed an *additional* route (to a VLAN, a set of VLANs, or a specific host - which should be via your L3 switch) along with the default gateway (which in this case would be pfsense, VyOS, etc., for accessing your WAN - or even VLANs you *don’t* designate for routing via the L3 switch).

edit: ninja’d by @jjacobs - good explainer on the difference. tl;dr DHCP option 121 gives you a way to set that gateway and a route without manually configuring individual routes on each client

 

[NateS]

It is generally not advisable to run Switching Mode Power Supplies on non-pure sine wave power, especially modern ones with Active PFC. You're basically rolling a dice with that. Might work, might not.

Why? Classic switch mode power supplies just rectify to DC as the first step anyway, so the sine wave being perfect shouldn't matter. Generally they'll run fine even with DC input. With active PFC, maybe not though -- I'm less familiar with that topology.

I have the non-poe version of 6610 with Rev A. It is running fine with an APC BX1500M, which is not sine wave. Am I doing something bad to the switch or the UPS?

Interesting. I've got that same UPS and I was planning to throw my 6610 on it. Mine's the PoE version though, so we'll see if that makes a difference. I've got both a rev A and a rev B power supply I can test with.

 

[darthray]

Alright, got my 6610s recently and set them up, licensed them, etc (thanks @fohdeesha!). I would like to stack them if possible (bear with me) but I also want to connect a server and a workstation to one of them, both at 40gbps. Not finding a solution based on what I've seen so far.

My understating is that the breakout ports cannot be used to connect to a desktop at 40Gbps. It also appears that I cannot use the breakout ports for stacking (right?).

The only option that comes to mind is to use 1 breakout port on each (i.e. 4x 10Gbps) and create a LAG over those 4 connections. Not sure if it would work. Could I do that over a MPO/MPO cable (i.e. no actual breakout cable would be used)?

Any other ideas?

Thanks!

 

[infoMatt]

Any other ideas?

Stack 'em using two other ports. If memory serves me, you'll have to use two (or more) ports at the front, you can't use two of the 40G breakout.

 

[DavidRa]

I know pfSense can't be used for DHCP when using the Brocade ICX-6610 as a layer 3 switch with VLANs.

Can anyone tell me if VyOS is capable of handling DHCP in a similar setup?

Why not? All you should need to have are:

• A scope matching the subnet in the far VLAN (specifically, a scope where the subnet includes the IP address of the switch/router in that VLAN)
• DHCP forwarding (often termed "DHCP Helper") on the switch/router.

This sort of thing has been common for decades - I don't immediately see why it wouldn't work regardless of the switch configuration and firewall?

 

[infoMatt]

Why not?

Because its configuration GUI won't let you configure DHCP scopes outside the networks that the firewall manages.

Technically the daemons underneath can do it, but there's no way to configure them accordingly; it's pretty easy however to install a small linux box with a DHCP server and configure as you've said (I've done this in my home network).

 

[DavidRa]

Because its configuration GUI won't let you configure DHCP scopes outside the networks that the firewall manages.

Technically the daemons underneath can do it, but there's no way to configure them accordingly; it's pretty easy however to install a small linux box with a DHCP server and configure as you've said (I've done this in my home network).

Oh my god. Never saw this on pf et al because I run DHCP elsewhere - I've always built multi-VLAN with all DHCP on a single box (or HA pair) and helpers on the switches.

 

[nerdalertdk]

Oh my god. Never saw this on pf et al because I run DHCP elsewhere - I've always built multi-VLAN with all DHCP on a single box (or HA pair) and helpers on the switches.

Vyos works Perfect in this setup

 

[fohdeesha]

Alright, got my 6610s recently and set them up, licensed them, etc (thanks @fohdeesha!). I would like to stack them if possible (bear with me) but I also want to connect a server and a workstation to one of them, both at 40gbps. Not finding a solution based on what I've seen so far.

My understating is that the breakout ports cannot be used to connect to a desktop at 40Gbps. It also appears that I cannot use the breakout ports for stacking (right?).

The only option that comes to mind is to use 1 breakout port on each (i.e. 4x 10Gbps) and create a LAG over those 4 connections. Not sure if it would work. Could I do that over a MPO/MPO cable (i.e. no actual breakout cable would be used)?

Any other ideas?

Thanks!

you can stack them using one breakout port and one 40gbE port, that way the stack connection is still redundant, but you still have two 40gbE ports and two breakout ports available. In fact, this is exactly my setup at home:

stack unit 1
  module 1 icx6610-48p-poe-port-management-module
  module 2 icx6610-qsfp-10-port-160g-module
  module 3 icx6610-8-port-10g-dual-mode-module
  no legacy-inline-power
  priority 128
  stack-trunk 1/2/1 to 1/2/2
  stack-port 1/2/1
stack unit 2
  module 1 icx6610-48p-poe-port-management-module
  module 2 icx6610-qsfp-10-port-160g-module
  module 3 icx6610-8-port-10g-dual-mode-module
  no legacy-inline-power
  stack-trunk 2/2/1 to 2/2/2
  stack-port 2/2/1

there's also a link to a post on how to do this in the bottom of the first post in this thread

 

[klui]

you can stack them using one breakout port and one 40gbE port ...

stack unit 1
.
.
stack-trunk 1/2/1 to 1/2/2
stack-port 1/2/1

stack unit 2
.
.
stack-trunk 2/2/1 to 2/2/2
stack-port 2/2/1

Are you referring to a linear ring topology with partial cabling per the Stacking Configuration Guide? Fig 13 on pg 56.

I don't understand how the definition is telling the system to use the breakout port. Wouldn't 1/2/1 and 2/2/1 be referring to the 40G ports? I don't have 2 ICXes to do the stacking, but wouldn't it be something like (Trunk 1 40G to 4x10G):

stack unit 1
.
.
stack-trunk 1/2/1 to 1/2/2
stack-port 1/2/1

stack unit 2
.
.
stack-trunk 2/2/1 to 2/2/2 (or 2/2/6 to 2/2/7 - Trunk 2)
stack-port 2/2/2 (or 2/2/7 - Trunk 2)

 

[fohdeesha]

Are you referring to a linear ring topology with partial cabling per the Stacking Configuration Guide? Fig 13 on pg 56.

I don't understand how the definition is telling the system to use the breakout port. Wouldn't 1/2/1 and 2/2/1 be referring to the 40G ports? I don't have 2 ICXes to do the stacking, but wouldn't it be something like (Trunk 1 40G to 4x10G):

stack unit 1
.
.
stack-trunk 1/2/1 to 1/2/2
stack-port 1/2/1

stack unit 2
.
.
stack-trunk 2/2/1 to 2/2/2 (or 2/2/6 to 2/2/7 - Trunk 2)
stack-port 2/2/2 (or 2/2/7 - Trunk 2)

It's not a ring topology, it's the first 40gbe port and first breakout port in a bond/trunk.

1/2/1 is the first 40gbe port, 1/2/2 is the first port in the first breakout port group. In the stacking section, you only specify the first port in the breakout group when using breakout ports. And just like in the regular fastiron lag/lacp config, when you want to use a lag for something like vlan tagging etc, you specify the first/primary port of the lag only


So "stack-trunk 1/2/1 to 1/2/2" is saying put the first 40gbe port and the first breakout port in a LAG together, and then "stack-port 1/2/1" is saying use that LAG group we just created as a stacking port. In fact those two statements are in the totally default 6610 config, there's just also another one doing the same for the second set of 40gb port + breakout port. My config removed that second statement so that two ports are free for normal use, while still having a two port LAG for redundant stacking. The 40gbe port and the 4 channels of the first breakout port:

telnet@ICX1>show stack connection
Probing the topology. Please wait ...
  1: 1/2/1 (T0) <---> 2/2/1 (T0)
  2: 1/2/2 (T0) <---> 2/2/2 (T0)
  3: 1/2/3 (T0) <---> 2/2/3 (T0)
  4: 1/2/4 (T0) <---> 2/2/4 (T0)
  5: 1/2/5 (T0) <---> 2/2/5 (T0)
CPU to CPU packets are fine between 2 units.

300 pages in and people still doubtin me on these sheeeeeeeeet

 

[fohdeesha]

Oh my god. Never saw this on pf et al because I run DHCP elsewhere - I've always built multi-VLAN with all DHCP on a single box (or HA pair) and helpers on the switches.

I agree, it's embarrassing to the point of disbelief, given dhcp for multiple scopes is a spec older than I am. There's a big github request years old for it on pfsense, I even offered the devs to pay. No response. As we found out recently, they were probably too busy paying sketchball felons to write horrifically insecure freebsd code. They nuked the issue ages ago but there's a 5 year old ticket filed with opnsense as well, offered money there too, no result again Feature: DHCP server able to handle non-interface configured subnets · Issue #1105 · opnsense/plugins

 

[klui]

300 pages in and people still doubtin me on these sheeeeeeeeet

Haha. I would never doubt you!

I want to understand how the def'n works and

"stack-port 1/2/1" is saying use that LAG group we just created as a stacking port.

The documentation states that stack-port "selects only one of the two stacking ports as a stacking port, which allows you to use the other port as a data port." So I thought stack-trunk 1/2/1 to 1/2/2 means the stack consists of the 40G and breakout 40G port and stack-port 1/2/1 means use the 40G as the stacking port. Then on stack unit 2, if I want to use the breakout 40G, I would reference 2/2/2 using stack-port. Basically connect between stack unit 1, port XL1 (1/2/1) and stack unit 2, port XL2-5 (1/2/2).

I feel it would really help if you could clarify on how cables are connected between stack units 1 and 2 based on the configuration you gave.

Thanks!

 

[NablaSquaredG]

So - Just to be sure:
An ICX6610 has four QSFP+ ports on the rear. Two of them are regular 40GBe ports, but the other two are special.

They can either be used as stacking ports or as QSFP+->4x10G SFP+ breakout ports, but not as regular 40GBe Ports, right?

 

[fohdeesha]

Haha. I would never doubt you!

I want to understand how the def'n works and


The documentation states that stack-port "selects only one of the two stacking ports as a stacking port, which allows you to use the other port as a data port." So I thought stack-trunk 1/2/1 to 1/2/2 means the stack consists of the 40G and breakout 40G port and stack-port 1/2/1 means use the 40G as the stacking port. Then on stack unit 2, if I want to use the breakout 40G, I would reference 2/2/2 using stack-port. Basically connect between stack unit 1, port XL1 (1/2/1) and stack unit 2, port XL2-5 (1/2/2).

I feel it would really help if you could clarify on how cables are connected between stack units 1 and 2 based on the configuration you gave.

Thanks!

You must be reading stacking information regarding a different model, as the ICX6610 has 4 stacking ports, not 2, and they definitely don't talk about using them for data - in fact it explicitly states they *can't* be used as data ports ever (thankfully this *feature* was snuck in in 8010 if I recall).

The top two ports on each switch are connected to each other for a redundant stacking connection in my setup. One 40gbE port, one breakout port. this leaves one free 40gbe port and one free breakout port per switch for me:

 

[fohdeesha]

So - Just to be sure:
An ICX6610 has four QSFP+ ports on the rear. Two of them are regular 40GBe ports, but the other two are special.

They can either be used as stacking ports or as QSFP+->4x10G SFP+ breakout ports, but not as regular 40GBe Ports, right?

check the third reply to this forum thread, it has a diagram, and explains it (at least what I thought) was pretty clearly. two are 40gbE only, two are 4x 10gbE breakout only. any of the ports can be used for regular ethernet ports, or stacking ports

 

[NablaSquaredG]

check the third reply to this forum thread, it has a diagram, and explains it (at least what I thought) was pretty clearly. two are 40gbE only, two are 4x 10gbE breakout only. any of the ports can be used for regular ethernet ports, or stacking ports

So yeah, basically what I said with the exception that the two regular 40GBe ports can also be used for stacking.

I'd like to use all four ports as uplink ports to other, non ICX6610 switches - I guess there is no way to build a reverse breakout cable?
ICX 6610 -> 40G QSFP+ to 4x10G Breakout -> 4x10G SFP+ to 40G QSFP+ Reverse Breakout -> QSFP+ port on another switch (not ICX6610)

I suppose replacing the Breakout - Reverse Breakout combination with a single QSFP+ cable wouldn't work either?

 

[fohdeesha]

So yeah, basically what I said with the exception that the two regular 40GBe ports can also be used for stacking.

I'd like to use all four ports as uplink ports to other, non ICX6610 switches - I guess there is no way to build a reverse breakout cable?
ICX 6610 -> 40G QSFP+ to 4x10G Breakout -> 4x10G SFP+ to 40G QSFP+ Reverse Breakout -> QSFP+ port on another switch (not ICX6610)

I suppose replacing the Breakout - Reverse Breakout combination with a single QSFP+ cable wouldn't work either?

You can absolutely use a single regular qsfp cable to connect the breakout port to another switch (assuming the other switches port is configured for breakout / 4x 10gbe). You can even connect a breakout port with a qsfp cable to a non breakout port on something, but only one lane will link up so it will run at 10gbps

Both 40gbe and 4x 10gbe breakout qsfp+ are electrically identical, it's 4x 10gbps lanes. Difference is in how the ASIC/PHY decides to split them (or mux them)