Drag to reposition cover

Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

itronin

Well-Known Member
Nov 24, 2018
618
356
63
Denver, Colorado
I ran a UPS test this weekend and my 6610 Rev A PoE power supplies won't run on simulated sine wave power. I have the PSUs plugged into a pair of Cyberpower CP1500AVR units. Anyone here have the same experience? Do Rev B or C run on simulated sine wave?
I've run single and dual 6610 POE's with dual rev a and dual rev b on cyberpower CP1500AVR 's (they are cheap)...
any slight power fluctuation that trips the UPS and the switches would crash and reboot.
My recollection is that the switches were more tolerant of minor power fluctuations off the CP UPS than on

Moved to an Eaton 5PX2200 and the switches are rock solid.
 

NablaSquaredG

Active Member
Aug 17, 2020
281
102
43
It is generally not advisable to run Switching Mode Power Supplies on non-pure sine wave power, especially modern ones with Active PFC. You're basically rolling a dice with that. Might work, might not.
 

Serhan

Member
Sep 22, 2017
56
12
8
49
I ran a UPS test this weekend and my 6610 Rev A PoE power supplies won't run on simulated sine wave power. I have the PSUs plugged into a pair of Cyberpower CP1500AVR units. Anyone here have the same experience? Do Rev B or C run on simulated sine wave?
I have the non-poe version of 6610 with Rev A. It is running fine with an APC BX1500M, which is not sine wave. Am I doing something bad to the switch or the UPS?
 

Rttg

New Member
May 21, 2020
3
3
3
According to the guide, no:


It's a good question, as I'll run into the same issue when I setup my DMZ in the future.
FWIW, the best option for that type of setup (e.g., VyOS, pfsense) is DHCP option 121. You can either explicitly add a route to one different VLAN subnet via your router (e.g. 192.168.1.0/24 via the ve on your Brocade) or to an entire range, depending on the address ranges you’re using (e.g., 192.168.0.0/16 via the ICX).

With your internet-facing router set as the default gateway and on the same subnet, your devices will know it can reach the gateway directly, avoiding any asymmetrical routing issues.
 
  • Like
Reactions: eduncan911

eduncan911

Active Member
Jul 27, 2015
189
93
28
eduncan911.com
FWIW, the best option for that type of setup (e.g., VyOS, pfsense) is DHCP option 121. You can either explicitly add a route to one different VLAN subnet via your router (e.g. 192.168.1.0/24 via the ve on your Brocade) or to an entire range, depending on the address ranges you’re using (e.g., 192.168.0.0/16 via the ICX).

With your internet-facing router set as the default gateway and on the same subnet, your devices will know it can reach the gateway directly, avoiding any asymmetrical routing issues.
I'm trying to figure this out, could you give more information/details/steps to DHCP option 121? I'm trying to find more information about it.

Alternatively, which may not be the best idea as I haven't done it yet, I was thinking I could tag all the VLANs on a trunk that I want DHCP for within the switch, and connect it to the router on another dedicated port. On the router, I could have the DHCP server(s) bind to an IP/bridge within those VLANs which are tagged on that interface.

The idea keeps InterVLAN routing on the switch by keeping the gateway on the switch's `ve`, but would route those DHCP helper-ip requests to the actual DHCP server IP address - on the same VLAN (or perhaps the DHCP server could actually reach the broadcasts of that VLAN).

I don't know if that would work, because the router would have a different default route.
 
  • Like
Reactions: jjacobs

jjacobs

Member
Dec 25, 2020
39
17
8
NC
Default gateway: were do I send my packets
Default route: were do I send some other guys packets and my own packets

So, DHCP option 121 tells some device what to do with other guys packets

*I* like the way this guy explains things, ymmv.

 
Last edited:
  • Like
Reactions: eduncan911

Rttg

New Member
May 21, 2020
3
3
3
I'm trying to figure this out, could you give more information/details/steps to DHCP option 121? I'm trying to find more information about it.
The gist of it can be found here (searching for ‘dhcp option 121‘ or ‘classless static route’ should provide some good add’l detail).

The specific configuration syntax varies by the router running your DHCP server, but the end result is that DHCP clients get pushed an *additional* route (to a VLAN, a set of VLANs, or a specific host - which should be via your L3 switch) along with the default gateway (which in this case would be pfsense, VyOS, etc., for accessing your WAN - or even VLANs you *don’t* designate for routing via the L3 switch).

edit: ninja’d by @jjacobs - good explainer on the difference. tl;dr DHCP option 121 gives you a way to set that gateway and a route without manually configuring individual routes on each client
 

NateS

Member
Apr 19, 2021
35
17
8
Sacramento, CA, US
It is generally not advisable to run Switching Mode Power Supplies on non-pure sine wave power, especially modern ones with Active PFC. You're basically rolling a dice with that. Might work, might not.
Why? Classic switch mode power supplies just rectify to DC as the first step anyway, so the sine wave being perfect shouldn't matter. Generally they'll run fine even with DC input. With active PFC, maybe not though -- I'm less familiar with that topology.

I have the non-poe version of 6610 with Rev A. It is running fine with an APC BX1500M, which is not sine wave. Am I doing something bad to the switch or the UPS?
Interesting. I've got that same UPS and I was planning to throw my 6610 on it. Mine's the PoE version though, so we'll see if that makes a difference. I've got both a rev A and a rev B power supply I can test with.
 

darthray

New Member
Apr 11, 2021
5
1
3
Alright, got my 6610s recently and set them up, licensed them, etc (thanks @fohdeesha!). I would like to stack them if possible (bear with me) but I also want to connect a server and a workstation to one of them, both at 40gbps. Not finding a solution based on what I've seen so far.

My understating is that the breakout ports cannot be used to connect to a desktop at 40Gbps. It also appears that I cannot use the breakout ports for stacking (right?).

The only option that comes to mind is to use 1 breakout port on each (i.e. 4x 10Gbps) and create a LAG over those 4 connections. Not sure if it would work. Could I do that over a MPO/MPO cable (i.e. no actual breakout cable would be used)?

Any other ideas?

Thanks!
 

DavidRa

Infrastructure Architect
Aug 3, 2015
283
128
43
Central Coast of NSW
www.pdconsec.net
I know pfSense can't be used for DHCP when using the Brocade ICX-6610 as a layer 3 switch with VLANs.

Can anyone tell me if VyOS is capable of handling DHCP in a similar setup?
Why not? All you should need to have are:
  • A scope matching the subnet in the far VLAN (specifically, a scope where the subnet includes the IP address of the switch/router in that VLAN)
  • DHCP forwarding (often termed "DHCP Helper") on the switch/router.
This sort of thing has been common for decades - I don't immediately see why it wouldn't work regardless of the switch configuration and firewall?
 

infoMatt

Active Member
Apr 16, 2019
210
90
28
Because its configuration GUI won't let you configure DHCP scopes outside the networks that the firewall manages.

Technically the daemons underneath can do it, but there's no way to configure them accordingly; it's pretty easy however to install a small linux box with a DHCP server and configure as you've said (I've done this in my home network).
 
  • Like
Reactions: DavidRa

DavidRa

Infrastructure Architect
Aug 3, 2015
283
128
43
Central Coast of NSW
www.pdconsec.net
Because its configuration GUI won't let you configure DHCP scopes outside the networks that the firewall manages.

Technically the daemons underneath can do it, but there's no way to configure them accordingly; it's pretty easy however to install a small linux box with a DHCP server and configure as you've said (I've done this in my home network).
Oh my god. Never saw this on pf et al because I run DHCP elsewhere - I've always built multi-VLAN with all DHCP on a single box (or HA pair) and helpers on the switches.
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,117
1,982
113
30
fohdeesha.com
Alright, got my 6610s recently and set them up, licensed them, etc (thanks @fohdeesha!). I would like to stack them if possible (bear with me) but I also want to connect a server and a workstation to one of them, both at 40gbps. Not finding a solution based on what I've seen so far.

My understating is that the breakout ports cannot be used to connect to a desktop at 40Gbps. It also appears that I cannot use the breakout ports for stacking (right?).

The only option that comes to mind is to use 1 breakout port on each (i.e. 4x 10Gbps) and create a LAG over those 4 connections. Not sure if it would work. Could I do that over a MPO/MPO cable (i.e. no actual breakout cable would be used)?

Any other ideas?

Thanks!
you can stack them using one breakout port and one 40gbE port, that way the stack connection is still redundant, but you still have two 40gbE ports and two breakout ports available. In fact, this is exactly my setup at home:

Code:
stack unit 1
  module 1 icx6610-48p-poe-port-management-module
  module 2 icx6610-qsfp-10-port-160g-module
  module 3 icx6610-8-port-10g-dual-mode-module
  no legacy-inline-power
  priority 128
  stack-trunk 1/2/1 to 1/2/2
  stack-port 1/2/1
stack unit 2
  module 1 icx6610-48p-poe-port-management-module
  module 2 icx6610-qsfp-10-port-160g-module
  module 3 icx6610-8-port-10g-dual-mode-module
  no legacy-inline-power
  stack-trunk 2/2/1 to 2/2/2
  stack-port 2/2/1
there's also a link to a post on how to do this in the bottom of the first post in this thread
 
  • Like
Reactions: eduncan911

klui

Active Member
Feb 3, 2019
291
123
43
you can stack them using one breakout port and one 40gbE port ...

stack unit 1
.
.
stack-trunk 1/2/1 to 1/2/2
stack-port 1/2/1

stack unit 2
.
.
stack-trunk 2/2/1 to 2/2/2
stack-port 2/2/1
Are you referring to a linear ring topology with partial cabling per the Stacking Configuration Guide? Fig 13 on pg 56.

I don't understand how the definition is telling the system to use the breakout port. Wouldn't 1/2/1 and 2/2/1 be referring to the 40G ports? I don't have 2 ICXes to do the stacking, but wouldn't it be something like (Trunk 1 40G to 4x10G):

stack unit 1
.
.
stack-trunk 1/2/1 to 1/2/2
stack-port 1/2/1

stack unit 2
.
.
stack-trunk 2/2/1 to 2/2/2 (or 2/2/6 to 2/2/7 - Trunk 2)
stack-port 2/2/2 (or 2/2/7 - Trunk 2)
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,117
1,982
113
30
fohdeesha.com
Are you referring to a linear ring topology with partial cabling per the Stacking Configuration Guide? Fig 13 on pg 56.

I don't understand how the definition is telling the system to use the breakout port. Wouldn't 1/2/1 and 2/2/1 be referring to the 40G ports? I don't have 2 ICXes to do the stacking, but wouldn't it be something like (Trunk 1 40G to 4x10G):

stack unit 1
.
.
stack-trunk 1/2/1 to 1/2/2
stack-port 1/2/1

stack unit 2
.
.
stack-trunk 2/2/1 to 2/2/2 (or 2/2/6 to 2/2/7 - Trunk 2)
stack-port 2/2/2 (or 2/2/7 - Trunk 2)
It's not a ring topology, it's the first 40gbe port and first breakout port in a bond/trunk.

1/2/1 is the first 40gbe port, 1/2/2 is the first port in the first breakout port group. In the stacking section, you only specify the first port in the breakout group when using breakout ports. And just like in the regular fastiron lag/lacp config, when you want to use a lag for something like vlan tagging etc, you specify the first/primary port of the lag only


So "stack-trunk 1/2/1 to 1/2/2" is saying put the first 40gbe port and the first breakout port in a LAG together, and then "stack-port 1/2/1" is saying use that LAG group we just created as a stacking port. In fact those two statements are in the totally default 6610 config, there's just also another one doing the same for the second set of 40gb port + breakout port. My config removed that second statement so that two ports are free for normal use, while still having a two port LAG for redundant stacking. The 40gbe port and the 4 channels of the first breakout port:

Code:
telnet@ICX1>show stack connection
Probing the topology. Please wait ...
  1: 1/2/1 (T0) <---> 2/2/1 (T0)
  2: 1/2/2 (T0) <---> 2/2/2 (T0)
  3: 1/2/3 (T0) <---> 2/2/3 (T0)
  4: 1/2/4 (T0) <---> 2/2/4 (T0)
  5: 1/2/5 (T0) <---> 2/2/5 (T0)
CPU to CPU packets are fine between 2 units.
300 pages in and people still doubtin me on these sheeeeeeeeet
 
  • Like
Reactions: infoMatt

fohdeesha

Kaini Industries
Nov 20, 2016
2,117
1,982
113
30
fohdeesha.com
Oh my god. Never saw this on pf et al because I run DHCP elsewhere - I've always built multi-VLAN with all DHCP on a single box (or HA pair) and helpers on the switches.
I agree, it's embarrassing to the point of disbelief, given dhcp for multiple scopes is a spec older than I am. There's a big github request years old for it on pfsense, I even offered the devs to pay. No response. As we found out recently, they were probably too busy paying sketchball felons to write horrifically insecure freebsd code. They nuked the issue ages ago but there's a 5 year old ticket filed with opnsense as well, offered money there too, no result again Feature: DHCP server able to handle non-interface configured subnets · Issue #1105 · opnsense/plugins
 

klui

Active Member
Feb 3, 2019
291
123
43
300 pages in and people still doubtin me on these sheeeeeeeeet
Haha. I would never doubt you! ;)

I want to understand how the def'n works and
"stack-port 1/2/1" is saying use that LAG group we just created as a stacking port.
The documentation states that stack-port "selects only one of the two stacking ports as a stacking port, which allows you to use the other port as a data port." So I thought stack-trunk 1/2/1 to 1/2/2 means the stack consists of the 40G and breakout 40G port and stack-port 1/2/1 means use the 40G as the stacking port. Then on stack unit 2, if I want to use the breakout 40G, I would reference 2/2/2 using stack-port. Basically connect between stack unit 1, port XL1 (1/2/1) and stack unit 2, port XL2-5 (1/2/2).

I feel it would really help if you could clarify on how cables are connected between stack units 1 and 2 based on the configuration you gave.

Thanks!
 

NablaSquaredG

Active Member
Aug 17, 2020
281
102
43
So - Just to be sure:
An ICX6610 has four QSFP+ ports on the rear. Two of them are regular 40GBe ports, but the other two are special.

They can either be used as stacking ports or as QSFP+->4x10G SFP+ breakout ports, but not as regular 40GBe Ports, right?