Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[servethehomefan]

@kapone Can you expand a bit further or maybe give some subtle hints on a basic inter-VLAN routing using only Brocade switches? I am going to soon be setting up an OPNsense firewall along with two 6450's (stacked). I plan on having 5 VLANs (management, IOT, cameras, trust, guest) and I am so use to the UI of OPNsense firewall that I planned to use it for all of the firewall rules (as you know, LAN type rules in pfSense/OPNsense only controls outbound traffic). So for example, all VLANs can access cameras VLAN but cameras cannot access anyone including internet. Only management VLAN can access OPNsense UI/ssh. Guest has only access to internet with no port blocking, etc.

I'm also familiar familiar wtih OPNsense firewall DHCP settings, statuses, reservations, etc. If I don't use the OPNsense for DHCP and only for firewall rules as it relates to the internet that would be great but it seems like I'll be having to go to two locations to truly understand what is going on (brocade and firewall).

I have a very beefy OPNsense box that is overkill for the home environment so I figured I wouldn't get much degregation, but it will certainly be less efficient than keeping everything at the L3 level. I just don't have the depth of knowledge of Brocade equipment and L3 in general to feel confident in my CLI-fu skills to set it up right so any direction is greatly appreciated.

 

[kpfleming]

@kapone Can you expand a bit further or maybe give some subtle hints on a basic inter-VLAN routing using only Brocade switches? I am going to soon be setting up an OPNsense firewall along with two 6450's (stacked). I plan on having 5 VLANs (management, IOT, cameras, trust, guest) and I am so use to the UI of OPNsense firewall that I planned to use it for all of the firewall rules (as you know, LAN type rules in pfSense/OPNsense only controls outbound traffic). So for example, all VLANs can access cameras VLAN but cameras cannot access anyone including internet. I'm also familiar with OPNsense firewall DHCP settings, statuses, reservations, etc.

I would recommend *not* going this direction; I tried, gave up, and went back to VLAN routing (and necessary filtering) in the attached router instead of the ICX. The ICX's ACL capabilities are quite limited and difficult to use, compared pretty much anything like OPNsense, pfSense, Vyatta, or even a Linux firewall manager.

In particular, the ICX's ACL system has no concept of 'connection tracking', so I could not find a practical way to permit asymmetric connections (VLAN 10 can connect to VLAN 20, but VLAN 20 cannot connect to VLAN 10).

 

[sultan.of.swing]

I would just do all routing with opnsense/firewall rules and not in the switch.

 

[ngr2001]

I am not sure why everyone thinks I have VLAN routing going on. This network is a very flat single default 0 vlan setup. PFSense for the most part is in a basic stock configuration, again I do not have any VLANs or VLAN routing going on in either PFSense or the ICX. PFSense is providing DHCP and DNS to all the clients on the ICX switch. All 48 Ports of the ICX are on the same default vlan.

I am 100% sure both the LAN NIC in PFSense and the SFP+ port on the ICX have properly negotiated 10Gb, I did install the magic Key on the ICX to get all 8(x) SFP+ 10Gb ports working.

The performance issue I am seeing with a very basic clean ICX configuration is that when clients on 1Gb ports perform a long sustained speed test or high speed download their speed begins to taper down pretty fast. They will start at 900Mbps and taper down to 500Mpbs like clockwork. If I physically move that uplink from 10Gb to 1Gb all my clients at 1Gb can sustain speed tests and downloads at 900Mbps with no loss in speed.

This is definitely not a PFSense issue being that the performance changes by changing the uplink port speed. Also If I move all my wiring over to my old Cisco 3650 I have 0 issues, but again bone stock the 3650 had the same exact issue, but it was easily fixed with that one global command which to my understanding allows ports when needed to consume a larger memory buffer. Prior to fixing this same issue in the Cisco world I was looking for fixes on the PFSense side, this is when one of their top engineers informed me my switch was at fault. I am way more comfortable with Cisco, this is my 1st Brocade and for the life of me I cant seem to figure out what the same fix would be; again for reference the IOS command. " qos queue-softmax-multiplier 1200" magically solved the same exact performance issue on my 3650. My further understanding is that the 7250 only has a 2Mb buffer which is not adequate for mixing both 1Gb and 10Gb clients, something closer to 8-9Mb is needed for 10Gb and 32Mb is generally required for 25Gb. Most of the PFSense community is running cisco gear hence why I am turning to this form for help on the Brocade. Reading all the manuals I feel something in the realm of Brocades QOS "ingress-buffer-profile" or "egress-buffer-profile" may be the solution, but really not sure. Another hunch is " buffer-sharing-full", sounds dangerous but interesting may give that a go.

Thanks for all the help, hopefully this is a simple fix.


My ICX 7250 Config:

SSH@romulus>show run
Current configuration:
!
ver 08.0.95pT213
!
stack unit 1
module 1 icx7250-48-port-management-module
module 2 icx7250-sfp-plus-8port-80g-module
stack-port 1/2/1
stack-port 1/2/3
!
vlan 1 name DEFAULT-VLAN by port
router-interface ve 1
!
!
symmetrical-flow-control enable
!
!
optical-monitor
optical-monitor non-ruckus-optic-enable
aaa authentication web-server default local
aaa authentication login default local
enable aaa console
hostname romulus
ip dhcp-client disable
ip dns server-address 10.0.0.1
ip route 0.0.0.0/0 10.0.0.1
!
no telnet server

!
clock timezone us Eastern
!
!
ntp
disable serve
server time.cloudflare.com
!
!
no web-management http
!
manager disable
!
!
manager port-list 987
!

!
interface ethernet 1/1/4
flow-control neg-on
!
interface ethernet 1/1/48
flow-control neg-on
!
interface ethernet 1/2/1
flow-control neg-on
!
interface ethernet 1/2/8
flow-control neg-on
!
interface ve 1
ip address 10.0.0.3 255.255.255.0
!
!
end

 

[kpfleming]

The performance issue I am seeing with a very basic clean ICX configuration is that when clients on 1Gb ports perform a long sustained speed test or high speed download their speed begins to taper down pretty fast. They will start at 900Mbps and taper down to 500Mpbs like clockwork. If I physically move that uplink from 10Gb to 1Gb all my clients at 1Gb can sustain speed tests and downloads at 900Mbps with no loss in speed.

Are these tests being run between two clients (hosts) on the ICX, or between a client and the firewall, or between a client and something outside the firewall?

 

[ngr2001]

To Clarify, the performance issue is being seen and tested by the following.

PFSense LAN NIC connected to SFP+ port 8 @ 10Gb, Windows 11 client connected to ICX-7240 Port 4 @ 1Gb, client navigates to speedtest.net. The benchmark will start at 850Mbps+ and quickly drop to 500Mbps. So the test is pure client to something external. Now this poor performance only happens when the Firewall's LAN NIC is connected to the ICX at 10Gb, as soon as I swap that connection to 1Gb the above poor performance goes away completely, same client runs a speed test and it stays pegged at 850Mbps.

Multiple Cisco experts confirmed its a buffer issue when you have ports running on the same switch at different speeds, again solved this on a Cisco 3650 but cant seem to figure out how to solve on the ICX-7250.

 

[kapone]

Not entirely true, as I have firewall rules that are easy to duplicate on L3 switch such as not letting some vlans talk to others, to impossible to replicate on L3 switch such as downloading DoT and DoH ip addresses, blocking dns queries to them and NAT regular DNS to my servers that block ads, trackers and all that at the network level so I see no ads. All above that it allows me to block sites that spy on me such as Google, CLoudflare, meta, etc etc. as much as possible.

And the firewall is exactly where that should be done. My point is...none of that requires VLAN routing on the firewall. A transit VLAN will work just perfect.

 

[kapone]

@kapone Can you expand a bit further or maybe give some subtle hints on a basic inter-VLAN routing using only Brocade switches? I am going to soon be setting up an OPNsense firewall along with two 6450's (stacked). I plan on having 5 VLANs (management, IOT, cameras, trust, guest) and I am so use to the UI of OPNsense firewall that I planned to use it for all of the firewall rules (as you know, LAN type rules in pfSense/OPNsense only controls outbound traffic). So for example, all VLANs can access cameras VLAN but cameras cannot access anyone including internet. Only management VLAN can access OPNsense UI/ssh. Guest has only access to internet with no port blocking, etc.

I'm also familiar familiar wtih OPNsense firewall DHCP settings, statuses, reservations, etc. If I don't use the OPNsense for DHCP and only for firewall rules as it relates to the internet that would be great but it seems like I'll be having to go to two locations to truly understand what is going on (brocade and firewall).

I have a very beefy OPNsense box that is overkill for the home environment so I figured I wouldn't get much degregation, but it will certainly be less efficient than keeping everything at the L3 level. I just don't have the depth of knowledge of Brocade equipment and L3 in general to feel confident in my CLI-fu skills to set it up right so any direction is greatly appreciated.

https://forums.servethehome.com/index.php?threads/layer-3-switch-w-pfsense.23236/#post-216621

 

[kapone]

To Clarify, the performance issue is being seen and tested by the following.

PFSense LAN NIC connected to SFP+ port 8 @ 10Gb, Windows 11 client connected to ICX-7240 Port 4 @ 1Gb, client navigates to speedtest.net. The benchmark will start at 850Mbps+ and quickly drop to 500Mbps. So the test is pure client to something external. Now this poor performance only happens when the Firewall's LAN NIC is connected to the ICX at 10Gb, as soon as I swap that connection to 1Gb the above poor performance goes away completely, same client runs a speed test and it stays pegged at 850Mbps.

Multiple Cisco experts confirmed its a buffer issue when you have ports running on the same switch at different speeds, again solved this on a Cisco 3650 but cant seem to figure out how to solve on the ICX-7250.

I see that you have flow control enabled on ports as well as SFC globally. Try disabling it?

Edit: I know this is not apples to apples...but still. Factory fresh Mellanox SX6036 (which replaced my ICX 6610) with no tuning on any end... connected to my OpnSense (that's irrelevant, could be pfSense) box over a 10gb port breakout to a 2gb FIOS fiber network.

 

[ngr2001]

I see that you have flow control enabled on ports as well as SFC globally. Try disabling it?

Edit: I know this is not apples to apples...but still. Factory fresh Mellanox SX6036 (which replaced my ICX 6610) with no tuning on any end... connected to my OpnSense (that's irrelevant, could be pfSense) box over a 10gb port breakout to a 2gb FIOS fiber network.

I started with Flow Control disabled, stock config on the ICX-7250, I enabled Flow Control as a test to see if it would fix the issue. Neither option improves the speed.

Your test above also does not jive with mine. You scored a download speed that is impossible for a 1Gb client. I left this out but my Windows clients that have a 10Gb NIC have 0 performance issues when the PFSense LAN NIC is also at 10Gb.

My WAN is at 2.5Gb May LAN Uplink is at 10Gb and my clients are just a few Windows boxes, some at 1Gb some at 10Gb.

Also I have noticed that the Waveform site will score the same on nearly every config, its a not a good test for what I am seeing, waveform is the best for sure but they are doing something special with their own buffering. Try fast.com and speedtest.net with one of your clients that is only at 1Gb and see if it can maintain speeds over 600Mbps. Also your switch may be much better than mine, i.e. maybe it has much larger default memory buffers and does not have this issue.

 

[TonyArrr]

not sure why everyone thinks I have VLAN routing going on

I’ll own that, I suggested it might be a difference between our configs earlier, and it kind of stuck, sorry!

It is interesting, the buffer bloat you’re catching, so I look forward to seeing the solution since my ISP might have me on multi-gig in a year or so.

Out of curiousity, do “real world” downloads get affected as much? Speed tests always taper off quite a lot for me (when I was on 100Mb until recently, it’d start in the 90s and end in the 60s, now on 1Gb, it starts off in the high 900s and ends in the 6-700s) but if I download a large steam game or Windows ISO (or very well seeded Debian ISO), I rarely see below 800.
Just for the data point anyway.

 

[kapone]

Also I have noticed that the Waveform site will score the same on nearly every config, its a not a good test for what I am seeing, waveform is the best for sure but they are doing something special with their own buffering. Try fast.com and speedtest.net with one of your clients that is only at 1Gb and see if it can maintain speeds over 600Mbps. Also your switch may be much better than mine, i.e. maybe it has much larger default memory buffers and does not have this issue.

Yes, they can. I have many clients at 1gb in the mix. They can all do >900mbps both ways. And no, this is mot because of the new switch. I had the same results with the 6610 as well.

That said, I do not have a 7250, so, my results are not directly applicable, but I’d be surprised, if the 7250 was materially different. All these switches are more than capable.

Edit: This triggered a memory. I did have to make one change, once I went past gigabit on the WAN side. On my OpnSense box (no settings change on the switches or clients), I had to turn on flow control on the WAN nic. The reason was because, with multi-gigabit, FIOS now gives you a 10g drop, even though the actual speeds are capped.

Not sure if that makes a difference in this case or not.

 

[dasbooter]

Setup issue.

@ngr2001 :

1. Your performance issues for the 1gb clients is almost certainly a config/user error issue. Just gotta get the configuration right. I ran pfSense with the ICX for the longest time and had zero issues with 1gb or 10gb (or 40gb) clients. But...read below.

2. Really, with these types of L3 switches (i.e powerful/line rate L3 switching), there is zero reason to do VLAN routing in your firewall. All VLAN routing should be at L3 level, with only external ingress/egress going to your firewall.

p.s. A lot of people confuse the fact that you don't VLANs in pfSense/OpnSense to do firewall rules. Aliases/Networks (which is what a VLAN is)/Single Hosts all work just fine. The only reason to create interfaces/VLANs in pfSense/OpnSense with a L3 switch, would be if you wanted your firewall for DHCP.... There's many ways to not do that.

My apologies this might not be the best place to ask this question. Almost all of the last 10 posts are above my head. I only ask because well fodheeshas docs have allowed me to set up @ 6450 with L3 firmware.
I've had the switch for a couple of years in a smart home setting and it's been forgiving. Everything is in the default vlan. I have a couple of ubiquity AP's and a residential Netgear r8000 with recent freshtomato firmware doing the routing. I probably have over 100 IOT devices in a smart home environment.
I have had over the years my security system which is "connected" go unavailable seemingly after using the Sonos sound system. This seems like a good place to start with vlans.
I'm really looking for advice on how to get started. I've watched quite of few videos and read quite a bit and I understand the basics of vlans.
Simply put given the hardware in place what is the best way for me a complete amateur to start with vlans.

 

[ngr2001]

My WAN & LAN NIC both have Flow Control enabled on the PFSense side.

Keep in mind I can reproduce this issue with a Cisco 3650 switch, only difference is that I can fix the issue on the Cisco side with a single config addition: "qos queue-softmax-multiplier 1200".

Case and point being, I bet most lower tier Ent switches have this issue.

I am hoping that someone here know a little more about tweaking the queues and buffers on the Brocade ICX Series, im sure there must be an apples to apples fix.

 

[TonyArrr]

It seems like an odd issue, I’ve just tried pulling data from the SMB share on my server (connected via 10Gb on the 7250) to my desktop (connected to 1/1/4, 1Gb), and comfortably stuck above 900.
So this can’t be a hardware limitation, and there has to be a way to configure out of it for you.
I’ve only configured ACLs and VLANs on mine though, never looked at QOS stuff

In the guides Fohdeesha includes on the download, have you looked in the “QOS and Traffic Management Guide” for FastIron? I skimmed the contents and there are sections of queue priority and ingress/egress buffers, but I’m far from an expert in this area so couldn’t say what to change for sure…

 

[klui]

You should try some other scenarios.

• Do you experience the same thing if you connect your Windows system to your pfSense's 1 Gb port?
• What controller is your Windows's NIC? I've experienced issues with older NICs and sometimes updating their firmware has had some positive effect.
• Run Linux to see if it has the same issue.
• Try to display error counters to see what's causing the slowdown.

 

[kapone]

My WAN & LAN NIC both have Flow Control enabled on the PFSense side.

Keep in mind I can reproduce this issue with a Cisco 3650 switch, only difference is that I can fix the issue on the Cisco side with a single config addition: "qos queue-softmax-multiplier 1200".

Case and point being, I bet most lower tier Ent switches have this issue.

I am hoping that someone here know a little more about tweaking the queues and buffers on the Brocade ICX Series, im sure there must be an apples to apples fix.

But, see…that’s the point. A<>B in this case. Just because your Cisco had this issue, doesn’t necessarily mean another switch would. I’ve never had to tweak QOS settings on my 6610 or the new SX6036, until I started configuring my setup for RDMA/RoceV2.

For normal operation, there should be no need for that. This thread is quite a few years old, and you’re the first one to bring up QOS in typical switching scenarios. There has to be something unique in your setup.

 

[kapone]

My apologies this might not be the best place to ask this question. Almost all of the last 10 posts are above my head. I only ask because well fodheeshas docs have allowed me to set up @ 6450 with L3 firmware.
I've had the switch for a couple of years in a smart home setting and it's been forgiving. Everything is in the default vlan. I have a couple of ubiquity AP's and a residential Netgear r8000 with recent freshtomato firmware doing the routing. I probably have over 100 IOT devices in a smart home environment.
I have had over the years my security system which is "connected" go unavailable seemingly after using the Sonos sound system. This seems like a good place to start with vlans.
I'm really looking for advice on how to get started. I've watched quite of few videos and read quite a bit and I understand the basics of vlans.
Simply put given the hardware in place what is the best way for me a complete amateur to start with vlans.

Honestly? Pay someone to do it. I don’t mean that in a facetious way, but this stuff is not entirely straightforward…and given the number of devices on your network…I see pain. Lots of pain.

 

[SuperMiguel]

Hello all here is my config, im trying to route vlan traffic on the switch instead of on the router.

But when i try setting up my gateway from example from vlan 10 to 10.0.10.2 and try pinging something on another vlan it doesnt work. Setting my gateway to my router works fine.

show conf
!
Startup-config data location is flash memory
!
Startup configuration:
!
ver 08.0.30tT7f3
!
stack unit 1
  module 1 icx6610-48p-poe-port-management-module
  module 2 icx6610-qsfp-10-port-160g-module
  module 3 icx6610-8-port-10g-dual-mode-module
!
global-stp
!
!
!
spanning-tree single
!
vlan 1 name DEFAULT-VLAN by port
 router-interface ve 1
 spanning-tree
!
vlan 10 name TRUSTED by port
 tagged ethe 1/1/1 to 1/1/3 ethe 1/1/36 ethe 1/1/38 ethe 1/1/41 ethe 1/1/43 ethe 1/3/1 to 1/3/2
 untagged ethe 1/1/8 to 1/1/9 ethe 1/1/12 ethe 1/1/16 ethe 1/1/18 ethe 1/1/22 ethe 1/1/24 to 1/1/28 ethe 1/1/30 ethe 1/3/3 ethe 1/3/8
 router-interface ve 10
 spanning-tree
!
vlan 69 name GUEST by port
 tagged ethe 1/1/1 to 1/1/3 ethe 1/1/36 ethe 1/1/41 ethe 1/1/43 ethe 1/3/1
 router-interface ve 69
 spanning-tree
!
vlan 88 name WORK by port
 tagged ethe 1/1/1 to 1/1/3 ethe 1/1/36 ethe 1/1/38 ethe 1/1/41 ethe 1/1/43 ethe 1/3/1
 router-interface ve 88
 spanning-tree
!
vlan 120 name IOT by port
 tagged ethe 1/1/1 to 1/1/3 ethe 1/1/36 ethe 1/1/41 ethe 1/1/43 ethe 1/3/1
 untagged ethe 1/1/37
 router-interface ve 120
 spanning-tree
!
vlan 130 name CCTV by port
 tagged ethe 1/1/1 to 1/1/3 ethe 1/1/36 ethe 1/1/41 ethe 1/1/43 ethe 1/3/1 to 1/3/2
 untagged ethe 1/1/14 to 1/1/15 ethe 1/1/19 ethe 1/1/31 to 1/1/32 ethe 1/1/34 ethe 1/1/39 to 1/1/40 ethe 1/1/45 ethe 1/1/47 to 1/1/48 ethe 1/3/7
 router-interface ve 130
 spanning-tree
!
!
spanning-tree single 802-1w
spanning-tree single 802-1w priority 0
!
!
!
!
aaa authentication web-server default local
aaa authentication enable default local
aaa authentication login default local
enable telnet authentication
hostname superbeefbox
ip dhcp-client disable
ip route 0.0.0.0/0 10.0.0.1
!
username root password .....
!
!
!
!
!
!
!
!
!
interface ethernet 1/1/1
 dual-mode
!
interface ethernet 1/1/2
 port-name DownStairs-AP
 dual-mode
 inline power
!
interface ethernet 1/1/3
 port-name UpStairs-AP
 dual-mode
 inline power
!
interface ethernet 1/1/4
 port-name Firewall-IPMI
!
interface ethernet 1/1/5
 port-name Unifi-CloudKey
 inline power
!
interface ethernet 1/1/8
 port-name Sonos-Amp
!
interface ethernet 1/1/9
 port-name Living-room-apple-tv
!
interface ethernet 1/1/12
 port-name AC_Closet_Switch
!
interface ethernet 1/1/14
 port-name Driveway-Camera
 inline power
!
interface ethernet 1/1/15
 port-name Backyard-Camera
 inline power
!
interface ethernet 1/1/16
 port-name SuperDesktop
!
interface ethernet 1/1/18
 port-name SonosArc
!
interface ethernet 1/1/19
 port-name Lanai-Camera
 inline power
!
interface ethernet 1/1/22
 port-name Gym-AppleTV
!
interface ethernet 1/1/24
 port-name MasterBed AppleTV
!
interface ethernet 1/1/25
 port-name Sonos-Amp-Pool
!
interface ethernet 1/1/26
 port-name Envisalink
!
interface ethernet 1/1/27
 port-name Hubitat Upstairs
!
interface ethernet 1/1/28
 port-name Sonos-Amp-Kitchen
!
interface ethernet 1/1/31
 port-name KidsHall-Camera
 inline power
!
interface ethernet 1/1/32
 port-name Stairs-Camera
 inline power
!
interface ethernet 1/1/34
 port-name LivingRoom-Camera
 inline power
!
interface ethernet 1/1/36
 port-name Lanai-AP
 dual-mode
 inline power
!
interface ethernet 1/1/37
 port-name GameRoomSwitch
!
interface ethernet 1/1/38
 port-name Yanelis-Office-Switch
 dual-mode
 inline power
!
interface ethernet 1/1/39
 port-name SmallGarage-Camera
 inline power
!
interface ethernet 1/1/40
 port-name LargeGarage-Camera
 inline power
!
interface ethernet 1/1/41
 port-name MiguelOffice-2nd
 dual-mode
!
interface ethernet 1/1/43
 port-name LGarage-AP
 dual-mode
 inline power
!
interface ethernet 1/1/45
 port-name Backyard-Side-Camera
 inline power
!
interface ethernet 1/1/47
 port-name Kitchen-Camera
 inline power
!
interface ethernet 1/1/48
 port-name FrontDoor-Camera
 inline power
!
interface ethernet 1/3/1
 port-name LAN/Firewall
 dual-mode
 speed-duplex 10G-full
!
interface ethernet 1/3/2
 port-name Beef_Garage
 dual-mode
 speed-duplex 10G-full
!
interface ethernet 1/3/3
 port-name SuperDesktop
 speed-duplex 10G-full
!
interface ethernet 1/3/4
 speed-duplex 10G-full
!
interface ethernet 1/3/5
 speed-duplex 10G-full
!
interface ethernet 1/3/6
 speed-duplex 10G-full
!
interface ethernet 1/3/7
 port-name BlueIris
 speed-duplex 10G-full
!
interface ethernet 1/3/8
 port-name BlueIris
 speed-duplex 10G-full
!
interface ve 1
 ip address 10.0.0.2 255.255.255.0
!
interface ve 10
 ip address 10.0.10.2 255.255.255.0
!
interface ve 69
 ip address 172.30.69.2 255.255.255.0
!
interface ve 88
 ip address 172.18.88.2 255.255.255.0
!
interface ve 120
 ip address 192.168.120.2 255.255.255.0
!
interface ve 130
 ip address 192.168.130.2 255.255.255.0
!
!
!
!
!
lldp run
!
!
!
!
end

 

[chlastakov]

Can be ICX6610 reversed? From exhaust to intake? I mean can I open vents and power sources and turn over fans, or it will not work as expected?

I already searched in previous discussion and there was this question few times, but was actually not answered. Only mentions about -I and -E versions on eBay.