Drag to reposition cover

Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

puffer

New Member
Mar 12, 2025
4
0
1
With a single public IP, and a Layer 3 switch...and the need to do inter-vlan routing on the switch...and HA firewalls... :) You kinda don't have too many choices. You can remove the FIOS router, but you'll still have "some" router that is single, and then double NAT to your HA config.

That's why I suggested a hot/cold firewall config. That gets you to a TRANSIT model, where the WAN from the ONT is terminated directly on the switch (in a VLAN, no VE), both hot/cold firewalls have their WAN interfaces connected to that VLAN, DHCP active for WAN, but only one firewall active at a time.
This is basically the long term plan, but most likely with a single firewall. Just not there yet, taking this all apart to get there is complicated and messy and has other implications I'm not ready for yet. Was hoping to just do the inter-vlan routing for the time being.
 

kapone

Well-Known Member
May 23, 2015
1,357
803
113
This is basically the long term plan, but most likely with a single firewall. Just not there yet, taking this all apart to get there is complicated and messy and has other implications I'm not ready for yet. Was hoping to just do the inter-vlan routing for the time being.
The inter-vlan routing can be done on the switch with your current config...but it won't remove your FIOS router.

And yes, it'll involve reconfiguring quite a few things as well.
 

steamroller223

New Member
Mar 14, 2025
3
1
3
I want to rule out the ICX switch as the issue here. I have my switch sitting between my openwrt router (port 1/1/8) and proxmox system (1/2/2).
1. I have the proxmox bridge devices set as VLAN aware and the network config with VLAN id 10 for the LXCs
2. I have open wrt tagging the output port 1/1/8 with vlan id 10 and an interface defined on it.

The problem is openwrt can't see the proxmox system and vice versa, openwrt can ping the switch on vlan 10, but cannot see anything else. I've tried with the LXCs with DHCP and a static ip, no router to open wrt. Here is my testing ICX 6450 config. This looks correct to me but it doesn't work.

Code:
Current configuration:


ver 08.0.30uT313

stack unit 1
  module 1 icx6450-24-port-management-module
  module 2 icx6450-sfp-plus-4port-40g-module



vlan 1 name DEFAULT-VLAN by port
router-interface ve 1



vlan 10 by port
tagged ethe 1/1/8 ethe 1/2/2
router-interface ve 10

aaa authentication web-server default local
aaa authentication login default local                        
enable aaa console
hostname ICX6450-24-Router
ip dhcp-client disable
no telnet server
username root password .....


interface ve 1
ip address 192.168.1.51 255.255.255.0


interface ve 10
ip address 192.168.10.2 255.255.255.0


end
 

kpfleming

Active Member
Dec 28, 2021
446
230
43
Pelham NY USA
That's why I suggested a hot/cold firewall config. That gets you to a TRANSIT model, where the WAN from the ONT is terminated directly on the switch (in a VLAN, no VE), both hot/cold firewalls have their WAN interfaces connected to that VLAN, DHCP active for WAN, but only one firewall active at a time.
This is what I do (also with FIOS), and it works really well. I use Pacemaker to manage the 'active'/'passive' modes of the firewalls/routers.
 

jode

Member
Jul 27, 2021
77
59
18
I want to rule out the ICX switch as the issue here. I have my switch sitting between my openwrt router (port 1/1/8) and proxmox system (1/2/2).
1. I have the proxmox bridge devices set as VLAN aware and the network config with VLAN id 10 for the LXCs
2. I have open wrt tagging the output port 1/1/8 with vlan id 10 and an interface defined on it.

The problem is openwrt can't see the proxmox system and vice versa, openwrt can ping the switch on vlan 10, but cannot see anything else. I've tried with the LXCs with DHCP and a static ip, no router to open wrt. Here is my testing ICX 6450 config. This looks correct to me but it doesn't work.
Both openwrt and proxmox are on the same net, L3 only, no routing. Both openwrt and proxmox need to be vlan aware and untag packets.

Did you set the IP for Proxmox on a vlan?

Screenshot From 2025-03-15 09-20-45.png
 

steamroller223

New Member
Mar 14, 2025
3
1
3
Both openwrt and proxmox are on the same net, L3 only, no routing. Both openwrt and proxmox need to be vlan aware and untag packets.

Did you set the IP for Proxmox on a vlan?

View attachment 42498
This is what the set up looks like right now. I did not set up management on a VLAN, eventually I will but not yet. I have a second interface that is receiving the tagged vlans
1742054905378.png
1. OpenWRT (Untagged VLan99) -> ICX 6450 (DEFUALT VLAN 1) ->PVE(vmbr0)
2. OpenWRT (tagged VLan99, 10 ,20 ,30) -> ICX 6450 (tagged VLan99, 10 ,20 ,30) ->PVE(vmbr1, VLAN Aware) ->PVE(LXC, vmbr1 VLAN Tag 10)


With route 1 I can communicate with everything since it's basically no VLANs so any DEFUALT VLAN 1 port on my ICX6450 works with it.

With route 2 is where the problem is, The LXC is not pulling an IP, however when I connected up a raspberry pi to a VLAN untagged 10 port on the ICX switch the raspberry pi was able to pull a DHCP lease (sometimes and only with specific VLAN numbers, however they configured the same in openwrt so don't really know what's going on here) . So it seems that the OpenWRT to ICX route is sometimes working... but not when using a tagged port with proxmox.

If it's helpful here is the updated ICX config, pve and openwrt config

Code:
Current configuration:
!
ver 08.0.30uT313
!
stack unit 1
  module 1 icx6450-24-port-management-module
  module 2 icx6450-sfp-plus-4port-40g-module
!
!
!
!
vlan 1 name DEFAULT-VLAN by port
 router-interface ve 1
!
vlan 10 by port
 tagged ethe 1/1/8 ethe 1/1/10 ethe 1/2/2
 untagged ethe 1/1/7
!
vlan 20 by port
 tagged ethe 1/1/8 ethe 1/1/10 ethe 1/2/2
!
vlan 30 by port
 tagged ethe 1/1/8 ethe 1/1/10 ethe 1/2/2
!                                                                 
vlan 99 by port
 tagged ethe 1/1/8 ethe 1/2/2
 untagged ethe 1/1/11
!
!
!
!
!
aaa authentication web-server default local
aaa authentication login default local
enable aaa console
hostname ICX6450-24-Router
ip dhcp-client disable
!
no telnet server
username root password .....
!
!
!
!
!
interface ve 1
 ip address 192.168.1.51 255.255.255.0                           
!
!
!
!
!
!
!
!
!
end
Code:
auto lo
iface lo inet loopback

iface enp2s0f1np1 inet manual

iface enp87s0 inet manual

iface enp90s0 inet manual

iface enp2s0f0np0 inet manual

auto vmbr0
iface vmbr0 inet static
        address 192.168.1.100/24
        bridge-ports enp2s0f1np1
        bridge-stp off
        bridge-fd 0

iface wlp91s0 inet manual

auto vmbr1
iface vmbr1 inet manual
        bridge-ports enp2s0f0np0
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-300

auto vmbr1.10
iface vmbr1.10 inet static
        address 192.168.10.100/24
        gateway 192.168.10.1

source /etc/network/interfaces.d/*
Code:
config interface 'loopback'
    option device 'lo'
    option proto 'static'
    option ipaddr '127.0.0.1'
    option netmask '255.0.0.0'

config globals 'globals'
    option ula_prefix 'fd07:3240:cfca::/48'
    option packet_steering '1'

config device
    option name 'br-lan'
    option type 'bridge'
    list ports 'eth0'
    list ports 'eth2'
    option stp '1'
    option igmp_snooping '1'
    option ipv6 '1'

config device
    option name 'eth2'
    option macaddr '76:07:ea:c9:df:d1'

config device
    option name 'eth0'
    option macaddr '76:07:ea:c9:df:d1'

config interface 'lan'
    option device 'br-lan.99'
    option proto 'static'
    option ipaddr '192.168.1.1'
    option netmask '255.255.255.0'
    option ip6assign '64'

config device
    option name 'eth1'
    option macaddr '76:07:ea:c9:df:d0'

config interface 'wan'
    option device 'wan.201'
    option proto 'pppoe'
    option username '<redacted>'
    option password '<redacted>'
    option ipv6 'auto'
    option peerdns '0'
    list dns '1.0.0.1'
    list dns '1.1.1.1'

config interface 'docker'
    option device 'docker0'
    option proto 'none'
    option auto '0'

config device
    option type 'bridge'
    option name 'docker0'

config device
    option type '8021q'
    option ifname 'eth1'
    option vid '201'
    option name 'wan.201'
    option ipv6 '1'

config interface 'iots'
    option proto 'static'
    option device 'br-lan.10'
    option ipaddr '192.168.10.1'
    option netmask '255.255.255.0'
    option ip6assign '64'

config bridge-vlan
    option device 'br-lan'
    option vlan '99'
    list ports 'eth0:t'
    list ports 'eth2'

config bridge-vlan
    option device 'br-lan'
    option vlan '10'
    list ports 'eth0:t'

config bridge-vlan
    option device 'br-lan'
    option vlan '20'
    list ports 'eth0:t'

config bridge-vlan
    option device 'br-lan'
    option vlan '30'
    list ports 'eth0:t*'

config interface 'niots'
    option proto 'static'
    option device 'br-lan.20'
    option ipaddr '192.168.20.1'
    option netmask '255.255.255.0'

config interface 'guest'
    option proto 'static'
    option device 'br-lan.30'
    list ipaddr '192.168.30.1/24'
 
Last edited:

jode

Member
Jul 27, 2021
77
59
18
With route 2 is where the problem is, The LXC is not pulling an IP, however when I connected up a raspberry pi to a VLAN untagged 10 port on the ICX switch the raspberry pi was able to pull a DHCP lease (sometimes and only with specific VLAN numbers, however they configured the same in openwrt so don't really know what's going on here) . So it seems that the OpenWRT to ICX route is sometimes working... but not when using a tagged port with proxmox.
I assume the openwrt address on vlan10 is 192.168.10.1?
- Can openwrt reach the vmbr1.10 interface in Proxmox? (openwrt> ping 192.168.10.100 )
- Can vmbr1.10 interface in Proxmox reach openwrt? ( proxmox> ping -I 192.168.10.100 192.168.10.1)
If neither works I'd look deeper into openwrt.

Anyhow, the switch looks fine due to the established connectivity with your raspi.
 

kevindd992002

Member
Oct 4, 2021
136
8
18
I've been using my ICX6450-48P (got from ebay) for over a year now. Recently, I've had problem with two ports in a span of just a few months and they have similar issues. They can supply PoE power to my PoE cameras but there is link established. Transferring the cameras to different ports fixes the issue so I know the issue is in the switchports themselves. What do I need to check? Maybe dirty pin contacts or something?
Do you guys have any ideas regarding this?
 

steamroller223

New Member
Mar 14, 2025
3
1
3
I assume the openwrt address on vlan10 is 192.168.10.1?
- Can openwrt reach the vmbr1.10 interface in Proxmox? (openwrt> ping 192.168.10.100 )
- Can vmbr1.10 interface in Proxmox reach openwrt? ( proxmox> ping -I 192.168.10.100 192.168.10.1)
If neither works I'd look deeper into openwrt.

Anyhow, the switch looks fine due to the established connectivity with your raspi.
Thanks for the help, I did end up figuring it out. I can edit the message later for posterity with the actual configs. I configured the ICX 7250 as a layer 2 device for each vlan (no interfaces on the vlans) and I tagged the ports/lag I wanted for each vlan.

On Openwrt I created static address interfaces with a bridge device and selected vlan filtering for the vlans id's I wanted and tagged the wired device output. Then created a firewall rule that had input as ACCEPT. I later configured it to isolate with REJECT and DNS and DHCP INPUT traffic rules, still worked.

On proxmox I did NOT set the bridge device as VLAN aware, on the LXC I set the network device to the bridge that had the tagged vlan and set the vlan id at the LXC network options.

Basically it was not a ICX 7250 issue (or ICX 6450, I upgraded since the last post), turns out that was the easiest to configure correctly even got ansible to re-provision it for me with out too much trouble. haha I think the problem originally was setting the proxmox bridge to VLAN aware and not setting the firewall rules in openWRT.
 
Last edited:
  • Like
Reactions: jode

edusico

New Member
Jul 7, 2022
3
1
3
Hello guys, sorry my very noob question, but I bought a ICX6450-48P on Ebay and I'm fighting to update the firmware with Fohdeesha files.

I'm able to connect to the management ethernet port with PuTTy and execute the commands on the bootloader prompt.

My question is: which port from switch do I need to connect to my laptop to execute 'update_primary' and 'update_uboot' commands and start the process from Tftpd64 (my IP address is defined as the same from command "setenv serverip 192.168.1.8")?
 

StarsAndBars

Member
Jul 13, 2020
32
3
8
Greetings All!

I was curious to know what the best approach is to stacking two 6610 units over a distance, say 150 feet or less? Do they make "transceivers" for the 40gb stacking ports that would allow me to run my own fiber in conduit instead of trying something ridiculous like a longer version of stacking cable? I shudder to think what that would even cost. I have two buildings on my property with 1" empty conduit laid for this sort of thing. If such transceivers exist, what is the part # and a good place (eBay?) to get them from? Also, what fiber (with what connectors) would I run between? I assume I can still use a breakout cable at each end for other devices connected to the switch, while still simultaneously using the 40gb "trunk" ports? Thanks in Advance!
 

dasbooter

New Member
Mar 16, 2022
18
1
3
Is there a specific recommendation for using an adapter eg:
https://www.servethehome.com/fs-sfp-10g-t-review-another-sfp-to-10gbase-t-option/
I'm interested in possibly running a docsis 3.1 modem 2.5 gig port to one of the sfp+ ports on a brocade 6450 48p.

After reading a bit it seems like flow control is a concern. Is it a straight forward hurdle to overcome. I did see a reddit thread where A Microtic adapter was used for a 6450 at 2.5 gig
 

clcorbin

Member
Feb 15, 2014
80
11
8
Greetings All!

I was curious to know what the best approach is to stacking two 6610 units over a distance, say 150 feet or less? Do they make "transceivers" for the 40gb stacking ports that would allow me to run my own fiber in conduit instead of trying something ridiculous like a longer version of stacking cable? I shudder to think what that would even cost. I have two buildings on my property with 1" empty conduit laid for this sort of thing. If such transceivers exist, what is the part # and a good place (eBay?) to get them from? Also, what fiber (with what connectors) would I run between? I assume I can still use a breakout cable at each end for other devices connected to the switch, while still simultaneously using the 40gb "trunk" ports? Thanks in Advance!
I had my two ICX 6610 stacked using the 40Gb ports with MPO fiber. About 50 meters (more than 150 feet) so this should be easy.

And yes, if you follow fohdeesha's detailed directions, you can easily use the first two ports for stacking at 40Gb/s each and then break out the other two ports for 8 extra 10Gb/s ports. The only limitation on using the break out ports is you will be using an MPO -> 4xLC break out cable, so you can't connect to 10Base-T cards, only SFP+ "stuff" that can accept an LC optical transceiver.