This is basically the long term plan, but most likely with a single firewall. Just not there yet, taking this all apart to get there is complicated and messy and has other implications I'm not ready for yet. Was hoping to just do the inter-vlan routing for the time being.With a single public IP, and a Layer 3 switch...and the need to do inter-vlan routing on the switch...and HA firewalls...You kinda don't have too many choices. You can remove the FIOS router, but you'll still have "some" router that is single, and then double NAT to your HA config.
That's why I suggested a hot/cold firewall config. That gets you to a TRANSIT model, where the WAN from the ONT is terminated directly on the switch (in a VLAN, no VE), both hot/cold firewalls have their WAN interfaces connected to that VLAN, DHCP active for WAN, but only one firewall active at a time.
Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)
- Thread starter fohdeesha
- Start date
Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
The inter-vlan routing can be done on the switch with your current config...but it won't remove your FIOS router.
And yes, it'll involve reconfiguring quite a few things as well.
I want to rule out the ICX switch as the issue here. I have my switch sitting between my openwrt router (port 1/1/8) and proxmox system (1/2/2).
1. I have the proxmox bridge devices set as VLAN aware and the network config with VLAN id 10 for the LXCs
2. I have open wrt tagging the output port 1/1/8 with vlan id 10 and an interface defined on it.
The problem is openwrt can't see the proxmox system and vice versa, openwrt can ping the switch on vlan 10, but cannot see anything else. I've tried with the LXCs with DHCP and a static ip, no router to open wrt. Here is my testing ICX 6450 config. This looks correct to me but it doesn't work.
1. I have the proxmox bridge devices set as VLAN aware and the network config with VLAN id 10 for the LXCs
2. I have open wrt tagging the output port 1/1/8 with vlan id 10 and an interface defined on it.
The problem is openwrt can't see the proxmox system and vice versa, openwrt can ping the switch on vlan 10, but cannot see anything else. I've tried with the LXCs with DHCP and a static ip, no router to open wrt. Here is my testing ICX 6450 config. This looks correct to me but it doesn't work.
Code:
Current configuration:
ver 08.0.30uT313
stack unit 1
module 1 icx6450-24-port-management-module
module 2 icx6450-sfp-plus-4port-40g-module
vlan 1 name DEFAULT-VLAN by port
router-interface ve 1
vlan 10 by port
tagged ethe 1/1/8 ethe 1/2/2
router-interface ve 10
aaa authentication web-server default local
aaa authentication login default local
enable aaa console
hostname ICX6450-24-Router
ip dhcp-client disable
no telnet server
username root password .....
interface ve 1
ip address 192.168.1.51 255.255.255.0
interface ve 10
ip address 192.168.10.2 255.255.255.0
endThis is what I do (also with FIOS), and it works really well. I use Pacemaker to manage the 'active'/'passive' modes of the firewalls/routers.
Both openwrt and proxmox are on the same net, L3 only, no routing. Both openwrt and proxmox need to be vlan aware and untag packets.
Did you set the IP for Proxmox on a vlan?
This is what the set up looks like right now. I did not set up management on a VLAN, eventually I will but not yet. I have a second interface that is receiving the tagged vlans
1. OpenWRT (Untagged VLan99) -> ICX 6450 (DEFUALT VLAN 1) ->PVE(vmbr0)
2. OpenWRT (tagged VLan99, 10 ,20 ,30) -> ICX 6450 (tagged VLan99, 10 ,20 ,30) ->PVE(vmbr1, VLAN Aware) ->PVE(LXC, vmbr1 VLAN Tag 10)
With route 1 I can communicate with everything since it's basically no VLANs so any DEFUALT VLAN 1 port on my ICX6450 works with it.
With route 2 is where the problem is, The LXC is not pulling an IP, however when I connected up a raspberry pi to a VLAN untagged 10 port on the ICX switch the raspberry pi was able to pull a DHCP lease (sometimes and only with specific VLAN numbers, however they configured the same in openwrt so don't really know what's going on here) . So it seems that the OpenWRT to ICX route is sometimes working... but not when using a tagged port with proxmox.
If it's helpful here is the updated ICX config, pve and openwrt config
Code:
Current configuration:
!
ver 08.0.30uT313
!
stack unit 1
module 1 icx6450-24-port-management-module
module 2 icx6450-sfp-plus-4port-40g-module
!
!
!
!
vlan 1 name DEFAULT-VLAN by port
router-interface ve 1
!
vlan 10 by port
tagged ethe 1/1/8 ethe 1/1/10 ethe 1/2/2
untagged ethe 1/1/7
!
vlan 20 by port
tagged ethe 1/1/8 ethe 1/1/10 ethe 1/2/2
!
vlan 30 by port
tagged ethe 1/1/8 ethe 1/1/10 ethe 1/2/2
!
vlan 99 by port
tagged ethe 1/1/8 ethe 1/2/2
untagged ethe 1/1/11
!
!
!
!
!
aaa authentication web-server default local
aaa authentication login default local
enable aaa console
hostname ICX6450-24-Router
ip dhcp-client disable
!
no telnet server
username root password .....
!
!
!
!
!
interface ve 1
ip address 192.168.1.51 255.255.255.0
!
!
!
!
!
!
!
!
!
end
Code:
auto lo
iface lo inet loopback
iface enp2s0f1np1 inet manual
iface enp87s0 inet manual
iface enp90s0 inet manual
iface enp2s0f0np0 inet manual
auto vmbr0
iface vmbr0 inet static
address 192.168.1.100/24
bridge-ports enp2s0f1np1
bridge-stp off
bridge-fd 0
iface wlp91s0 inet manual
auto vmbr1
iface vmbr1 inet manual
bridge-ports enp2s0f0np0
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes
bridge-vids 2-300
auto vmbr1.10
iface vmbr1.10 inet static
address 192.168.10.100/24
gateway 192.168.10.1
source /etc/network/interfaces.d/*
Code:
config interface 'loopback'
option device 'lo'
option proto 'static'
option ipaddr '127.0.0.1'
option netmask '255.0.0.0'
config globals 'globals'
option ula_prefix 'fd07:3240:cfca::/48'
option packet_steering '1'
config device
option name 'br-lan'
option type 'bridge'
list ports 'eth0'
list ports 'eth2'
option stp '1'
option igmp_snooping '1'
option ipv6 '1'
config device
option name 'eth2'
option macaddr '76:07:ea:c9:df:d1'
config device
option name 'eth0'
option macaddr '76:07:ea:c9:df:d1'
config interface 'lan'
option device 'br-lan.99'
option proto 'static'
option ipaddr '192.168.1.1'
option netmask '255.255.255.0'
option ip6assign '64'
config device
option name 'eth1'
option macaddr '76:07:ea:c9:df:d0'
config interface 'wan'
option device 'wan.201'
option proto 'pppoe'
option username '<redacted>'
option password '<redacted>'
option ipv6 'auto'
option peerdns '0'
list dns '1.0.0.1'
list dns '1.1.1.1'
config interface 'docker'
option device 'docker0'
option proto 'none'
option auto '0'
config device
option type 'bridge'
option name 'docker0'
config device
option type '8021q'
option ifname 'eth1'
option vid '201'
option name 'wan.201'
option ipv6 '1'
config interface 'iots'
option proto 'static'
option device 'br-lan.10'
option ipaddr '192.168.10.1'
option netmask '255.255.255.0'
option ip6assign '64'
config bridge-vlan
option device 'br-lan'
option vlan '99'
list ports 'eth0:t'
list ports 'eth2'
config bridge-vlan
option device 'br-lan'
option vlan '10'
list ports 'eth0:t'
config bridge-vlan
option device 'br-lan'
option vlan '20'
list ports 'eth0:t'
config bridge-vlan
option device 'br-lan'
option vlan '30'
list ports 'eth0:t*'
config interface 'niots'
option proto 'static'
option device 'br-lan.20'
option ipaddr '192.168.20.1'
option netmask '255.255.255.0'
config interface 'guest'
option proto 'static'
option device 'br-lan.30'
list ipaddr '192.168.30.1/24'
Last edited:
I assume the openwrt address on vlan10 is 192.168.10.1?
- Can openwrt reach the vmbr1.10 interface in Proxmox? (
openwrt> ping 192.168.10.100 )- Can vmbr1.10 interface in Proxmox reach openwrt? (
proxmox> ping -I 192.168.10.100 192.168.10.1)If neither works I'd look deeper into openwrt.
Anyhow, the switch looks fine due to the established connectivity with your raspi.
Thanks for the help, I did end up figuring it out. I can edit the message later for posterity with the actual configs. I configured the ICX 7250 as a layer 2 device for each vlan (no interfaces on the vlans) and I tagged the ports/lag I wanted for each vlan.
On Openwrt I created static address interfaces with a bridge device and selected vlan filtering for the vlans id's I wanted and tagged the wired device output. Then created a firewall rule that had input as ACCEPT. I later configured it to isolate with REJECT and DNS and DHCP INPUT traffic rules, still worked.
On proxmox I did NOT set the bridge device as VLAN aware, on the LXC I set the network device to the bridge that had the tagged vlan and set the vlan id at the LXC network options.
Basically it was not a ICX 7250 issue (or ICX 6450, I upgraded since the last post), turns out that was the easiest to configure correctly even got ansible to re-provision it for me with out too much trouble. haha I think the problem originally was setting the proxmox bridge to VLAN aware and not setting the firewall rules in openWRT.
Last edited:
Hello guys, sorry my very noob question, but I bought a ICX6450-48P on Ebay and I'm fighting to update the firmware with Fohdeesha files.
I'm able to connect to the management ethernet port with PuTTy and execute the commands on the bootloader prompt.
My question is: which port from switch do I need to connect to my laptop to execute 'update_primary' and 'update_uboot' commands and start the process from Tftpd64 (my IP address is defined as the same from command "setenv serverip 192.168.1.8")?
I'm able to connect to the management ethernet port with PuTTy and execute the commands on the bootloader prompt.
My question is: which port from switch do I need to connect to my laptop to execute 'update_primary' and 'update_uboot' commands and start the process from Tftpd64 (my IP address is defined as the same from command "setenv serverip 192.168.1.8")?
Hello, thanks for reply but I think my question wasn't clear. I drew the current scenario:
The answer you already received: the management port, which is directly below the console port on the left end.
I can't seem to not select by member? Is the problem.sure you can, just type igmp in the search box up top, and select "this thread":
edit: images seem completely broken on sth now, either trying to embed a file directly, or even posting an image link?
Greetings All!
I was curious to know what the best approach is to stacking two 6610 units over a distance, say 150 feet or less? Do they make "transceivers" for the 40gb stacking ports that would allow me to run my own fiber in conduit instead of trying something ridiculous like a longer version of stacking cable? I shudder to think what that would even cost. I have two buildings on my property with 1" empty conduit laid for this sort of thing. If such transceivers exist, what is the part # and a good place (eBay?) to get them from? Also, what fiber (with what connectors) would I run between? I assume I can still use a breakout cable at each end for other devices connected to the switch, while still simultaneously using the 40gb "trunk" ports? Thanks in Advance!
I was curious to know what the best approach is to stacking two 6610 units over a distance, say 150 feet or less? Do they make "transceivers" for the 40gb stacking ports that would allow me to run my own fiber in conduit instead of trying something ridiculous like a longer version of stacking cable? I shudder to think what that would even cost. I have two buildings on my property with 1" empty conduit laid for this sort of thing. If such transceivers exist, what is the part # and a good place (eBay?) to get them from? Also, what fiber (with what connectors) would I run between? I assume I can still use a breakout cable at each end for other devices connected to the switch, while still simultaneously using the 40gb "trunk" ports? Thanks in Advance!
Is there a specific recommendation for using an adapter eg:
https://www.servethehome.com/fs-sfp-10g-t-review-another-sfp-to-10gbase-t-option/
I'm interested in possibly running a docsis 3.1 modem 2.5 gig port to one of the sfp+ ports on a brocade 6450 48p.
After reading a bit it seems like flow control is a concern. Is it a straight forward hurdle to overcome. I did see a reddit thread where A Microtic adapter was used for a 6450 at 2.5 gig
https://www.servethehome.com/fs-sfp-10g-t-review-another-sfp-to-10gbase-t-option/
I'm interested in possibly running a docsis 3.1 modem 2.5 gig port to one of the sfp+ ports on a brocade 6450 48p.
After reading a bit it seems like flow control is a concern. Is it a straight forward hurdle to overcome. I did see a reddit thread where A Microtic adapter was used for a 6450 at 2.5 gig
I had my two ICX 6610 stacked using the 40Gb ports with MPO fiber. About 50 meters (more than 150 feet) so this should be easy.
And yes, if you follow fohdeesha's detailed directions, you can easily use the first two ports for stacking at 40Gb/s each and then break out the other two ports for 8 extra 10Gb/s ports. The only limitation on using the break out ports is you will be using an MPO -> 4xLC break out cable, so you can't connect to 10Base-T cards, only SFP+ "stuff" that can accept an LC optical transceiver.