Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[ArmedAviator]

well that's wild. Never seen that before, I have a bunch of v6 ACLs on a stack of 2 6610's here at home and never ran into any issues. Could you post the problematic config, and what firmware version are you on

I'm running version 08.0.30uT7f3.

Here's the 2 ACLs that caused me issues.

ipv6 access-list iot-v6
remark DENY ADMIN ACCESS TO SWITCH
deny tcp any host 2605:a000:d401:7a03::1 eq ssh log
deny tcp any host 2605:a000:d401:7a03::1 eq telnet log
deny tcp any host 2605:a000:d401:7a03::1 eq http log
deny tcp any host 2605:a000:d401:7a03::1 eq ssl log
remark ALLOW SAME VLAN TRAFFIC
permit ipv6 any 2605:a000:d401:7a03::/64
remark ALLOW DHCP
permit udp any any eq bootps
permit udp any any eq bootpc
remark ALLOW ICMP
permit icmp any any
remark ALLOW ESTABLISHED TCP TRAFFIC
permit tcp any any established
remark ALLOW DNS REQUESTS
permit udp any host 2605:a000:d401:7a26::3 eq dns
permit tcp any host 2605:a000:d401:7a26::3 eq dns
permit udp any host 2605:a000:d401:7a26::5 eq dns
permit tcp any host 2605:a000:d401:7a26::5 eq dns                            
remark ALLOW RETURN OF SNMP TRAFFIC TO LIBRENMS SERVER
permit udp any eq snmp host 2605:a000:d401:7a26::81
permit udp any eq snmp-trap host 2605:a000:d401:7a26::81
remark DENY ALL OTHER INTER-VLAN TRAFFIC
deny ipv6 any 2605:a000:d401:7a00::/56 log
remark ALLOW REMAINING TRAFFIC
permit ipv6 any any
!
ipv6 access-list voip-v6
remark DENY ADMIN ACCESS TO SWITCH
deny tcp any host 2605:a000:d401:7a02::1 eq ssh log
deny tcp any host 2605:a000:d401:7a02::1 eq telnet log
deny tcp any host 2605:a000:d401:7a02::1 eq http log
deny tcp any host 2605:a000:d401:7a02::1 eq ssl log
remark ALLOW SAME VLAN TRAFFIC
permit ipv6 any 2605:a000:d401:7a02::/64
remark ALLOW DHCP
permit udp any any eq bootps
permit udp any any eq bootpc
remark ALLOW ICMP
permit icmp any any
remark ALLOW ESTABLISHED TCP TRAFFIC                            
permit tcp any any established
remark ALLOW DNS REQUESTS
permit udp any host 2605:a000:d401:7a26::3 eq dns
permit tcp any host 2605:a000:d401:7a26::3 eq dns
permit udp any host 2605:a000:d401:7a26::5 eq dns
permit tcp any host 2605:a000:d401:7a26::5 eq dns
remark ALLOW RETURN OF SNMP TRAFFIC TO LIBRENMS SERVER
permit udp any eq snmp host 2605:a000:d401:7a26::81
permit udp any eq snmp-trap host 2605:a000:d401:7a26::81
remark DENY ALL OTHER INTER-VLAN TRAFFIC
deny ipv6 any 2605:a000:d401:7a00::/56 log
remark ALLOW REMAINING TRAFFIC
permit ipv6 any any

EDIT:

I gave it another shot tonight, and so far no problems. I'm testing IPv6 ACLs on these two networks because there's only a handful of devices on them that support IPv6 before I roll out more ACLs to my other VLANs.

 

[ArmedAviator]

Ok so that goes along with my gut feeling that I'm trying to do something that I shouldn't, my ignorance showing.

So If I'm following my options are:

a) Continue to use the USG to define all my vlans, and just use the brocade in more of a dumb layer 2 mode?

b) Scrap the vlans and just use one big flat dumb network (aka give in to my lack of knowledge)

b) Make the USG less aware of the downstream switching, define everything on the brocade except for a simple uplink / default route? I guess i'm missing something here around where the NAT happens and the routing. So much to learn.

I started writing this reply thinking I knew a path forward just to realize I'm further away than I thought. I'll have to search some more for some examples.

Option B (the first one) is closer to what you want, probably. Might as well not tag VLAN traffic between the switch and USG if it might make things easier. I don't tag traffic between my switch and OPNSense just for simplicity's sake.

Try this set up....

USG

• LAN IP: 192.168.123.1/30 (no VLANs)
• *Create static route: 192.168.0.0/16 via 192.168.123.2
• *NAT the whole 192.168.0.0/16 network

* I have no idea how to do this on the USG

Switch

vlan 10 name Trusted
tagged ethernet 1/1/2 to 1/1/24
router-interface ve 10

vlan 20 name Servers
tagged ethernet 1/1/2 to 1/1/24
router-interface ve 20

vlan 30 name Guest
tagged ethernet 1/1/2 to 1/1/24
router-interface ve 30


ip route 0.0.0.0/0 192.168.123.1

interface ethernet 1/1/1
port-name USG-uplink
route-only
ip address 192.168.123.2/30

interface loopback 1
port-name Management
ip address 192.168.0.1/32

interface ve 10
port-name Trusted
ip address 192.168.10.1/24
ip helper-address 192.168.123.1

interface ve 20
port-name Servers
ip address 192.168.20.1/24
ip helper-address 192.168.123.1

interface ve 30
port-name Guest
ip address 192.168.30.1/24
ip helper-address 192.168.123.1

interface ethernet 1/1/2 to 1/1/24
dual-mode 30

This is a basic setup, but something to experiment with.

It's a smart idea to have a management IP on a loopback interface. Is 1/1/1 is down/disabled, than 192.168.123.1 is inaccessible from any other configured switch addresses, but 192.168.0.1 will always be up.

If the USG will not hand out addresses on the non-local networks (anything outside of 192.168.123.0/30, than you must set up a proper DHCP server and adjust the helper-address appropriately.

 

[HaxSmash]

Option B (the first one) is closer to what you want, probably. Might as well not tag VLAN traffic between the switch and USG if it might make things easier. I don't tag traffic between my switch and OPNSense just for simplicity's sake.
...

I did play around with this some, and ended up finding a setup that appears to do most of what I would like. For experimentation, I allocated a /20 block to play with. 192.168.48.0/20


As suggested, in the Unifi controller I configured the following:

- The transit/uplink network on LAN2, as 192.168.123.1/30 no DHCP services.
- Created a static route 192.168.48.0/20, next hop 192.168.123.2

On the Brocade:
- I wasn't able to figure out a way to get it to let me assign an IP directly to e 1/1/1, even without it tagged in any network, and no ip assigned to VE1. I ended up just assigning the 192.168.123.2 IP to VE1, which appears to be working.
- I set the default gateway to 192.168.123.1
- I then created my VLANs wholly within the switch, using the DHCP server built into the switch, as nothing I tried would get the USG to give out IP, since it doesn't know about those VLAN networks.

With things setup like this, I can plug into either my "50" vlan or "60" vlan on the switch, and get the correct IP, and can reach the rest of my network, and outside world correctly. The one oddity is that the Unifi controller lists the IP's as being owned by the MAC of the switch, but that isn't really an issue as far as I can tell.

Thank you so much for your suggestions, they did help guide me towards something that worked!

 

[victimofareload]

Hey everyone, Sort of long time lurker. first time poster. I've been using ebay cisco switches for a few years now and have always been happy. But want to expand my knowledge of other platforms as well as bring 10G to the home lab.

I got a ICX6610-48P-E on the way. Is getting licenses still a thing? Got the switch from a friend of a friend of a friend and I'm pretty sure it has no license on it. But I'll know more when I get my hands on it.

Thanks!

 

[ArmedAviator]

I did play around with this some, and ended up finding a setup that appears to do most of what I would like. For experimentation, I allocated a /20 block to play with. 192.168.48.0/20


As suggested, in the Unifi controller I configured the following:

- The transit/uplink network on LAN2, as 192.168.123.1/30 no DHCP services.
- Created a static route 192.168.48.0/20, next hop 192.168.123.2

On the Brocade:
- I wasn't able to figure out a way to get it to let me assign an IP directly to e 1/1/1, even without it tagged in any network, and no ip assigned to VE1. I ended up just assigning the 192.168.123.2 IP to VE1, which appears to be working.
- I set the default gateway to 192.168.123.1
- I then created my VLANs wholly within the switch, using the DHCP server built into the switch, as nothing I tried would get the USG to give out IP, since it doesn't know about those VLAN networks.

With things setup like this, I can plug into either my "50" vlan or "60" vlan on the switch, and get the correct IP, and can reach the rest of my network, and outside world correctly. The one oddity is that the Unifi controller lists the IP's as being owned by the MAC of the switch, but that isn't really an issue as far as I can tell.

Thank you so much for your suggestions, they did help guide me towards something that worked!

If the ethernet interface is a part of any VLAN, whether tagged or untagged, you can not assign it an IP address or set route-only mode. Using the VE as you are will work and for some cases are necessary - for example, running pfSense/OPSense/VyOS on a virtual machine. For your setup, I'd still recommend ditching the needless VLAN at all for the connection to your USG. Even if you set up a LAG between the USG and switch, you can set route-only and IPv4/IPv6 addresses on the lag (via the LAG's primary-port). route-only just disables L2 traffic on that interface.

Once you get things working, since you're doing this for much of the same reasons I am - learning and home network performance, you might look into port security on the port(s) connected to your router. I don't believe the USG supports OSPF but if you do get your EdgeRouter working I believe that supports OSPF which would make LAN routing more automated vs. the static route.

Here's the pertinent configuration from a test I just did on my switch:

lag test dynamic id 2
ports ethernet 1/1/23 to 1/1/24                               
primary-port 1/1/23
lacp-timeout short
deploy

interface ethernet 1/1/23
route-only
ip address 192.168.123.2 255.255.255.252
no flow-control

Be cautious of the DHCP server on the switch. I'm not sure about the ICX7xxx series, but the ICX6xxx series DHCP server is not authoritative, so some devices will not accept DHCP offers from the switch. I recommend a simple Linux container or VM with ISC DHCP and BIND for a DHCP + DDNS combo. Didn't take too long to set up for my first time using either and works really well.

Hey everyone, Sort of long time lurker. first time poster. I've been using ebay cisco switches for a few years now and have always been happy. But want to expand my knowledge of other platforms as well as bring 10G to the home lab.

I got a ICX6610-48P-E on the way. Is getting licenses still a thing? Got the switch from a friend of a friend of a friend and I'm pretty sure it has no license on it. But I'll know more when I get my hands on it.

Thanks!

Yes, there are still licenses available. See the original post of this thread for information on getting them (still for free).

 

[Vesalius]

Hey everyone, Sort of long time lurker. first time poster. I've been using ebay cisco switches for a few years now and have always been happy. But want to expand my knowledge of other platforms as well as bring 10G to the home lab.

I got a ICX6610-48P-E on the way. Is getting licenses still a thing? Got the switch from a friend of a friend of a friend and I'm pretty sure it has no license on it. But I'll know more when I get my hands on it.

Thanks!

It is still a thing and Note 2 at the top of post 1 still applies.

 

[HaxSmash]

Be cautious of the DHCP server on the switch. I'm not sure about the ICX7xxx series, but the ICX6xxx series DHCP server is not authoritative, so some devices will not accept DHCP offers from the switch. I recommend a simple Linux container or VM with ISC DHCP and BIND for a DHCP + DDNS combo. Didn't take too long to set up for my first time using either and works really well.

I do remember reading that now that you mentioned it. I'll have to setup an ISC server somewhere and go from there. I'll also work some more on trying to get that ip address directly assigned to the port instead of a VE. I might have to wipe the config and start fresh at this point.

Again thats for the tips, it really helped save me some headaches.

 

[ArmedAviator]

No need to start fresh. You will need to head to your basement with the console cable, however.

show vlan ethernet 1/1/1

Go to any VLANs that the port is a member of and remove it from the VLAN.

Example:

vlan 25
no untagged eth 1/1/1

If it doesn't let you remove the ethernet interface from the VLAN because it's in dual-mode, then do the following and then redo the above:

int eth 1/1/1
no dual-mode 25

When you do show vlan eth 1/1/1, you should only see the port as a member of the Default VLAN (1).

Now that the port is not a member of any VLANs (besides the default), you can assign it an IP address directly and use route-only mode if you so choose (hint: you should).

 

[i386]

Wtf
Just noticed the thread "thumbnail

 

[ArmedAviator]

Yeah I noticed that immediately after my last post and lost it laughing bigly for a good 5 minutes.

 

[nerdalertdk]

Hi guys

Can see a lot of people are having trouble with the whole no vlan on the router

I just setup my edge router and icx switch with multiple vlans and multiple subnets
Is this something people want to look at for reference ?

right now there is no advanced features just vlans and static routing

still need ACL but that’s a different story

 

[layer710]

Hi guys

Can see a lot of people are having trouble with the whole no vlan on the router

I just setup my edge router and icx switch with multiple vlans and multiple subnets
Is this something people want to look at for reference ?

right now there is no advanced features just vlans and static routing

still need ACL but that’s a different story

I'm interested! Up until a couple weeks ago I was running a RoaS setup on pfSense with 9 vlans. I recently got a 6450 (now fully updated and licensed) so I'm trying to shake things up and transition into a true L3 design.

Right now I've got an OPNsense vm as my edge router/firewall, and another vm running dns/dhcp. Flat topo (192.168.1.0/24).

I'm currently stuck trying to get a 10g transit link working (aiming for 192.168.10.0/30), but no luck yet. I feel like I'm probably missing something simple. Examples of a proper setup would be a godsend right about now!

 

[ArmedAviator]

I'm currently stuck trying to get a 10g transit link working (aiming for 192.168.10.0/30), but no luck yet. I feel like I'm probably missing something simple. Examples of a proper setup would be a godsend right about now!

https://forums.servethehome.com/index.php?threads/brocade-icx-series-cheap-powerful-10gbe-40gbe-switching.21107/page-228#post-283781

 

[layer710]

https://forums.servethehome.com/index.php?threads/brocade-icx-series-cheap-powerful-10gbe-40gbe-switching.21107/page-228#post-283781

Thank you.

Over the last few weeks, I feel like I've read just about every page of this thread.. but somehow I missed that post. It was very helpful, answered some questions, and I think I may be on the home stretch.

After doing some more reading and gathering my thoughts, I configured my 10g transit to sit on 10.10.10.0/31, so OPNsense is 10.10.10.0 and the 6450 is 10.10.10.1

My config (please feel free to point out anything that looks wrong):

Current configuration:
!
ver 08.0.30tT313
!
stack unit 1
  module 1 icx6450-48p-poe-port-management-module
  module 2 icx6450-sfp-plus-4port-40g-module
  stack-port 1/2/3
!
global-stp
!
!
!
vlan 1 name DEFAULT-VLAN by port
 router-interface ve 1
 spanning-tree 802-1w
!
vlan 10 name TRANSIT by port
 tagged ethe 1/2/1
 router-interface ve 10
 spanning-tree 802-1w
!
vlan 20 name VL20 by port
 tagged ethe 1/1/20                                 
 router-interface ve 20
 spanning-tree 802-1w
!
!
!
!
!
aaa authentication web-server default local
aaa authentication enable default local
aaa authentication login default local
enable telnet authentication
fast port-span exclude ethe 1/2/1
hostname ruckus
ip dhcp-client disable
ip dhcp-server server-identifier 192.168.1.35
ip dns server-address 192.168.1.35
ip route 0.0.0.0/0 10.10.10.0
!
username root password .....
!
!
clock summer-time
clock timezone gmt GMT-08                                         
!
!
ntp
 disable serve
 server 216.239.35.0
 server 216.239.35.4
!
!
!
!
!
interface ethernet 1/1/2
 inline power power-by-class 3
!
interface ethernet 1/1/20
 dual-mode
!
interface ethernet 1/2/1
 dual-mode  10
 no spanning-tree
!
interface ve 1
 ip address 192.168.1.8 255.255.255.0                             
!
interface ve 10
 ip address 10.10.10.1 255.255.255.254
!
interface ve 20
 ip address 192.168.20.2 255.255.255.0
 ip helper-address 1 192.168.1.35
!
!
!
!
!
!
!
!
!
end

I was able to spin up a container on one of my proxmox hosts (using int e 1/1/20) tagged it vlan 20, and let DHCP sort the rest... and holy shit, it was able to pull updates. Did I... just actually get this working? HAH! Feels too good to be true, I'm sure something will time out after I post this....

Now, a nooby question... I currently have three physical interfaces on OPNsense. The default 1g WAN/LAN, and my 10g TRANSIT. LAN is carrying 192.168.1.0/24 (aka my entire network at the moment), but once I begin moving my devices into their respective vlans and 192.168.1.0/24 thins out, what would be the best way to handle that interface? Is there any harm in just leaving the three interfaces up and using the LAN as a management interface? OPNsense is sitting on a DL360 with 4x 1g and 2x 10g, so I don't need the physical port back or anything..

 

[ArmedAviator]

Glad to hear it helped! I know I was totally lost not long ago and thanks to a whole lot of free time on my hands lately, I am learning alot and want to help others.

Regarding the 1G and 10G links, you can set the 1G link to be a backup link in case your 10G link goes down. I know it's easy to do on the switch using route metric (Administrative Distance on Cisco devices), but not sure how to do it in OPNSense/FreeBSD/Linux.

Here's an example of how you would do this, atleast on the switch. This doesn't work too well without the route metrics set manually on your edge router, though, I'm pretty certain.

10G link on OPNSense: 10.10.10.0/31
10G link (physical port or ve if dealing with VMs) on Switch (ve 10): 10.10.10.1/31
1G link on OPSense: 10.10.10.2/31
1G link (physical port or ve if dealing with VMs) on Switch: 10.10.10.3/31

Switch manual routes:

ip route 0.0.0.0/0 10.10.10.0
ip route 0.0.0.0/0 10.10.10.2 5   (the 5 means lower priority than the default of 1 for static routes)

If the 10G link goes down, than the lower priority route over the 1G link takes precedence on the switch. As far as OPNSense....not sure how to set the route metric manually.

Another option is to use OSPF for dynamic routing and that should update the route metrics appropriately and automatically on OPNSense.

EDIT: Just wanted to clarify that gateway weighting is NOT the same as the route metric. Weights are for load-balancing links such as Multi-WAN, but not a fail-over P2P situation like this.

EDIT 2: Regarding tips to your config....

It sounds like you're using a physical system as your edge router (OPNSense). I'm certainly no professional yet, so take my tips with a grain of salt, but I think it's a good idea to remove the VLAN where it's not needed and this would be one of those cases. The VLANs would be necessary for configurations with the edge router (OPSense/pfSense/VyOS, etc.) as a virtual machine.

Instead of making VLAN 10, adding a tagged port to it, then effectively untagging it with dual-mode 10, just add the IP address to the interface itself.

no vlan 10

interface ethernet 1/2/1
port-name TRANSIT-10G
ip address 10.10.10.1 255.255.255.254
route-only
port security
  enable

The port security configration makes it so ONLY your physical router can connect to that port - not needed but a helpful security feature incase you mis-wire things in the future. It will auto-populate the secure-mac-address under port security with the currently-connected mac-address and can see it populated in show running-config.

 

[ArmedAviator]

Okay, here's a new issue I haven't had yet.

SSH@ks-icx-01(config-vif-6)#ipv6 traffic-filter bmc-v6 in
Error: Insufficient hardware resource for binding the V6 ACL bmc-v6 to interface v6.

Here's all of the ACLs configured on the system, and each of them are used on a VE, except the bmc-v6 one.

ipv6 access-list iot-v6 
 remark DENY ADMIN ACCESS TO SWITCH                               
 deny tcp any host fe80::768e:f8ff:fee7:b4b0 eq ssh log 
 deny tcp any host fe80::768e:f8ff:fee7:b4b0 eq telnet log 
 deny tcp any host fe80::768e:f8ff:fee7:b4b0 eq http log 
 deny tcp any host fe80::768e:f8ff:fee7:b4b0 eq ssl log 
 deny tcp any host 2605:aaaa:bbbb:7a03::1 eq ssh log 
 deny tcp any host 2605:aaaa:bbbb:7a03::1 eq telnet log 
 deny tcp any host 2605:aaaa:bbbb:7a03::1 eq http log 
 deny tcp any host 2605:aaaa:bbbb:7a03::1 eq ssl log 
 remark ALLOW SAME VLAN TRAFFIC
 permit ipv6 any 2605:aaaa:bbbb:7a03::/64 
 remark ALLOW DHCP
 permit udp any any eq bootps 
 permit udp any any eq bootpc 
 remark ALLOW ICMP
 permit icmp any any 
 remark ALLOW ESTABLISHED TCP TRAFFIC
 permit tcp any any established 
 remark ALLOW DNS REQUESTS
 permit udp any host 2605:aaaa:bbbb:7a26::3 eq dns 
 permit tcp any host 2605:aaaa:bbbb:7a26::3 eq dns 
 permit udp any host 2605:aaaa:bbbb:7a26::5 eq dns 
 permit tcp any host 2605:aaaa:bbbb:7a26::5 eq dns 
 remark ALLOW RETURN OF SNMP TRAFFIC TO LIBRENMS SERVER           
 permit udp any eq snmp host 2605:aaaa:bbbb:7a26::81 
 permit udp any eq snmp-trap host 2605:aaaa:bbbb:7a26::81 
 remark DENY ALL OTHER INTER-VLAN TRAFFIC
 deny ipv6 any 2605:aaaa:bbbb:7a00::/56 log 
 remark ALLOW REMAINING TRAFFIC
 permit ipv6 any any 
!
ipv6 access-list voip-v6 
 remark DENY ADMIN ACCESS TO SWITCH
 deny tcp any host fe80::768e:f8ff:fee7:b4b0 eq ssh log 
 deny tcp any host fe80::768e:f8ff:fee7:b4b0 eq telnet log 
 deny tcp any host fe80::768e:f8ff:fee7:b4b0 eq http log 
 deny tcp any host fe80::768e:f8ff:fee7:b4b0 eq ssl log 
 deny tcp any host 2605:aaaa:bbbb:7a02::1 eq ssh log 
 deny tcp any host 2605:aaaa:bbbb:7a02::1 eq telnet log 
 deny tcp any host 2605:aaaa:bbbb:7a02::1 eq http log 
 deny tcp any host 2605:aaaa:bbbb:7a02::1 eq ssl log 
 remark ALLOW SAME VLAN TRAFFIC
 permit ipv6 any 2605:aaaa:bbbb:7a02::/64 
 remark ALLOW DHCP
 permit udp any any eq bootps 
 permit udp any any eq bootpc 
 remark ALLOW ICMP                                                
 permit icmp any any 
 remark ALLOW ESTABLISHED TCP TRAFFIC
 permit tcp any any established 
 remark ALLOW DNS REQUESTS
 permit udp any host 2605:aaaa:bbbb:7a26::3 eq dns 
 permit tcp any host 2605:aaaa:bbbb:7a26::3 eq dns 
 permit udp any host 2605:aaaa:bbbb:7a26::5 eq dns 
 permit tcp any host 2605:aaaa:bbbb:7a26::5 eq dns 
 remark ALLOW RETURN OF SNMP TRAFFIC TO LIBRENMS SERVER
 permit udp any eq snmp host 2605:aaaa:bbbb:7a26::81 
 permit udp any eq snmp-trap host 2605:aaaa:bbbb:7a26::81 
 remark DENY ALL OTHER INTER-VLAN TRAFFIC
 deny ipv6 any 2605:aaaa:bbbb:7a00::/56 log 
 remark ALLOW REMAINING TRAFFIC
 permit ipv6 any any 
!
ipv6 access-list bmc-v6 
 remark DENY ADMIN ACCESS TO SWITCH
 deny tcp any host fe80::768e:f8ff:fee7:b4b0 eq ssh log 
 deny tcp any host fe80::768e:f8ff:fee7:b4b0 eq telnet log 
 deny tcp any host fe80::768e:f8ff:fee7:b4b0 eq http log 
 deny tcp any host fe80::768e:f8ff:fee7:b4b0 eq ssl log 
 deny tcp any host 2605:aaaa:bbbb:7a06::1 eq ssh log              
 deny tcp any host 2605:aaaa:bbbb:7a06::1 eq telnet log 
 deny tcp any host 2605:aaaa:bbbb:7a06::1 eq http log 
 deny tcp any host 2605:aaaa:bbbb:7a06::1 eq ssl log 
 remark ALLOW SAME VLAN TRAFFIC
 permit ipv6 any 2605:aaaa:bbbb:7a06::/64 
 remark ALLOW DHCP
 permit udp any any eq bootps 
 permit udp any any eq bootpc 
 remark ALLOW ICMP
 permit icmp any any 
 remark ALLOW ESTABLISHED TCP TRAFFIC
 permit tcp any any established 
 remark ALLOW DNS REQUESTS
 permit udp any host 2605:aaaa:bbbb:7a26::3 eq dns 
 permit tcp any host 2605:aaaa:bbbb:7a26::3 eq dns 
 permit udp any host 2605:aaaa:bbbb:7a26::5 eq dns 
 permit tcp any host 2605:aaaa:bbbb:7a26::5 eq dns 
 remark ALLOW RETURN OF SNMP TRAFFIC TO LIBRENMS SERVER
 permit udp any eq snmp host 2605:aaaa:bbbb:7a26::81 
 permit udp any eq snmp-trap host 2605:aaaa:bbbb:7a26::81 
 remark ALLOW IPMI
 permit udp any eq asf-rmcp any 
 remark DENY ALL OTHER INTER-VLAN TRAFFIC                         
 deny ipv6 any 2605:aaaa:bbbb:7a00::/56 log 
 remark ALLOW REMAINING TRAFFIC
 permit ipv6 any any 
!
!
ip access-list extended bmc-v4
 remark DENY ADMIN ACCESS TO SWITCH
 deny tcp any host 10.1.6.1 eq ssh log 
 deny tcp any host 10.1.6.1 eq telnet log 
 deny tcp any host 10.1.6.1 eq http log 
 deny tcp any host 10.1.6.1 eq ssl log 
 remark ALLOW SAME VLAN TRAFFIC
 permit ip any 10.1.6.0 0.0.0.255 
 remark ALLOW DHCP
 permit udp any any eq bootps
 permit udp any any eq bootpc
 remark ALLOW ICMP
 permit icmp any any 
 remark ALLOW ESTABLISHED TCP TRAFFIC
 permit tcp any any established 
 remark ALLOW DNS REQUESTS
 permit udp any host 10.1.26.3 eq dns
 permit tcp any host 10.1.26.3 eq dns                             
 permit udp any host 10.1.26.5 eq dns
 permit tcp any host 10.1.26.5 eq dns
 remark ALLOW RETURN OF SNMP TRAFFIC TO LIBRENMS SERVER
 permit udp any eq snmp host 10.1.26.81 
 permit udp any eq snmp-trap host 10.1.26.81 
 remark ALLOW IPMI
 permit udp any eq asf-rmcp any 
 remark DENY ALL OTHER INTER-VLAN TRAFFIC
 deny ip any 10.0.0.0 0.255.255.255 log 
 remark ALLOW REMAINING TRAFFIC
 permit ip any any 
!
ip access-list extended iot-v4
 remark DENY ADMIN ACCESS TO SWITCH
 deny tcp any host 10.1.3.1 eq ssh log 
 deny tcp any host 10.1.3.1 eq telnet log 
 deny tcp any host 10.1.3.1 eq http log 
 deny tcp any host 10.1.3.1 eq ssl log 
 remark ALLOW SAME VLAN TRAFFIC
 permit ip any 10.1.3.0 0.0.0.255 
 remark ALLOW DHCP
 permit udp any any eq bootps
 permit udp any any eq bootpc                                     
 remark ALLOW ICMP
 permit icmp any any 
 remark ALLOW ESTABLISHED TCP TRAFFIC
 permit tcp any any established 
 remark ALLOW DNS REQUESTS
 permit udp any host 10.1.26.3 eq dns
 permit tcp any host 10.1.26.3 eq dns
 permit udp any host 10.1.26.5 eq dns
 permit tcp any host 10.1.26.5 eq dns
 remark ALLOW RETURN OF SNMP TRAFFIC TO LIBRENMS SERVER
 permit udp any eq snmp host 10.1.26.81 
 permit udp any eq snmp-trap host 10.1.26.81 
 remark ALLOW NVR01 SSDP AND BROADCAST TRAFFIC
 permit udp host 10.1.3.20 host 239.255.255.250 eq 3702
 permit udp host 10.1.3.20 eq 3000 any eq 2000
 permit udp any eq 3702 host 10.1.3.20 
 permit udp any eq 2000 host 10.1.3.20 eq 3000
 remark ALLOW FTP BETWEEN NVR01 AND SYNC01
 permit tcp host 10.1.3.20 host 10.1.26.71 eq ftp
 remark DENY ALL OTHER INTER-VLAN TRAFFIC
 deny ip any 10.0.0.0 0.255.255.255 log 
 remark PERMIT REMAINING TRAFFIC
 permit ip any any                                                
!
ip access-list extended voip-v4
 remark DENY ADMIN ACCESS TO SWITCH                               
 deny tcp any host 10.1.2.1 eq ssh log 
 deny tcp any host 10.1.2.1 eq telnet log 
 deny tcp any host 10.1.2.1 eq http log 
 deny tcp any host 10.1.2.1 eq ssl log 
 remark ALLOW SAME VLAN TRAFFIC
 permit ip any 10.1.2.0 0.0.0.255 
 remark ALLOW DHCP
 permit udp any any eq bootps
 permit udp any any eq bootpc
 remark ALLOW ICMP
 permit icmp any any 
 remark ALLOW ESTABLISHED TCP TRAFFIC
 permit tcp any any established 
 remark ALLOW DNS REQUESTS
 permit udp any host 10.1.26.3 eq dns
 permit tcp any host 10.1.26.3 eq dns
 permit udp any host 10.1.26.5 eq dns
 permit tcp any host 10.1.26.5 eq dns
 remark ALLOW RETURN OF SNMP TRAFFIC TO LIBRENMS SERVER
 permit udp any eq snmp host 10.1.26.81 
 permit udp any eq snmp-trap host 10.1.26.81 
 remark DENY ALL OTHER INTER-VLAN TRAFFIC
 deny ip any 10.0.0.0 0.255.255.255 log                           
 remark ALLOW REMAINING TRAFFIC
 permit ip any any

There's nothing in the logs as to why it's saying that and I can't find any limits on the ACLs, specifically the IPv6 ACLs as I can still add more IPv4 ACLs to VEs no problem.

 

[fohdeesha]

Okay, here's a new issue I haven't had yet.

SSH@ks-icx-01(config-vif-6)#ipv6 traffic-filter bmc-v6 in
Error: Insufficient hardware resource for binding the V6 ACL bmc-v6 to interface v6.

Here's all of the ACLs configured on the system, and each of them are used on a VE, except the bmc-v6 one.

ipv6 access-list iot-v6
remark DENY ADMIN ACCESS TO SWITCH                             
deny tcp any host fe80::768e:f8ff:fee7:b4b0 eq ssh log
deny tcp any host fe80::768e:f8ff:fee7:b4b0 eq telnet log
deny tcp any host fe80::768e:f8ff:fee7:b4b0 eq http log
deny tcp any host fe80::768e:f8ff:fee7:b4b0 eq ssl log
deny tcp any host 2605:aaaa:bbbb:7a03::1 eq ssh log
deny tcp any host 2605:aaaa:bbbb:7a03::1 eq telnet log
deny tcp any host 2605:aaaa:bbbb:7a03::1 eq http log
deny tcp any host 2605:aaaa:bbbb:7a03::1 eq ssl log
remark ALLOW SAME VLAN TRAFFIC
permit ipv6 any 2605:aaaa:bbbb:7a03::/64
remark ALLOW DHCP
permit udp any any eq bootps
permit udp any any eq bootpc
remark ALLOW ICMP
permit icmp any any
remark ALLOW ESTABLISHED TCP TRAFFIC
permit tcp any any established
remark ALLOW DNS REQUESTS
permit udp any host 2605:aaaa:bbbb:7a26::3 eq dns
permit tcp any host 2605:aaaa:bbbb:7a26::3 eq dns
permit udp any host 2605:aaaa:bbbb:7a26::5 eq dns
permit tcp any host 2605:aaaa:bbbb:7a26::5 eq dns
remark ALLOW RETURN OF SNMP TRAFFIC TO LIBRENMS SERVER         
permit udp any eq snmp host 2605:aaaa:bbbb:7a26::81
permit udp any eq snmp-trap host 2605:aaaa:bbbb:7a26::81
remark DENY ALL OTHER INTER-VLAN TRAFFIC
deny ipv6 any 2605:aaaa:bbbb:7a00::/56 log
remark ALLOW REMAINING TRAFFIC
permit ipv6 any any
!
ipv6 access-list voip-v6
remark DENY ADMIN ACCESS TO SWITCH
deny tcp any host fe80::768e:f8ff:fee7:b4b0 eq ssh log
deny tcp any host fe80::768e:f8ff:fee7:b4b0 eq telnet log
deny tcp any host fe80::768e:f8ff:fee7:b4b0 eq http log
deny tcp any host fe80::768e:f8ff:fee7:b4b0 eq ssl log
deny tcp any host 2605:aaaa:bbbb:7a02::1 eq ssh log
deny tcp any host 2605:aaaa:bbbb:7a02::1 eq telnet log
deny tcp any host 2605:aaaa:bbbb:7a02::1 eq http log
deny tcp any host 2605:aaaa:bbbb:7a02::1 eq ssl log
remark ALLOW SAME VLAN TRAFFIC
permit ipv6 any 2605:aaaa:bbbb:7a02::/64
remark ALLOW DHCP
permit udp any any eq bootps
permit udp any any eq bootpc
remark ALLOW ICMP                                              
permit icmp any any
remark ALLOW ESTABLISHED TCP TRAFFIC
permit tcp any any established
remark ALLOW DNS REQUESTS
permit udp any host 2605:aaaa:bbbb:7a26::3 eq dns
permit tcp any host 2605:aaaa:bbbb:7a26::3 eq dns
permit udp any host 2605:aaaa:bbbb:7a26::5 eq dns
permit tcp any host 2605:aaaa:bbbb:7a26::5 eq dns
remark ALLOW RETURN OF SNMP TRAFFIC TO LIBRENMS SERVER
permit udp any eq snmp host 2605:aaaa:bbbb:7a26::81
permit udp any eq snmp-trap host 2605:aaaa:bbbb:7a26::81
remark DENY ALL OTHER INTER-VLAN TRAFFIC
deny ipv6 any 2605:aaaa:bbbb:7a00::/56 log
remark ALLOW REMAINING TRAFFIC
permit ipv6 any any
!
ipv6 access-list bmc-v6
remark DENY ADMIN ACCESS TO SWITCH
deny tcp any host fe80::768e:f8ff:fee7:b4b0 eq ssh log
deny tcp any host fe80::768e:f8ff:fee7:b4b0 eq telnet log
deny tcp any host fe80::768e:f8ff:fee7:b4b0 eq http log
deny tcp any host fe80::768e:f8ff:fee7:b4b0 eq ssl log
deny tcp any host 2605:aaaa:bbbb:7a06::1 eq ssh log            
deny tcp any host 2605:aaaa:bbbb:7a06::1 eq telnet log
deny tcp any host 2605:aaaa:bbbb:7a06::1 eq http log
deny tcp any host 2605:aaaa:bbbb:7a06::1 eq ssl log
remark ALLOW SAME VLAN TRAFFIC
permit ipv6 any 2605:aaaa:bbbb:7a06::/64
remark ALLOW DHCP
permit udp any any eq bootps
permit udp any any eq bootpc
remark ALLOW ICMP
permit icmp any any
remark ALLOW ESTABLISHED TCP TRAFFIC
permit tcp any any established
remark ALLOW DNS REQUESTS
permit udp any host 2605:aaaa:bbbb:7a26::3 eq dns
permit tcp any host 2605:aaaa:bbbb:7a26::3 eq dns
permit udp any host 2605:aaaa:bbbb:7a26::5 eq dns
permit tcp any host 2605:aaaa:bbbb:7a26::5 eq dns
remark ALLOW RETURN OF SNMP TRAFFIC TO LIBRENMS SERVER
permit udp any eq snmp host 2605:aaaa:bbbb:7a26::81
permit udp any eq snmp-trap host 2605:aaaa:bbbb:7a26::81
remark ALLOW IPMI
permit udp any eq asf-rmcp any
remark DENY ALL OTHER INTER-VLAN TRAFFIC                       
deny ipv6 any 2605:aaaa:bbbb:7a00::/56 log
remark ALLOW REMAINING TRAFFIC
permit ipv6 any any
!
!
ip access-list extended bmc-v4
remark DENY ADMIN ACCESS TO SWITCH
deny tcp any host 10.1.6.1 eq ssh log
deny tcp any host 10.1.6.1 eq telnet log
deny tcp any host 10.1.6.1 eq http log
deny tcp any host 10.1.6.1 eq ssl log
remark ALLOW SAME VLAN TRAFFIC
permit ip any 10.1.6.0 0.0.0.255
remark ALLOW DHCP
permit udp any any eq bootps
permit udp any any eq bootpc
remark ALLOW ICMP
permit icmp any any
remark ALLOW ESTABLISHED TCP TRAFFIC
permit tcp any any established
remark ALLOW DNS REQUESTS
permit udp any host 10.1.26.3 eq dns
permit tcp any host 10.1.26.3 eq dns                           
permit udp any host 10.1.26.5 eq dns
permit tcp any host 10.1.26.5 eq dns
remark ALLOW RETURN OF SNMP TRAFFIC TO LIBRENMS SERVER
permit udp any eq snmp host 10.1.26.81
permit udp any eq snmp-trap host 10.1.26.81
remark ALLOW IPMI
permit udp any eq asf-rmcp any
remark DENY ALL OTHER INTER-VLAN TRAFFIC
deny ip any 10.0.0.0 0.255.255.255 log
remark ALLOW REMAINING TRAFFIC
permit ip any any
!
ip access-list extended iot-v4
remark DENY ADMIN ACCESS TO SWITCH
deny tcp any host 10.1.3.1 eq ssh log
deny tcp any host 10.1.3.1 eq telnet log
deny tcp any host 10.1.3.1 eq http log
deny tcp any host 10.1.3.1 eq ssl log
remark ALLOW SAME VLAN TRAFFIC
permit ip any 10.1.3.0 0.0.0.255
remark ALLOW DHCP
permit udp any any eq bootps
permit udp any any eq bootpc                                   
remark ALLOW ICMP
permit icmp any any
remark ALLOW ESTABLISHED TCP TRAFFIC
permit tcp any any established
remark ALLOW DNS REQUESTS
permit udp any host 10.1.26.3 eq dns
permit tcp any host 10.1.26.3 eq dns
permit udp any host 10.1.26.5 eq dns
permit tcp any host 10.1.26.5 eq dns
remark ALLOW RETURN OF SNMP TRAFFIC TO LIBRENMS SERVER
permit udp any eq snmp host 10.1.26.81
permit udp any eq snmp-trap host 10.1.26.81
remark ALLOW NVR01 SSDP AND BROADCAST TRAFFIC
permit udp host 10.1.3.20 host 239.255.255.250 eq 3702
permit udp host 10.1.3.20 eq 3000 any eq 2000
permit udp any eq 3702 host 10.1.3.20
permit udp any eq 2000 host 10.1.3.20 eq 3000
remark ALLOW FTP BETWEEN NVR01 AND SYNC01
permit tcp host 10.1.3.20 host 10.1.26.71 eq ftp
remark DENY ALL OTHER INTER-VLAN TRAFFIC
deny ip any 10.0.0.0 0.255.255.255 log
remark PERMIT REMAINING TRAFFIC
permit ip any any                                              
!
ip access-list extended voip-v4
remark DENY ADMIN ACCESS TO SWITCH                             
deny tcp any host 10.1.2.1 eq ssh log
deny tcp any host 10.1.2.1 eq telnet log
deny tcp any host 10.1.2.1 eq http log
deny tcp any host 10.1.2.1 eq ssl log
remark ALLOW SAME VLAN TRAFFIC
permit ip any 10.1.2.0 0.0.0.255
remark ALLOW DHCP
permit udp any any eq bootps
permit udp any any eq bootpc
remark ALLOW ICMP
permit icmp any any
remark ALLOW ESTABLISHED TCP TRAFFIC
permit tcp any any established
remark ALLOW DNS REQUESTS
permit udp any host 10.1.26.3 eq dns
permit tcp any host 10.1.26.3 eq dns
permit udp any host 10.1.26.5 eq dns
permit tcp any host 10.1.26.5 eq dns
remark ALLOW RETURN OF SNMP TRAFFIC TO LIBRENMS SERVER
permit udp any eq snmp host 10.1.26.81
permit udp any eq snmp-trap host 10.1.26.81
remark DENY ALL OTHER INTER-VLAN TRAFFIC
deny ip any 10.0.0.0 0.255.255.255 log                         
remark ALLOW REMAINING TRAFFIC
permit ip any any

There's nothing in the logs as to why it's saying that and I can't find any limits on the ACLs, specifically the IPv6 ACLs as I can still add more IPv4 ACLs to VEs no problem.

Is there already an ipv6 ACL assigned to the VE? try removing the old v6 acl from the VE first, then assigning the new one. If not, it may be TCAM allocation related. the TCAM on these devices is split up into TCAM for routes, ACLs, mac filters etc. you can see the default break down by running " show default values ":

ICX1#show default values
sys log buffers:50         mac age time:300 sec       telnet sessions:5

ip arp age:10 min          bootp relay max hops:4     ip ttl:64 hops
ip addr per intf:24

when multicast enabled :
igmp group memb.:260 sec   igmp query:125 sec         hardware drop: enabled

when ospf enabled :
ospf dead:40 sec           ospf hello:10 sec          ospf retrans:5 sec
ospf transit delay:1 sec

when bgp enabled :
bgp local pref.:100        bgp keep alive:60 sec      bgp hold:180 sec
bgp metric:10              bgp local as:1             bgp cluster id:0
bgp ext. distance:20       bgp int. distance:200      bgp local distance:200

System Parameters    Default    Maximum    Current    Configured
ip-arp               4000       64000      4000       4000
ip-static-arp        512        6000       512        512
ip-cache             10000      32768      10000      10000
ip-filter-port       3066       3066       3066       3066
ip-filter-sys        2048       8192       2048       2048
l3-vlan              32         1024       32         32
ip-qos-session       1024       16000      1024       1024
mac                  32768      32768      32768      32768
ip-route             12000      15168      12000      12000
ip-static-route      64         2048       64         64
vlan                 64         4095       64         64
spanning-tree        32         254        32         32
mac-filter-port      16         256        16         16
mac-filter-sys       32         512        32         32
ip-subnet-port       24         128        24         24
session-limit        8192       16384      8192       8192
view                 10         65535      10         10
virtual-interface    255        512        255        255
hw-traffic-condition 896        896        896        896
rmon-entries         1024       32768      1024       1024
igmp-snoop-mcache    512        8192       512        512
mld-snoop-mcache     512        8192       512        512
ip6-route            908        2884       908        908
ip6-static-route     178        576        181        181
ip6-cache            908        2884       908        908
msdp-sa-cache        4096       8192       4096       4096
gre-tunnels          16         64         16         16
hw-ip-route-tcam     16384      16384      16384      16384
ip-vrf               16         16         16         16
ip-route-default-vrf 12000      15168      12000      12000
ip6-route-default-vr 908        2884       908        908
ip-route-vrf         1024       15168      1024       1024
ip6-route-vrf        100        2884       100        100
pim-hw-mcache        1024       6144       1024       1024
pim6-hw-mcache       512        1024       512        512
igmp-snoop-group-add 4096       8192       4096       4096
mld-snoop-group-addr 4096       8192       4096       4096
mac-notification-buf 4000       16000      4000       4000
traffic-policies-sys 1024       1024       1024       1024
dot1x-mka-policy-gro 8          8          8          8
openflow-flow-entrie 1024       12000      1024       1024
openflow-pvlan-entri 40         256        40         40
openflow-unprotected 40         256        40         40
openflow-group-selec 0          120        0          0
openflow-nexthop-ent 0          1024       0          0
max-dhcp-snoop-entri 1024       3072       1024       1024
max-static-inspect-a 512        1024       512        512

I know the hard limit on IPv6 permit/deny statements on the ICX is 1536 per port region, and IPv4 permit/deny statement limit of 3067 per port region, and I don't think you've reached that: ICX 6610 device port regions

So it might be a case of reallocating some TCAM from IP routes for example (I doubt you need 9000) to ACLs/filters instead. I don't remember which one offhand controls IPv6 ACLs, but whichever one it is you would increase it, after decreasing the slots allocated to routes. something like:

##for an ICX6610, ICX6450 numbers will probably be very different
##decrease tcam allocated to routes to make some room

system-max ip-route-default-vrf 1000
system-max ip-route-vrf 128
system-max ip-route 1500

system-max ip6-route-default-vrf 100
system-max ip6-route-vrf 30
system-max ip6-route 150



##now increase the slots allocated to filtering, don't know off the top of my head which affects IPv6 ACLs
##I'll take a guess:
system-max ip-filter-sys 4096
system-max ip-qos-session 2048
write mem
exit
reload

If that doesn't do it I can look further, but I'm pretty sure (hopefully) your issue is a simple case of not removing the old IPv6 filter on the VE first

 

[ArmedAviator]

@fohdeesha ,

Thanks alot for the map down the right path.

Unfortunately it was not an already-configured ipv6 ACL on the ve, but it appears the TCAM was the issue. Here's the changes I made that allowed it to work:

system-max ip-filter-sys 4096
system-max l3-vlan 0
system-max ip-qos-session 2048
system-max ip-route 4096
system-max vlan 32
system-max virtual-interface 33
system-max ip-route-default-vrf 1024
system-max ip-route-vrf 128

I find it very interesting how quickly the TCAM filled up with really not that many running IPv6 ACLs. Makes me a bit curious how to better write the rules to be more efficient as I want to add similar inter-VLAN ACLs to all of my VLANs (9 and counting).

 

[layer710]

EDIT 2: Regarding tips to your config....

It sounds like you're using a physical system as your edge router (OPNSense). I'm certainly no professional yet, so take my tips with a grain of salt, but I think it's a good idea to remove the VLAN where it's not needed and this would be one of those cases. The VLANs would be necessary for configurations with the edge router (OPSense/pfSense/VyOS, etc.) as a virtual machine.

Instead of making VLAN 10, adding a tagged port to it, then effectively untagging it with dual-mode 10, just add the IP address to the interface itself.

no vlan 10

interface ethernet 1/2/1
port-name TRANSIT-10G
ip address 10.10.10.1 255.255.255.254
route-only
port security
  enable

The port security configration makes it so ONLY your physical router can connect to that port - not needed but a helpful security feature incase you mis-wire things in the future. It will auto-populate the secure-mac-address under port security with the currently-connected mac-address and can see it populated in show running-config.

That makes sense. However, I didn't mean to mislead you.. most of my rack consists of Proxmox cluster nodes, so OPNsense is indeed virtualized in my case. I wasn't aware of the port security option, though.. still very new to Brocade. Thank you for bringing that to my attention!

 

[ArmedAviator]

That makes sense. However, I didn't mean to mislead you.. most of my rack consists of Proxmox cluster nodes, so OPNsense is indeed virtualized in my case. I wasn't aware of the port security option, though.. still very new to Brocade. Thank you for bringing that to my attention!

Ah, in that case ignore what I said about ditching a VLAN unless you're using PCIe pass-through for the NIC.