Drag to reposition cover

Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
H

HaxSmash

New Member
Oct 15, 2020
8
2
3
ArmedAviator said:
Be cautious of the DHCP server on the switch. I'm not sure about the ICX7xxx series, but the ICX6xxx series DHCP server is not authoritative, so some devices will not accept DHCP offers from the switch. I recommend a simple Linux container or VM with ISC DHCP and BIND for a DHCP + DDNS combo. Didn't take too long to set up for my first time using either and works really well.
Click to expand...
I do remember reading that now that you mentioned it. I'll have to setup an ISC server somewhere and go from there. I'll also work some more on trying to get that ip address directly assigned to the port instead of a VE. I might have to wipe the config and start fresh at this point.

Again thats for the tips, it really helped save me some headaches.
 
A

ArmedAviator

Member
May 16, 2020
91
56
18
Kansas
No need to start fresh. You will need to head to your basement with the console cable, however.

Code: 
show vlan ethernet 1/1/1
Go to any VLANs that the port is a member of and remove it from the VLAN.

Example:
Code: 
vlan 25
no untagged eth 1/1/1
If it doesn't let you remove the ethernet interface from the VLAN because it's in dual-mode, then do the following and then redo the above:
Code: 
int eth 1/1/1
no dual-mode 25
When you do show vlan eth 1/1/1, you should only see the port as a member of the Default VLAN (1).

Now that the port is not a member of any VLANs (besides the default), you can assign it an IP address directly and use route-only mode if you so choose (hint: you should).
 
nerdalertdk

nerdalertdk

Fleet Admiral
Mar 9, 2017
228
119
43
::1
Hi guys

Can see a lot of people are having trouble with the whole no vlan on the router

I just setup my edge router and icx switch with multiple vlans and multiple subnets
Is this something people want to look at for reference ?

right now there is no advanced features just vlans and static routing

still need ACL but that’s a different story :)
 
L

layer710

New Member
Sep 23, 2020
3
0
1
Cascadia
nerdalertdk said:
Hi guys

Can see a lot of people are having trouble with the whole no vlan on the router

I just setup my edge router and icx switch with multiple vlans and multiple subnets
Is this something people want to look at for reference ?

right now there is no advanced features just vlans and static routing

still need ACL but that’s a different story :)
Click to expand...
I'm interested! Up until a couple weeks ago I was running a RoaS setup on pfSense with 9 vlans. I recently got a 6450 (now fully updated and licensed) so I'm trying to shake things up and transition into a true L3 design.

Right now I've got an OPNsense vm as my edge router/firewall, and another vm running dns/dhcp. Flat topo (192.168.1.0/24).

I'm currently stuck trying to get a 10g transit link working (aiming for 192.168.10.0/30), but no luck yet. I feel like I'm probably missing something simple. Examples of a proper setup would be a godsend right about now! :D
 
L

layer710

New Member
Sep 23, 2020
3
0
1
Cascadia
ArmedAviator said:
Click to expand...
Thank you.

Over the last few weeks, I feel like I've read just about every page of this thread.. but somehow I missed that post. It was very helpful, answered some questions, and I think I may be on the home stretch.

After doing some more reading and gathering my thoughts, I configured my 10g transit to sit on 10.10.10.0/31, so OPNsense is 10.10.10.0 and the 6450 is 10.10.10.1

My config (please feel free to point out anything that looks wrong):

Code: 
Current configuration:
!
ver 08.0.30tT313
!
stack unit 1
  module 1 icx6450-48p-poe-port-management-module
  module 2 icx6450-sfp-plus-4port-40g-module
  stack-port 1/2/3
!
global-stp
!
!
!
vlan 1 name DEFAULT-VLAN by port
 router-interface ve 1
 spanning-tree 802-1w
!
vlan 10 name TRANSIT by port
 tagged ethe 1/2/1
 router-interface ve 10
 spanning-tree 802-1w
!
vlan 20 name VL20 by port
 tagged ethe 1/1/20                                 
 router-interface ve 20
 spanning-tree 802-1w
!
!
!
!
!
aaa authentication web-server default local
aaa authentication enable default local
aaa authentication login default local
enable telnet authentication
fast port-span exclude ethe 1/2/1
hostname ruckus
ip dhcp-client disable
ip dhcp-server server-identifier 192.168.1.35
ip dns server-address 192.168.1.35
ip route 0.0.0.0/0 10.10.10.0
!
username root password .....
!
!
clock summer-time
clock timezone gmt GMT-08                                         
!
!
ntp
 disable serve
 server 216.239.35.0
 server 216.239.35.4
!
!
!
!
!
interface ethernet 1/1/2
 inline power power-by-class 3
!
interface ethernet 1/1/20
 dual-mode
!
interface ethernet 1/2/1
 dual-mode  10
 no spanning-tree
!
interface ve 1
 ip address 192.168.1.8 255.255.255.0                             
!
interface ve 10
 ip address 10.10.10.1 255.255.255.254
!
interface ve 20
 ip address 192.168.20.2 255.255.255.0
 ip helper-address 1 192.168.1.35
!
!
!
!
!
!
!
!
!
end
I was able to spin up a container on one of my proxmox hosts (using int e 1/1/20) tagged it vlan 20, and let DHCP sort the rest... and holy shit, it was able to pull updates. Did I... just actually get this working? HAH! Feels too good to be true, I'm sure something will time out after I post this....

Now, a nooby question... I currently have three physical interfaces on OPNsense. The default 1g WAN/LAN, and my 10g TRANSIT. LAN is carrying 192.168.1.0/24 (aka my entire network at the moment), but once I begin moving my devices into their respective vlans and 192.168.1.0/24 thins out, what would be the best way to handle that interface? Is there any harm in just leaving the three interfaces up and using the LAN as a management interface? OPNsense is sitting on a DL360 with 4x 1g and 2x 10g, so I don't need the physical port back or anything..
 
A

ArmedAviator

Member
May 16, 2020
91
56
18
Kansas
Glad to hear it helped! I know I was totally lost not long ago and thanks to a whole lot of free time on my hands lately, I am learning alot and want to help others.

Regarding the 1G and 10G links, you can set the 1G link to be a backup link in case your 10G link goes down. I know it's easy to do on the switch using route metric (Administrative Distance on Cisco devices), but not sure how to do it in OPNSense/FreeBSD/Linux.

Here's an example of how you would do this, atleast on the switch. This doesn't work too well without the route metrics set manually on your edge router, though, I'm pretty certain.

10G link on OPNSense: 10.10.10.0/31
10G link (physical port or ve if dealing with VMs) on Switch (ve 10): 10.10.10.1/31
1G link on OPSense: 10.10.10.2/31
1G link (physical port or ve if dealing with VMs) on Switch: 10.10.10.3/31

Switch manual routes:
Code: 
ip route 0.0.0.0/0 10.10.10.0
ip route 0.0.0.0/0 10.10.10.2 5   (the 5 means lower priority than the default of 1 for static routes)
If the 10G link goes down, than the lower priority route over the 1G link takes precedence on the switch. As far as OPNSense....not sure how to set the route metric manually.

Another option is to use OSPF for dynamic routing and that should update the route metrics appropriately and automatically on OPNSense.

EDIT: Just wanted to clarify that gateway weighting is NOT the same as the route metric. Weights are for load-balancing links such as Multi-WAN, but not a fail-over P2P situation like this.

EDIT 2: Regarding tips to your config....

It sounds like you're using a physical system as your edge router (OPNSense). I'm certainly no professional yet, so take my tips with a grain of salt, but I think it's a good idea to remove the VLAN where it's not needed and this would be one of those cases. The VLANs would be necessary for configurations with the edge router (OPSense/pfSense/VyOS, etc.) as a virtual machine.

Instead of making VLAN 10, adding a tagged port to it, then effectively untagging it with dual-mode 10, just add the IP address to the interface itself.

Code: 
no vlan 10

interface ethernet 1/2/1
port-name TRANSIT-10G
ip address 10.10.10.1 255.255.255.254
route-only
port security
  enable
The port security configration makes it so ONLY your physical router can connect to that port - not needed but a helpful security feature incase you mis-wire things in the future. It will auto-populate the secure-mac-address under port security with the currently-connected mac-address and can see it populated in show running-config.
 
Last edited:
  • Like
Reactions: MrGuvernment, Vesalius and layer710
A

ArmedAviator

Member
May 16, 2020
91
56
18
Kansas
Okay, here's a new issue I haven't had yet.

Code: 
SSH@ks-icx-01(config-vif-6)#ipv6 traffic-filter bmc-v6 in
Error: Insufficient hardware resource for binding the V6 ACL bmc-v6 to interface v6.
Here's all of the ACLs configured on the system, and each of them are used on a VE, except the bmc-v6 one.

Code: 
ipv6 access-list iot-v6 
 remark DENY ADMIN ACCESS TO SWITCH                               
 deny tcp any host fe80::768e:f8ff:fee7:b4b0 eq ssh log 
 deny tcp any host fe80::768e:f8ff:fee7:b4b0 eq telnet log 
 deny tcp any host fe80::768e:f8ff:fee7:b4b0 eq http log 
 deny tcp any host fe80::768e:f8ff:fee7:b4b0 eq ssl log 
 deny tcp any host 2605:aaaa:bbbb:7a03::1 eq ssh log 
 deny tcp any host 2605:aaaa:bbbb:7a03::1 eq telnet log 
 deny tcp any host 2605:aaaa:bbbb:7a03::1 eq http log 
 deny tcp any host 2605:aaaa:bbbb:7a03::1 eq ssl log 
 remark ALLOW SAME VLAN TRAFFIC
 permit ipv6 any 2605:aaaa:bbbb:7a03::/64 
 remark ALLOW DHCP
 permit udp any any eq bootps 
 permit udp any any eq bootpc 
 remark ALLOW ICMP
 permit icmp any any 
 remark ALLOW ESTABLISHED TCP TRAFFIC
 permit tcp any any established 
 remark ALLOW DNS REQUESTS
 permit udp any host 2605:aaaa:bbbb:7a26::3 eq dns 
 permit tcp any host 2605:aaaa:bbbb:7a26::3 eq dns 
 permit udp any host 2605:aaaa:bbbb:7a26::5 eq dns 
 permit tcp any host 2605:aaaa:bbbb:7a26::5 eq dns 
 remark ALLOW RETURN OF SNMP TRAFFIC TO LIBRENMS SERVER           
 permit udp any eq snmp host 2605:aaaa:bbbb:7a26::81 
 permit udp any eq snmp-trap host 2605:aaaa:bbbb:7a26::81 
 remark DENY ALL OTHER INTER-VLAN TRAFFIC
 deny ipv6 any 2605:aaaa:bbbb:7a00::/56 log 
 remark ALLOW REMAINING TRAFFIC
 permit ipv6 any any 
!
ipv6 access-list voip-v6 
 remark DENY ADMIN ACCESS TO SWITCH
 deny tcp any host fe80::768e:f8ff:fee7:b4b0 eq ssh log 
 deny tcp any host fe80::768e:f8ff:fee7:b4b0 eq telnet log 
 deny tcp any host fe80::768e:f8ff:fee7:b4b0 eq http log 
 deny tcp any host fe80::768e:f8ff:fee7:b4b0 eq ssl log 
 deny tcp any host 2605:aaaa:bbbb:7a02::1 eq ssh log 
 deny tcp any host 2605:aaaa:bbbb:7a02::1 eq telnet log 
 deny tcp any host 2605:aaaa:bbbb:7a02::1 eq http log 
 deny tcp any host 2605:aaaa:bbbb:7a02::1 eq ssl log 
 remark ALLOW SAME VLAN TRAFFIC
 permit ipv6 any 2605:aaaa:bbbb:7a02::/64 
 remark ALLOW DHCP
 permit udp any any eq bootps 
 permit udp any any eq bootpc 
 remark ALLOW ICMP                                                
 permit icmp any any 
 remark ALLOW ESTABLISHED TCP TRAFFIC
 permit tcp any any established 
 remark ALLOW DNS REQUESTS
 permit udp any host 2605:aaaa:bbbb:7a26::3 eq dns 
 permit tcp any host 2605:aaaa:bbbb:7a26::3 eq dns 
 permit udp any host 2605:aaaa:bbbb:7a26::5 eq dns 
 permit tcp any host 2605:aaaa:bbbb:7a26::5 eq dns 
 remark ALLOW RETURN OF SNMP TRAFFIC TO LIBRENMS SERVER
 permit udp any eq snmp host 2605:aaaa:bbbb:7a26::81 
 permit udp any eq snmp-trap host 2605:aaaa:bbbb:7a26::81 
 remark DENY ALL OTHER INTER-VLAN TRAFFIC
 deny ipv6 any 2605:aaaa:bbbb:7a00::/56 log 
 remark ALLOW REMAINING TRAFFIC
 permit ipv6 any any 
!
ipv6 access-list bmc-v6 
 remark DENY ADMIN ACCESS TO SWITCH
 deny tcp any host fe80::768e:f8ff:fee7:b4b0 eq ssh log 
 deny tcp any host fe80::768e:f8ff:fee7:b4b0 eq telnet log 
 deny tcp any host fe80::768e:f8ff:fee7:b4b0 eq http log 
 deny tcp any host fe80::768e:f8ff:fee7:b4b0 eq ssl log 
 deny tcp any host 2605:aaaa:bbbb:7a06::1 eq ssh log              
 deny tcp any host 2605:aaaa:bbbb:7a06::1 eq telnet log 
 deny tcp any host 2605:aaaa:bbbb:7a06::1 eq http log 
 deny tcp any host 2605:aaaa:bbbb:7a06::1 eq ssl log 
 remark ALLOW SAME VLAN TRAFFIC
 permit ipv6 any 2605:aaaa:bbbb:7a06::/64 
 remark ALLOW DHCP
 permit udp any any eq bootps 
 permit udp any any eq bootpc 
 remark ALLOW ICMP
 permit icmp any any 
 remark ALLOW ESTABLISHED TCP TRAFFIC
 permit tcp any any established 
 remark ALLOW DNS REQUESTS
 permit udp any host 2605:aaaa:bbbb:7a26::3 eq dns 
 permit tcp any host 2605:aaaa:bbbb:7a26::3 eq dns 
 permit udp any host 2605:aaaa:bbbb:7a26::5 eq dns 
 permit tcp any host 2605:aaaa:bbbb:7a26::5 eq dns 
 remark ALLOW RETURN OF SNMP TRAFFIC TO LIBRENMS SERVER
 permit udp any eq snmp host 2605:aaaa:bbbb:7a26::81 
 permit udp any eq snmp-trap host 2605:aaaa:bbbb:7a26::81 
 remark ALLOW IPMI
 permit udp any eq asf-rmcp any 
 remark DENY ALL OTHER INTER-VLAN TRAFFIC                         
 deny ipv6 any 2605:aaaa:bbbb:7a00::/56 log 
 remark ALLOW REMAINING TRAFFIC
 permit ipv6 any any 
!
!
ip access-list extended bmc-v4
 remark DENY ADMIN ACCESS TO SWITCH
 deny tcp any host 10.1.6.1 eq ssh log 
 deny tcp any host 10.1.6.1 eq telnet log 
 deny tcp any host 10.1.6.1 eq http log 
 deny tcp any host 10.1.6.1 eq ssl log 
 remark ALLOW SAME VLAN TRAFFIC
 permit ip any 10.1.6.0 0.0.0.255 
 remark ALLOW DHCP
 permit udp any any eq bootps
 permit udp any any eq bootpc
 remark ALLOW ICMP
 permit icmp any any 
 remark ALLOW ESTABLISHED TCP TRAFFIC
 permit tcp any any established 
 remark ALLOW DNS REQUESTS
 permit udp any host 10.1.26.3 eq dns
 permit tcp any host 10.1.26.3 eq dns                             
 permit udp any host 10.1.26.5 eq dns
 permit tcp any host 10.1.26.5 eq dns
 remark ALLOW RETURN OF SNMP TRAFFIC TO LIBRENMS SERVER
 permit udp any eq snmp host 10.1.26.81 
 permit udp any eq snmp-trap host 10.1.26.81 
 remark ALLOW IPMI
 permit udp any eq asf-rmcp any 
 remark DENY ALL OTHER INTER-VLAN TRAFFIC
 deny ip any 10.0.0.0 0.255.255.255 log 
 remark ALLOW REMAINING TRAFFIC
 permit ip any any 
!
ip access-list extended iot-v4
 remark DENY ADMIN ACCESS TO SWITCH
 deny tcp any host 10.1.3.1 eq ssh log 
 deny tcp any host 10.1.3.1 eq telnet log 
 deny tcp any host 10.1.3.1 eq http log 
 deny tcp any host 10.1.3.1 eq ssl log 
 remark ALLOW SAME VLAN TRAFFIC
 permit ip any 10.1.3.0 0.0.0.255 
 remark ALLOW DHCP
 permit udp any any eq bootps
 permit udp any any eq bootpc                                     
 remark ALLOW ICMP
 permit icmp any any 
 remark ALLOW ESTABLISHED TCP TRAFFIC
 permit tcp any any established 
 remark ALLOW DNS REQUESTS
 permit udp any host 10.1.26.3 eq dns
 permit tcp any host 10.1.26.3 eq dns
 permit udp any host 10.1.26.5 eq dns
 permit tcp any host 10.1.26.5 eq dns
 remark ALLOW RETURN OF SNMP TRAFFIC TO LIBRENMS SERVER
 permit udp any eq snmp host 10.1.26.81 
 permit udp any eq snmp-trap host 10.1.26.81 
 remark ALLOW NVR01 SSDP AND BROADCAST TRAFFIC
 permit udp host 10.1.3.20 host 239.255.255.250 eq 3702
 permit udp host 10.1.3.20 eq 3000 any eq 2000
 permit udp any eq 3702 host 10.1.3.20 
 permit udp any eq 2000 host 10.1.3.20 eq 3000
 remark ALLOW FTP BETWEEN NVR01 AND SYNC01
 permit tcp host 10.1.3.20 host 10.1.26.71 eq ftp
 remark DENY ALL OTHER INTER-VLAN TRAFFIC
 deny ip any 10.0.0.0 0.255.255.255 log 
 remark PERMIT REMAINING TRAFFIC
 permit ip any any                                                
!
ip access-list extended voip-v4
 remark DENY ADMIN ACCESS TO SWITCH                               
 deny tcp any host 10.1.2.1 eq ssh log 
 deny tcp any host 10.1.2.1 eq telnet log 
 deny tcp any host 10.1.2.1 eq http log 
 deny tcp any host 10.1.2.1 eq ssl log 
 remark ALLOW SAME VLAN TRAFFIC
 permit ip any 10.1.2.0 0.0.0.255 
 remark ALLOW DHCP
 permit udp any any eq bootps
 permit udp any any eq bootpc
 remark ALLOW ICMP
 permit icmp any any 
 remark ALLOW ESTABLISHED TCP TRAFFIC
 permit tcp any any established 
 remark ALLOW DNS REQUESTS
 permit udp any host 10.1.26.3 eq dns
 permit tcp any host 10.1.26.3 eq dns
 permit udp any host 10.1.26.5 eq dns
 permit tcp any host 10.1.26.5 eq dns
 remark ALLOW RETURN OF SNMP TRAFFIC TO LIBRENMS SERVER
 permit udp any eq snmp host 10.1.26.81 
 permit udp any eq snmp-trap host 10.1.26.81 
 remark DENY ALL OTHER INTER-VLAN TRAFFIC
 deny ip any 10.0.0.0 0.255.255.255 log                           
 remark ALLOW REMAINING TRAFFIC
 permit ip any any
There's nothing in the logs as to why it's saying that and I can't find any limits on the ACLs, specifically the IPv6 ACLs as I can still add more IPv4 ACLs to VEs no problem.
 
fohdeesha

fohdeesha

Kaini Industries
Nov 20, 2016
2,925
3,480
113
34
fohdeesha.com
ArmedAviator said:
Okay, here's a new issue I haven't had yet.

Code: 
SSH@ks-icx-01(config-vif-6)#ipv6 traffic-filter bmc-v6 in
Error: Insufficient hardware resource for binding the V6 ACL bmc-v6 to interface v6.
Here's all of the ACLs configured on the system, and each of them are used on a VE, except the bmc-v6 one.

Code: 
ipv6 access-list iot-v6
remark DENY ADMIN ACCESS TO SWITCH                             
deny tcp any host fe80::768e:f8ff:fee7:b4b0 eq ssh log
deny tcp any host fe80::768e:f8ff:fee7:b4b0 eq telnet log
deny tcp any host fe80::768e:f8ff:fee7:b4b0 eq http log
deny tcp any host fe80::768e:f8ff:fee7:b4b0 eq ssl log
deny tcp any host 2605:aaaa:bbbb:7a03::1 eq ssh log
deny tcp any host 2605:aaaa:bbbb:7a03::1 eq telnet log
deny tcp any host 2605:aaaa:bbbb:7a03::1 eq http log
deny tcp any host 2605:aaaa:bbbb:7a03::1 eq ssl log
remark ALLOW SAME VLAN TRAFFIC
permit ipv6 any 2605:aaaa:bbbb:7a03::/64
remark ALLOW DHCP
permit udp any any eq bootps
permit udp any any eq bootpc
remark ALLOW ICMP
permit icmp any any
remark ALLOW ESTABLISHED TCP TRAFFIC
permit tcp any any established
remark ALLOW DNS REQUESTS
permit udp any host 2605:aaaa:bbbb:7a26::3 eq dns
permit tcp any host 2605:aaaa:bbbb:7a26::3 eq dns
permit udp any host 2605:aaaa:bbbb:7a26::5 eq dns
permit tcp any host 2605:aaaa:bbbb:7a26::5 eq dns
remark ALLOW RETURN OF SNMP TRAFFIC TO LIBRENMS SERVER         
permit udp any eq snmp host 2605:aaaa:bbbb:7a26::81
permit udp any eq snmp-trap host 2605:aaaa:bbbb:7a26::81
remark DENY ALL OTHER INTER-VLAN TRAFFIC
deny ipv6 any 2605:aaaa:bbbb:7a00::/56 log
remark ALLOW REMAINING TRAFFIC
permit ipv6 any any
!
ipv6 access-list voip-v6
remark DENY ADMIN ACCESS TO SWITCH
deny tcp any host fe80::768e:f8ff:fee7:b4b0 eq ssh log
deny tcp any host fe80::768e:f8ff:fee7:b4b0 eq telnet log
deny tcp any host fe80::768e:f8ff:fee7:b4b0 eq http log
deny tcp any host fe80::768e:f8ff:fee7:b4b0 eq ssl log
deny tcp any host 2605:aaaa:bbbb:7a02::1 eq ssh log
deny tcp any host 2605:aaaa:bbbb:7a02::1 eq telnet log
deny tcp any host 2605:aaaa:bbbb:7a02::1 eq http log
deny tcp any host 2605:aaaa:bbbb:7a02::1 eq ssl log
remark ALLOW SAME VLAN TRAFFIC
permit ipv6 any 2605:aaaa:bbbb:7a02::/64
remark ALLOW DHCP
permit udp any any eq bootps
permit udp any any eq bootpc
remark ALLOW ICMP                                              
permit icmp any any
remark ALLOW ESTABLISHED TCP TRAFFIC
permit tcp any any established
remark ALLOW DNS REQUESTS
permit udp any host 2605:aaaa:bbbb:7a26::3 eq dns
permit tcp any host 2605:aaaa:bbbb:7a26::3 eq dns
permit udp any host 2605:aaaa:bbbb:7a26::5 eq dns
permit tcp any host 2605:aaaa:bbbb:7a26::5 eq dns
remark ALLOW RETURN OF SNMP TRAFFIC TO LIBRENMS SERVER
permit udp any eq snmp host 2605:aaaa:bbbb:7a26::81
permit udp any eq snmp-trap host 2605:aaaa:bbbb:7a26::81
remark DENY ALL OTHER INTER-VLAN TRAFFIC
deny ipv6 any 2605:aaaa:bbbb:7a00::/56 log
remark ALLOW REMAINING TRAFFIC
permit ipv6 any any
!
ipv6 access-list bmc-v6
remark DENY ADMIN ACCESS TO SWITCH
deny tcp any host fe80::768e:f8ff:fee7:b4b0 eq ssh log
deny tcp any host fe80::768e:f8ff:fee7:b4b0 eq telnet log
deny tcp any host fe80::768e:f8ff:fee7:b4b0 eq http log
deny tcp any host fe80::768e:f8ff:fee7:b4b0 eq ssl log
deny tcp any host 2605:aaaa:bbbb:7a06::1 eq ssh log            
deny tcp any host 2605:aaaa:bbbb:7a06::1 eq telnet log
deny tcp any host 2605:aaaa:bbbb:7a06::1 eq http log
deny tcp any host 2605:aaaa:bbbb:7a06::1 eq ssl log
remark ALLOW SAME VLAN TRAFFIC
permit ipv6 any 2605:aaaa:bbbb:7a06::/64
remark ALLOW DHCP
permit udp any any eq bootps
permit udp any any eq bootpc
remark ALLOW ICMP
permit icmp any any
remark ALLOW ESTABLISHED TCP TRAFFIC
permit tcp any any established
remark ALLOW DNS REQUESTS
permit udp any host 2605:aaaa:bbbb:7a26::3 eq dns
permit tcp any host 2605:aaaa:bbbb:7a26::3 eq dns
permit udp any host 2605:aaaa:bbbb:7a26::5 eq dns
permit tcp any host 2605:aaaa:bbbb:7a26::5 eq dns
remark ALLOW RETURN OF SNMP TRAFFIC TO LIBRENMS SERVER
permit udp any eq snmp host 2605:aaaa:bbbb:7a26::81
permit udp any eq snmp-trap host 2605:aaaa:bbbb:7a26::81
remark ALLOW IPMI
permit udp any eq asf-rmcp any
remark DENY ALL OTHER INTER-VLAN TRAFFIC                       
deny ipv6 any 2605:aaaa:bbbb:7a00::/56 log
remark ALLOW REMAINING TRAFFIC
permit ipv6 any any
!
!
ip access-list extended bmc-v4
remark DENY ADMIN ACCESS TO SWITCH
deny tcp any host 10.1.6.1 eq ssh log
deny tcp any host 10.1.6.1 eq telnet log
deny tcp any host 10.1.6.1 eq http log
deny tcp any host 10.1.6.1 eq ssl log
remark ALLOW SAME VLAN TRAFFIC
permit ip any 10.1.6.0 0.0.0.255
remark ALLOW DHCP
permit udp any any eq bootps
permit udp any any eq bootpc
remark ALLOW ICMP
permit icmp any any
remark ALLOW ESTABLISHED TCP TRAFFIC
permit tcp any any established
remark ALLOW DNS REQUESTS
permit udp any host 10.1.26.3 eq dns
permit tcp any host 10.1.26.3 eq dns                           
permit udp any host 10.1.26.5 eq dns
permit tcp any host 10.1.26.5 eq dns
remark ALLOW RETURN OF SNMP TRAFFIC TO LIBRENMS SERVER
permit udp any eq snmp host 10.1.26.81
permit udp any eq snmp-trap host 10.1.26.81
remark ALLOW IPMI
permit udp any eq asf-rmcp any
remark DENY ALL OTHER INTER-VLAN TRAFFIC
deny ip any 10.0.0.0 0.255.255.255 log
remark ALLOW REMAINING TRAFFIC
permit ip any any
!
ip access-list extended iot-v4
remark DENY ADMIN ACCESS TO SWITCH
deny tcp any host 10.1.3.1 eq ssh log
deny tcp any host 10.1.3.1 eq telnet log
deny tcp any host 10.1.3.1 eq http log
deny tcp any host 10.1.3.1 eq ssl log
remark ALLOW SAME VLAN TRAFFIC
permit ip any 10.1.3.0 0.0.0.255
remark ALLOW DHCP
permit udp any any eq bootps
permit udp any any eq bootpc                                   
remark ALLOW ICMP
permit icmp any any
remark ALLOW ESTABLISHED TCP TRAFFIC
permit tcp any any established
remark ALLOW DNS REQUESTS
permit udp any host 10.1.26.3 eq dns
permit tcp any host 10.1.26.3 eq dns
permit udp any host 10.1.26.5 eq dns
permit tcp any host 10.1.26.5 eq dns
remark ALLOW RETURN OF SNMP TRAFFIC TO LIBRENMS SERVER
permit udp any eq snmp host 10.1.26.81
permit udp any eq snmp-trap host 10.1.26.81
remark ALLOW NVR01 SSDP AND BROADCAST TRAFFIC
permit udp host 10.1.3.20 host 239.255.255.250 eq 3702
permit udp host 10.1.3.20 eq 3000 any eq 2000
permit udp any eq 3702 host 10.1.3.20
permit udp any eq 2000 host 10.1.3.20 eq 3000
remark ALLOW FTP BETWEEN NVR01 AND SYNC01
permit tcp host 10.1.3.20 host 10.1.26.71 eq ftp
remark DENY ALL OTHER INTER-VLAN TRAFFIC
deny ip any 10.0.0.0 0.255.255.255 log
remark PERMIT REMAINING TRAFFIC
permit ip any any                                              
!
ip access-list extended voip-v4
remark DENY ADMIN ACCESS TO SWITCH                             
deny tcp any host 10.1.2.1 eq ssh log
deny tcp any host 10.1.2.1 eq telnet log
deny tcp any host 10.1.2.1 eq http log
deny tcp any host 10.1.2.1 eq ssl log
remark ALLOW SAME VLAN TRAFFIC
permit ip any 10.1.2.0 0.0.0.255
remark ALLOW DHCP
permit udp any any eq bootps
permit udp any any eq bootpc
remark ALLOW ICMP
permit icmp any any
remark ALLOW ESTABLISHED TCP TRAFFIC
permit tcp any any established
remark ALLOW DNS REQUESTS
permit udp any host 10.1.26.3 eq dns
permit tcp any host 10.1.26.3 eq dns
permit udp any host 10.1.26.5 eq dns
permit tcp any host 10.1.26.5 eq dns
remark ALLOW RETURN OF SNMP TRAFFIC TO LIBRENMS SERVER
permit udp any eq snmp host 10.1.26.81
permit udp any eq snmp-trap host 10.1.26.81
remark DENY ALL OTHER INTER-VLAN TRAFFIC
deny ip any 10.0.0.0 0.255.255.255 log                         
remark ALLOW REMAINING TRAFFIC
permit ip any any
There's nothing in the logs as to why it's saying that and I can't find any limits on the ACLs, specifically the IPv6 ACLs as I can still add more IPv4 ACLs to VEs no problem.
Click to expand...
Is there already an ipv6 ACL assigned to the VE? try removing the old v6 acl from the VE first, then assigning the new one. If not, it may be TCAM allocation related. the TCAM on these devices is split up into TCAM for routes, ACLs, mac filters etc. you can see the default break down by running " show default values ":

Code: 
ICX1#show default values
sys log buffers:50         mac age time:300 sec       telnet sessions:5

ip arp age:10 min          bootp relay max hops:4     ip ttl:64 hops
ip addr per intf:24

when multicast enabled :
igmp group memb.:260 sec   igmp query:125 sec         hardware drop: enabled

when ospf enabled :
ospf dead:40 sec           ospf hello:10 sec          ospf retrans:5 sec
ospf transit delay:1 sec

when bgp enabled :
bgp local pref.:100        bgp keep alive:60 sec      bgp hold:180 sec
bgp metric:10              bgp local as:1             bgp cluster id:0
bgp ext. distance:20       bgp int. distance:200      bgp local distance:200

System Parameters    Default    Maximum    Current    Configured
ip-arp               4000       64000      4000       4000
ip-static-arp        512        6000       512        512
ip-cache             10000      32768      10000      10000
ip-filter-port       3066       3066       3066       3066
ip-filter-sys        2048       8192       2048       2048
l3-vlan              32         1024       32         32
ip-qos-session       1024       16000      1024       1024
mac                  32768      32768      32768      32768
ip-route             12000      15168      12000      12000
ip-static-route      64         2048       64         64
vlan                 64         4095       64         64
spanning-tree        32         254        32         32
mac-filter-port      16         256        16         16
mac-filter-sys       32         512        32         32
ip-subnet-port       24         128        24         24
session-limit        8192       16384      8192       8192
view                 10         65535      10         10
virtual-interface    255        512        255        255
hw-traffic-condition 896        896        896        896
rmon-entries         1024       32768      1024       1024
igmp-snoop-mcache    512        8192       512        512
mld-snoop-mcache     512        8192       512        512
ip6-route            908        2884       908        908
ip6-static-route     178        576        181        181
ip6-cache            908        2884       908        908
msdp-sa-cache        4096       8192       4096       4096
gre-tunnels          16         64         16         16
hw-ip-route-tcam     16384      16384      16384      16384
ip-vrf               16         16         16         16
ip-route-default-vrf 12000      15168      12000      12000
ip6-route-default-vr 908        2884       908        908
ip-route-vrf         1024       15168      1024       1024
ip6-route-vrf        100        2884       100        100
pim-hw-mcache        1024       6144       1024       1024
pim6-hw-mcache       512        1024       512        512
igmp-snoop-group-add 4096       8192       4096       4096
mld-snoop-group-addr 4096       8192       4096       4096
mac-notification-buf 4000       16000      4000       4000
traffic-policies-sys 1024       1024       1024       1024
dot1x-mka-policy-gro 8          8          8          8
openflow-flow-entrie 1024       12000      1024       1024
openflow-pvlan-entri 40         256        40         40
openflow-unprotected 40         256        40         40
openflow-group-selec 0          120        0          0
openflow-nexthop-ent 0          1024       0          0
max-dhcp-snoop-entri 1024       3072       1024       1024
max-static-inspect-a 512        1024       512        512
I know the hard limit on IPv6 permit/deny statements on the ICX is 1536 per port region, and IPv4 permit/deny statement limit of 3067 per port region, and I don't think you've reached that: ICX 6610 device port regions

So it might be a case of reallocating some TCAM from IP routes for example (I doubt you need 9000) to ACLs/filters instead. I don't remember which one offhand controls IPv6 ACLs, but whichever one it is you would increase it, after decreasing the slots allocated to routes. something like:

Code: 
##for an ICX6610, ICX6450 numbers will probably be very different
##decrease tcam allocated to routes to make some room

system-max ip-route-default-vrf 1000
system-max ip-route-vrf 128
system-max ip-route 1500

system-max ip6-route-default-vrf 100
system-max ip6-route-vrf 30
system-max ip6-route 150



##now increase the slots allocated to filtering, don't know off the top of my head which affects IPv6 ACLs
##I'll take a guess:
system-max ip-filter-sys 4096
system-max ip-qos-session 2048
write mem
exit
reload

If that doesn't do it I can look further, but I'm pretty sure (hopefully) your issue is a simple case of not removing the old IPv6 filter on the VE first
 
A

ArmedAviator

Member
May 16, 2020
91
56
18
Kansas
@fohdeesha ,

Thanks alot for the map down the right path.

Unfortunately it was not an already-configured ipv6 ACL on the ve, but it appears the TCAM was the issue. Here's the changes I made that allowed it to work:

Code: 
system-max ip-filter-sys 4096
system-max l3-vlan 0
system-max ip-qos-session 2048
system-max ip-route 4096
system-max vlan 32
system-max virtual-interface 33
system-max ip-route-default-vrf 1024
system-max ip-route-vrf 128
I find it very interesting how quickly the TCAM filled up with really not that many running IPv6 ACLs. Makes me a bit curious how to better write the rules to be more efficient as I want to add similar inter-VLAN ACLs to all of my VLANs (9 and counting).
 
L

layer710

New Member
Sep 23, 2020
3
0
1
Cascadia
ArmedAviator said:
EDIT 2: Regarding tips to your config....

It sounds like you're using a physical system as your edge router (OPNSense). I'm certainly no professional yet, so take my tips with a grain of salt, but I think it's a good idea to remove the VLAN where it's not needed and this would be one of those cases. The VLANs would be necessary for configurations with the edge router (OPSense/pfSense/VyOS, etc.) as a virtual machine.

Instead of making VLAN 10, adding a tagged port to it, then effectively untagging it with dual-mode 10, just add the IP address to the interface itself.

Code: 
no vlan 10

interface ethernet 1/2/1
port-name TRANSIT-10G
ip address 10.10.10.1 255.255.255.254
route-only
port security
  enable
The port security configration makes it so ONLY your physical router can connect to that port - not needed but a helpful security feature incase you mis-wire things in the future. It will auto-populate the secure-mac-address under port security with the currently-connected mac-address and can see it populated in show running-config.
Click to expand...
That makes sense. However, I didn't mean to mislead you.. most of my rack consists of Proxmox cluster nodes, so OPNsense is indeed virtualized in my case. I wasn't aware of the port security option, though.. still very new to Brocade. Thank you for bringing that to my attention!
 
A

ArmedAviator

Member
May 16, 2020
91
56
18
Kansas
layer710 said:
That makes sense. However, I didn't mean to mislead you.. most of my rack consists of Proxmox cluster nodes, so OPNsense is indeed virtualized in my case. I wasn't aware of the port security option, though.. still very new to Brocade. Thank you for bringing that to my attention!
Click to expand...
Ah, in that case ignore what I said about ditching a VLAN unless you're using PCIe pass-through for the NIC.
 
fohdeesha

fohdeesha

Kaini Industries
Nov 20, 2016
2,925
3,480
113
34
fohdeesha.com
ArmedAviator said:
@fohdeesha ,

Thanks alot for the map down the right path.

Unfortunately it was not an already-configured ipv6 ACL on the ve, but it appears the TCAM was the issue. Here's the changes I made that allowed it to work:

Code: 
system-max ip-filter-sys 4096
system-max l3-vlan 0
system-max ip-qos-session 2048
system-max ip-route 4096
system-max vlan 32
system-max virtual-interface 33
system-max ip-route-default-vrf 1024
system-max ip-route-vrf 128
I find it very interesting how quickly the TCAM filled up with really not that many running IPv6 ACLs. Makes me a bit curious how to better write the rules to be more efficient as I want to add similar inter-VLAN ACLs to all of my VLANs (9 and counting).
Click to expand...
Did that indeed solve it? Are you on a 6450 or 6610? I wonder which variable did it, I would guess the "ip-filter-sys", but even that by default is already 2048 and it definitely doesn't look like you're using anywhere near that. Any reason you cranked "l3-vlan" down to 0? I'm pretty that's the max number of VLANs with a VE interface (any VLAN doing routing), cranking it down would only save 32 tcam slots from what I'm reading
 
A

ArmedAviator

Member
May 16, 2020
91
56
18
Kansas
Yes that did fix it. Even with l3-vlan at 0, everything I need is working.

I remember reading about the L3 VLANs, but they are not the same as vlan by port + router-interfaces. Unfortunately, I can't find it in the documentation at the moment but will have more time later to take a look.

The L2 guide briefly discusses changing the number of routing interfaces by adjusting the virtual-interfaces parameter here.
 
nerdalertdk

nerdalertdk

Fleet Admiral
Mar 9, 2017
228
119
43
::1
Hi guys

how have you solved mDNS routing between subnets ?
I’ve look at IP PIM but don’t think that can solve it?
 
I

infoMatt

Active Member
Apr 16, 2019
222
100
43
nerdalertdk said:
Hi guys

how have you solved mDNS routing between subnets ?
I’ve look at IP PIM but don’t think that can solve it?
Click to expand...
You can't, because the packets have a TTL=1 and so they will expire at the first router. You have to install a reflector like Avahi in a linux VM or similar, and give it an address (and thus an interface, it could be on a trunk) on each VLAN where you want to forward to the queries and replies.
 
H

humm3r1

New Member
Nov 14, 2020
3
0
1
Hello,

Thank you everyone for the wealth of very useful and helpful information and links!

I wanted to seek some advice re: multicast setup.

with a Cisco 3750E stack, I enabled PIM Sparse and pim multicast routing, and had a Cisco WLC and a VM running avahi set up so that with IGMP snooping on the WLC and my SG300 switch connected to the 3750E, I could see chromecast and roku from VLAN 41 (IPTV network) into VLAN 31/32/33 where my clients reside.

I'm getting unexpected results at the moment when setting things up. I've tried setting ip pim sparse on the ve interfaces (31/32/33/41), as well as setting up an RP on VLAN 900 (with ip pim sparse).

What I find so interesting is I seem to be able to find the Roku's on wired, but not on wireless. I can find the Nest/chromecast on my phone wirelessly, but not on wired. Basically, wired = roku but no chromecast, while wireless = chromecast, but no roku. BOTH roku's and the Nest are connected to the same wifi network on VLAN 41, and I am testing from VLAN 32. I find it odd I can see 2/3 devices on wired and not the Nest, and only 1/3 on wireless.


Basic Topology:
Web -> PfSense -> ICX 6610 -> WLC -> AP -> Clients (Wireless)
Web -> PfSense -> ICX 6610 -> SG300 -> Clients (Wired, may also go directly into 6610 and skip SG300 depending on physical location, since SG300 is an office switch to extend ports)

Am I missing anything special here?

I put my config into Pastebin here for sake of easiness. I'd appreciate any guidance you can provide! I'm sure I am just misunderstanding or misconfiguring something here.

pastebin.com

SSH@ICX6610A#sh runCurrent configuration:!ver 08.0.30tT7f3!stack unit - Pastebin.com

Pastebin.com is the number one paste tool since 2002. Pastebin is a website where you can store text online for a set period of time.
pastebin.com pastebin.com

Thanks in advance!



EDIT: This config works fine, spotify on desktop sees all clients, but the roku's on iPhone hide in the airplay menu which I never thought of checking previously.
 
Last edited:
R

Roelf Zomerman

Active Member
Jan 10, 2019
149
28
28
blog.azureinfra.com
and its in!.. now need to silence this thing with same mod that was in this thread already..

UNIT 1: SL 1: ICX6610-48P POE 48-port Management Module
Serial #: BXK3816L033
License: ICX6610_PREM_ROUTER_SOFT_PACKAGE (LID: dzmINGLnFII)
P-ENGINE 0: type E02B, rev 01
P-ENGINE 1: type E02B, rev 01

:)

where would I see if there is a POE module as well? enabling inline power on a port doesnt do much
 
Last edited:
You must log in or register to reply here.
Top Bottom