Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[ArmedAviator]

From experience, do not use pfSense as your router and DHCP server. Keep pfSense unaware of your VLANs. Use a point-to-point VLAN if pfSense is virtualized -OR- dedicate a port/lag to pfSense if it's a physical device.

Use your L3 switch as your internal router and pfSense as your edge router/firewall/NAT only so it will only receive internet-bound traffic.

With the ICX64R (L3) firmware, you will want to add a router-interface (ve) to each of the VLANs that need to be routed at all. Give each VE an IP address and appropriate subnet, and use that IP address as any connected device's Default Gateway (aka next-hop) address on each VLAN respectively.

Look up my post history and you'll find a good detailed post on how I have a similar setup with some good helpful info in this very thread.

 

[MrGuvernment]

Thank you @ArmedAviator for that info, never thought to do things that way at all! I presume I then also manage all of the ACL's with in the ICX also to limit access to systems between VLANS. I guess I was thinking as PFSense to manage all of this as it is very familiar and easy to do (spoiled by GUIs). On the other hand, i also do not want to be spending the next 2 weeks figuring our CLI syntax to create everything switch side. I also do a lot of IP reservations as well my home lab which is a lot of windows boxes and stuff. Managing it all from PfSense would be easier for me in the end with out needing to become a networking expert.

 

[tommybackeast]

From experience, do not use pfSense as your router and DHCP server. Keep pfSense unaware of your VLANs. Use a point-to-point VLAN if pfSense is virtualized -OR- dedicate a port/lag to pfSense if it's a physical device.

Use your L3 switch as your internal router and pfSense as your edge router/firewall/NAT only so it will only receive internet-bound traffic.

With the ICX64R (L3) firmware, you will want to add a router-interface (ve) to each of the VLANs that need to be routed at all. Give each VE an IP address and appropriate subnet, and use that IP address as any connected device's Default Gateway (aka next-hop) address on each VLAN respectively.

Look up my post history and you'll find a good detailed post on how I have a similar setup with some good helpful info in this very thread.

Gently asking: might you have a URL handy ? (of your prior detailed post you mentioned here), thank you

 

[dragonian]

Thank you @ArmedAviator for that info, never thought to do things that way at all! I presume I then also manage all of the ACL's with in the ICX also to limit access to systems between VLANS. I guess I was thinking as PFSense to manage all of this as it is very familiar and easy to do (spoiled by GUIs). On the other hand, i also do not want to be spending the next 2 weeks figuring our CLI syntax to create everything switch side. I also do a lot of IP reservations as well my home lab which is a lot of windows boxes and stuff. Managing it all from PfSense would be easier for me in the end with out needing to become a networking expert.

Using the ICX as the L3 router to handle all vlans and ACLs is the *proper* way, aka the way it's intended to be used. That being said, there are quite a few folks here that are simply using it as a L2 switch, and doing inter-VLAN routing on pfsense or whatever.

There are pros & cons of either approach. These are my opinions:

Approach	Pro	Con
L3 Routing	Wire speed between vlans Robust ACLs support	ICX DHCP server is not authoritative- need another box for DHCP Learning curve for CLI
L2 Switch with pfsense (etc)	fancy GUI lots of examples firewall/ DHCP/ DNS all in one place	not wire speed - can't do 10 GBit routing in SW

Personally I'm still using the L2 switch with OPNsense, as this was the way I had it setup before i acquired the ICX. It was easy to drop in the new switch and change some vlan tags, add some 10 Git DACs for servers and I'm off running. I have considered switching, but the DHCP sever challenge is my biggest hangup. I don't really want another box to "break the internet".

I don't know why your previous attempt didn't work, but it definitely can, and there are lot of working setups that do it that way.

 

[Vesalius]

Question for the experts, as this is my first layer 3 switch I am really digging into. I my ICX 6450-24p flashed with the R firmware. Now, do I have to use VLANS in a layer 3 sense? using virtual interfaces and such as per the docs I have found.

I am trying to set up my VLANS, using PFSense as my router and have the VLANS set up there. I am going from PFSense ---> 6450 via a single link. On the ICX6450, I have "tagged" the VLAN's I have in pfsense on the port 1/2/1 (10Gbps Fiber) with the default vlan 1 untagged.

However when I set say port 1/1/1 to VLAN 2 (wireless and tagged) as it goes to my Asus AP - no traffic appears to pass through at all and clioent can not get an DHCP address from PFSense. My last switch was an HP procurve and this set up worked.

Or do I need a DHCP / IP Helper set up on the ICX6450?

You almost certainly need to set 1/1/1 to untagged vlan2. The 6450 will then accept all untagged traffic coming into 1/1/1 and place it in Vlan2. That traffic will then go across your 1/2/1 port as tagged in vlan2. Would be shocked if the asus AP was smart enough to understand vlans and tag the traffic itself. As it stands now the 1/1/1 port is blocking traffic from your AP because it is not tagged vlan2 already. Try that first and your setup may just work as is. I ran mine as you described for quite some time.

Are you doing inter-VLAN routing that requires high speed? if not and things are simply segregated into vlans then there really is no need for you to go down the rabbit hole of setting up separate DHCP servers and ACLs for the 6450. You can just tag all the vlans on the 6450 switch interface connected to pfsense, then use the pfsense gui to set up everything.

 

[Vesalius]

Gently asking: might you have a URL handy ? (of your prior detailed post you mentioned here), thank you

The below should get you started.

https://forums.servethehome.com/index.php?posts/283978/

https://forums.servethehome.com/index.php?posts/285032/

https://forums.servethehome.com/index.php?posts/283781/

 

[tommybackeast]

The below should get you started.

https://forums.servethehome.com/index.php?posts/283978/

https://forums.servethehome.com/index.php?posts/285032/

https://forums.servethehome.com/index.php?posts/283781/

thank you

 

[MrGuvernment]

Appreciate the responses. Part of me is just for ease of use, but getting wire speed across VLANS would be nice for my lab and storage. But the other side of me does want to do this "properly", just me and keeping my patience to get it up and running!

Will post back with any progress or road blocks!

 

[ArmedAviator]

Thanks for providing those links, @Vesalius . I was wayyyyyy too tired last night and was up well past bed time responding already lol.

Using the ICX as the L3 router to handle all vlans and ACLs is the *proper* way, aka the way it's intended to be used. That being said, there are quite a few folks here that are simply using it as a L2 switch, and doing inter-VLAN routing on pfsense or whatever.

There are pros & cons of either approach. These are my opinions:

Approach	Pro	Con
L3 Routing	Wire speed between vlans Robust ACLs support	ICX DHCP server is not authoritative- need another box for DHCP Learning curve for CLI
L2 Switch with pfsense (etc)	fancy GUI lots of examples firewall/ DHCP/ DNS all in one place	not wire speed - can't do 10 GBit routing in SW

Personally I'm still using the L2 switch with OPNsense, as this was the way I had it setup before i acquired the ICX. It was easy to drop in the new switch and change some vlan tags, add some 10 Git DACs for servers and I'm off running. I have considered switching, but the DHCP sever challenge is my biggest hangup. I don't really want another box to "break the internet".

I don't know why your previous attempt didn't work, but it definitely can, and there are lot of working setups that do it that way.

This is all very true. I also started out with 10G Netgear L3 switches and these Brocades as L2-only due to being easier and what I knew how to do. With just a few hours of research of finding example setups, asking questions here, and trial-and-error, I am way happier with using the L3 switch as an L3 switch and move DHCP from pfSense to a virtual machine running the ISC DHCP server and BIND DNS server. It's convenient being able to reboot pfSense (OPNSense in my case) and the entire LAN runs without issue - DNS/DHCP still works, all VLAN routing still works, etc.

My biggest hangup I've been having is the miserable experience of rolling out IPv6 on my LAN. Specifically, assigning DNS AAAA records not being automatic because SLAAC and when the ISP gives a new IPv6 /56 I have to change static IPs on various servers and switches.

 

[Vesalius]

@ArmedAviator happy to help even if only a little. Your post and Kapones helped me sort things out from clueless, to hey this finally works. Have it running for a couple vlans, but have not yet moved my WORK vlan/vpn over to the transit network, being completely isolated it would not really benefit and I am leaving well enough alone for now. I have those 3 post bookmarked under my user account so they were only 1 click away for me.

 

[MrGuvernment]

Just to check, if i want to use L2 for now, i do have to re-flash with the L2 S firmware instead of the R firmware?

 

[infoMatt]

Just to check, if i want to use L2 for now, i do have to re-flash with the L2 S firmware instead of the R firmware?

No, that's not needed. Just don't configure a VE on VLANs that you don't want to route with the switch.

 

[ArmedAviator]

Just to check, if i want to use L2 for now, i do have to re-flash with the L2 S firmware instead of the R firmware?

As @infoMatt said, no you do not, but if you're going to L2-only it might be worth a try. I decided to run my ICX6450 with the S image. It did make a few things easier. No VEs, just give it an IP, set up the VLANs, and you're done. It would be good idea to add management-vlan in the VLAN you'll use for management access - the same thing you'd do by adding a VE and an IP to the VE in the R image.

 

[MrGuvernment]

Good to know, I was also just reading over - https://forums.servethehome.com/index.php?threads/layer-3-switch-w-pfsense.23236/

And i guess i did not think about some things as I just assumed they would work the same (DHCP / DNS). While i love the idea of going L3 properly, with work so busy right now and other stuff, my brain i dont think right now is in the state to set it all up how i want it. Surely if i gave it a weekend and had the time i could get it all sorted based on the great notes and guides on this site people like you both have done and info provided.

 

[Vesalius]

Good to know, I was also just reading over - https://forums.servethehome.com/index.php?threads/layer-3-switch-w-pfsense.23236/

And i guess i did not think about some things as I just assumed they would work the same (DHCP / DNS). While i love the idea of going L3 properly, with work so busy right now and other stuff, my brain i dont think right now is in the state to set it all up how i want it. Surely if i gave it a weekend and had the time i could get it all sorted based on the great notes and guides on this site people like you both have done and info provided.

It definitely takes an investment in time to set up for those unsure and/or inexperienced. I know it did for me. if you have a smart home with mdns traffic and WiFi printers and such might require some maintenance after setup to get it right. Even the op of the thread you linked ultimately switched back and even flattened his lan to allow for easier connectivity.

took Me a bit to get the nerve to try. I set up my guest vlan to use a transit link first to prove I could make it work. Also took some reading and research to set up the kea-dhcp vm I decided to use, which required a detour into virtualization and proxmox and I learned a ton. next moved the gaming lan over, but stopped there and have not touched my work lan or general lan. Not sure how the switch will handle all the iOt traffic nor do I want to risk the wrath of my family as internet access is a lifeline to work, school and friends during this pandemic.

 

[MrGuvernment]

Def something that interests me and knowing meI will start playing with it cause i want to suck every last MB of performance out of my network now so I know things will just fly! Which means doing the proper Brocade config.....but i do love the ease of use in pfsense to do everything from rules, and filters to pfblockerNG and monitoring traffic between networks to see what is trying to do what and what should not be doing what..


I just did a quick iperf test to my pfsense box, and while reading that pfsense iperf somewhat sucks, i was only able to hit about1Gbps from my lab to pfsense which goes lab--- 10Gbps---> Brocade ---- 10Gbps--> Pfsense.

So ordered the 2 other DAC's i need to split my lab up between my 2 10Gbps dual port cards and will test throughput that way. Considering the way over kill server my pfsense is on right now (physical) it should easily be able to handle 10Gbps even if it does do all the VLAN routing....

The other weird part is uploading image files from my desktop (VLAN 1 basically) to my lab box

PC ---1Gb--netgear 5 port switch-----1Gbps---> brocade- 10Gbps--->pfsense to VLAN 5 ---> 10GBps to Lab box

I am only getting like 30-50MB/s through put...I had this problem before as well and i was not even hair pinning my VLANS but performance just sucked, one of the reasons i decided to jump on the 10Gbps train! But copying files to my QNAP from my desktop i can max out at 112MB/s steady. Almost feels like PFSense did something to kill VLAN performance trying to make you buy their devices instead or something.

 

[kapone]

Nope. pfSense didn't do anything you didn't tell it to... *cough* ...user error.

 

[ArmedAviator]

Def something that interests me and knowing meI will start playing with it cause i want to suck every last MB of performance out of my network now so I know things will just fly! Which means doing the proper Brocade config.....but i do love the ease of use in pfsense to do everything from rules, and filters to pfblockerNG and monitoring traffic between networks to see what is trying to do what and what should not be doing what..

You can log deny rules in ACLs. It may not be in a web GUI, but it's very easy to spot something that's trying to access something it isn't. It's also useful when something should access another VLAN but rules are blocking it. It's a simple as show log.

An example:

SSH@ks-icx-01#show log
Syslog logging: enabled ( 12809 messages dropped, 0 flushes, 1 overruns)
Buffer logging: level ACDMENW, 500 messages logged
level code: A=alert C=critical D=debugging M=emergency E=error
I=informational N=notification W=warning

Dynamic Log Buffer (500 lines):
Dec 5 07:08:24:W:ACL: ACL: List cam-v4 denied udp 10.1.3.27(19905)( v3 ec71.db7e.f9e6) -> 10.1.10.10(0), 2 event(s)
Dec 5 07:07:24:W:ACL: ACL: List cam-v4 denied udp 10.1.3.27(34877)( v3 ec71.db7e.f9e6) -> 10.1.10.10(0), 2 event(s)
Dec 5 07:06:24:W:ACL: ACL: List cam-v4 denied udp 10.1.3.27(14346)( v3 ec71.db7e.f9e6) -> 10.1.10.10(0), 2 event(s)
Dec 5 07:06:24:W:ACL: ACL: List cam-v4 denied udp 10.1.3.27(55421)( v3 ec71.db7e.f9e6) -> 10.1.10.10(0), 2 event(s)
Dec 5 07:06:24:W:ACL: ACL: List cam-v4 denied udp 10.1.3.27(49365)( v3 ec71.db7e.f9e6) -> 10.1.10.10(0), 2 event(s)

Just like pfSense does, shows me the ACL name or number it's hitting on (cam-v4 in this case), the protocol, the source IP address and port, VLAN ID and MAC address of source, destination IP address(port), and how many events are related.

For those things that need to stay denied but keep attempting connections, I just don't log those specific rules in the ACLs so it doesn't clutter up the log.

I just did a quick iperf test to my pfsense box, and while reading that pfsense iperf somewhat sucks, i was only able to hit about1Gbps from my lab to pfsense which goes lab--- 10Gbps---> Brocade ---- 10Gbps--> Pfsense.

So ordered the 2 other DAC's i need to split my lab up between my 2 10Gbps dual port cards and will test throughput that way. Considering the way over kill server my pfsense is on right now (physical) it should easily be able to handle 10Gbps even if it does do all the VLAN routing....

The other weird part is uploading image files from my desktop (VLAN 1 basically) to my lab box

PC ---1Gb--netgear 5 port switch-----1Gbps---> brocade- 10Gbps--->pfsense to VLAN 5 ---> 10GBps to Lab box

I am only getting like 30-50MB/s through put...I had this problem before as well and i was not even hair pinning my VLANS but performance just sucked, one of the reasons i decided to jump on the 10Gbps train! But copying files to my QNAP from my desktop i can max out at 112MB/s steady. Almost feels like PFSense did something to kill VLAN performance trying to make you buy their devices instead or something.

From my previous experiences with a similar router-on-a-stick design, pfSense had to be tuned to get near 10G speeds even on way overkill hardware. I since converted that overkill hardware to be my third Proxmox node to form a cluster and virtualized OPNSense on it. I can live migrate the OPNSense and DHCP/DNS virtual machines when taking down a hypervisor for maintenance. Redundancy, and reduced down time (yes, this is just in a home and I'm the only one here 60% of the time) for a few hours of initial migration from router-on-a-stick to L3 routing on switch and a DHCP/DNS virtual machine configured and running. Ease of maintenance is about the same to me - more CLI and config file editing and less GUI use, but it's not complicated.

 

[PGlover]

How do I isolate my "ioT" devices from my home lan?

I am running pfSense and have the ICX6610 switch as well. For wireless, I have a number of Ruckus R720 APs.

I am trying to figure out the least complicated and most secure why of isolating my wired and wireless ioT devices using the network infrastructure I currently own or have setup (pfSense, ICX6610, Ruckus APs).

I currently have a number of VLANs created for my home setup on the ICX6610 switch. All of my routing is performed on my Brocade switch. I have a Windows 2016 server that serves as my DHCP server.

Please provide some guidance and direction on the best approach.

If you need more detail on my setup, I can provide that information as well.

Thanks

 

[ArmedAviator]

You need to create access-lists for any VLANs that you want to prohibit or limit access to other VLANs. These will be on your ICX6610 doing the routing.

Here are some of the IPv4 ACLs I have in place on my ICX6610, although I'm no guru and have just been adding and changing them through trial and error. The IPv6 ACL are pretty much exact copies with the exception of the addresses.

ip access-list extended bmc-v4
remark DENY ADMIN ACCESS TO SWITCH
deny tcp any host 10.1.6.1 eq ssh log
deny tcp any host 10.1.6.1 eq telnet log
deny tcp any host 10.1.6.1 eq http log
deny tcp any host 10.1.6.1 eq ssl log
remark ALLOW SAME VLAN TRAFFIC
permit ip any 10.1.6.0 0.0.0.255
remark ALLOW DHCP
permit udp any any eq bootps
permit udp any any eq bootpc
remark ALLOW ICMP
permit icmp any any
remark ALLOW ESTABLISHED TCP TRAFFIC
permit tcp any any established
remark ALLOW DNS REQUESTS
permit udp any host 10.1.26.3 eq dns
permit tcp any host 10.1.26.3 eq dns
permit udp any host 10.1.26.5 eq dns
permit tcp any host 10.1.26.5 eq dns
remark ALLOW RETURN OF SNMP TRAFFIC TO LIBRENMS SERVER
permit udp any eq snmp host 10.1.26.81
permit udp any eq snmp-trap host 10.1.26.81
remark ALLOW IPMI
permit udp any eq asf-rmcp any
remark DENY ALL OTHER INTER-VLAN TRAFFIC
deny ip any 10.0.0.0 0.255.255.255 log
remark ALLOW REMAINING TRAFFIC
permit ip any any
enable-accounting
!
ip access-list extended cam-v4
remark DENY ADMIN ACCESS TO SWITCH
deny tcp any host 10.1.3.1 eq ssh log
deny tcp any host 10.1.3.1 eq telnet log
deny tcp any host 10.1.3.1 eq http log
deny tcp any host 10.1.3.1 eq ssl log
remark REOLINK PROPERIETARY VIDEO STREAM DSCP
permit tcp any eq 9000 any dscp-marking 32
permit tcp any any eq 9000 dscp-marking 32
remark RTSP DSCP
permit tcp any eq rtsp any dscp-marking 32
permit tcp any any eq rtsp dscp-marking 32
remark RTP VIDEO DSCP
permit udp any range 6790 6999 any dscp-marking 32
permit udp any any range 6790 6999 dscp-marking 32
remark ALLOW SAME VLAN TRAFFIC
permit ip any 10.1.3.0 0.0.0.255
remark ALLOW DHCP
permit udp any any eq bootps
permit udp any any eq bootpc
remark ALLOW ICMP
permit icmp any any
remark ALLOW ESTABLISHED TCP TRAFFIC
permit tcp any any established
remark ALLOW DNS REQUESTS
permit udp any host 10.1.26.3 eq dns
permit tcp any host 10.1.26.3 eq dns
permit udp any host 10.1.26.5 eq dns
permit tcp any host 10.1.26.5 eq dns
remark ALLOW RETURN OF SNMP TRAFFIC TO LIBRENMS SERVER
permit udp any eq snmp host 10.1.26.81
permit udp any eq snmp-trap host 10.1.26.81
remark ALLOW NVR01 SSDP AND BROADCAST TRAFFIC
permit udp host 10.1.3.20 host 239.255.255.250 eq 3702
permit udp host 10.1.3.20 eq 3000 any eq 2000
permit udp any eq 3702 host 10.1.3.20
permit udp any eq 2000 host 10.1.3.20 eq 3000
remark ALLOW NVR01 STREAM TO CLI VLAN
permit udp host 10.1.3.20 10.1.10.0 0.0.0.255
remark ALLOW FTP TO SYNC01
permit tcp any host 10.1.26.71 eq ftp
remark DENY ALL OTHER INTER-VLAN TRAFFIC
deny ip any 10.0.0.0 0.255.255.255 log
remark PERMIT REMAINING TRAFFIC
permit ip any any
enable-accounting
!
ip access-list extended iot-v4
remark DENY ADMIN ACCESS TO SWITCH
deny tcp any host 10.1.7.1 eq ssh log
deny tcp any host 10.1.7.1 eq telnet log
deny tcp any host 10.1.7.1 eq http log
deny tcp any host 10.1.7.1 eq ssl log
remark ALLOW SAME VLAN TRAFFIC
permit ip any 10.1.7.0 0.0.0.255
remark ALLOW DHCP
permit udp any any eq bootps
permit udp any any eq bootpc
remark ALLOW ICMP
permit icmp any any
remark ALLOW ESTABLISHED TCP TRAFFIC
permit tcp any any established
remark ALLOW DNS REQUESTS
permit udp any host 10.1.26.3 eq dns
permit tcp any host 10.1.26.3 eq dns
permit udp any host 10.1.26.5 eq dns
permit tcp any host 10.1.26.5 eq dns
remark ALLOW RETURN OF SNMP TRAFFIC TO LIBRENMS SERVER
permit udp any eq snmp host 10.1.26.81
permit udp any eq snmp-trap host 10.1.26.81
remark ALLOW SNMP FROM KS-PRNT-01 TO ANY
permit udp host 10.1.7.31 eq snmp any
remark DENY ALL OTHER INTER-VLAN TRAFFIC
deny ip any 10.0.0.0 0.255.255.255 log
remark PERMIT REMAINING TRAFFIC
permit ip any any
enable-accounting
!
ip access-list extended voip-v4
remark DENY ADMIN ACCESS TO SWITCH
deny tcp any host 10.1.2.1 eq ssh log
deny tcp any host 10.1.2.1 eq telnet log
deny tcp any host 10.1.2.1 eq http log
deny tcp any host 10.1.2.1 eq ssl log
remark ALLOW SAME VLAN TRAFFIC
permit ip any 10.1.2.0 0.0.0.255
remark ALLOW DHCP
permit udp any any eq bootps
permit udp any any eq bootpc
remark ALLOW ICMP
permit icmp any any
remark ALLOW ESTABLISHED TCP TRAFFIC
permit tcp any any established
remark ALLOW DNS REQUESTS
permit udp any host 10.1.26.3 eq dns
permit tcp any host 10.1.26.3 eq dns
permit udp any host 10.1.26.5 eq dns
permit tcp any host 10.1.26.5 eq dns
remark ALLOW RETURN OF SNMP TRAFFIC TO LIBRENMS SERVER
permit udp any eq snmp host 10.1.26.81
permit udp any eq snmp-trap host 10.1.26.81
remark DENY ALL OTHER INTER-VLAN TRAFFIC
deny ip any 10.0.0.0 0.255.255.255 log
remark ALLOW REMAINING TRAFFIC
permit ip any any
enable-accounting

Once you set up the ACLs needed, add them to the appropriate VE with

int ve 123
ip access-group acl-name-here in
ipv6 traffic-filter acl-name-here in (*if adding an IPv6 ACL also)

You can only do one IPv4 and one IPv6 ACL per VE, as far as I understand. You want to filter ingress traffic to the switch, not out from the switch. Not every VE needs an ACL, just the ones that you want to limit access to other VLANs.

You can use the ACLs to prevent internet access by removing the permit ip any any from the last line but I find it's easier to just let the edge firewall do it's job so I only need to add a permit rule in one place instead of two if I ever need to open up a port. It's important to remember that just like most firewall appliances and software, the ACLs imply a deny any any rule at the end and just like any other firewall, order matters.