I'm having a hard time finding my specific scenario anywhere. I currently have a segmented L2 network but it's overtaxing my pfSense firewall. I am switching my Cisco SG300 with an ICX-6610 and want to have my trusted high-speed VLANs routed via L3 on the switch while keeping some on the firewall.
DHCP/DNS is on Windows Server. I know I have to create router interfaces for my 4 trusted VLANs, change DHCP option 3 to the IP of that interface and then it will route on the switch. If I ping a device on VLAN 200-Storage from VLAN 20-Trusted, it works. However, what happens when a device on 20 wants to ping a device on VLAN 30-IoT? The rules on the firewall allow that, but how do I ensure what is NOT going to my L3 VLANs get routed to pfSense?
I have a similar setup (dhcp is on pfsense and primary dns is a debian vm running pihole/unbound). Multiple VLANs, some trusted (home, infrastructure), some not (guest, IOT). I have a transit VLAN between the ICX and pfsense. My trusted VLANs (and only my trusted VLANs) have router interfaces on the ICX. The default gateway on the DHCP server for the trusted VLANs is the switch, while for untrusted VLANs it's pfsense. The default route for the switch is to the IP of the pfsense box on the transit VLAN.
Because only my trusted VLANs have router interfaces, I don't have to deal with any ACLs on the switch--all of my trusted VLANs can access everything, and all access control to/from other VLANs is handled via firewall rules on pfsense. This of course means I only get line speed routing between my trusted VLANs, but I'm OK with that. I rarely move much traffic between trusted/untrusted VLANs, and when I do I get 4-5gb/s across VLANs on my virtualized pfsense box, which is good enough for me.