PfSense Starting Fresh: Advice?

whiskytangofoxconn

New Member
Aug 17, 2021
27
5
3
I bought new hardware and it is finally all here. New pfsense hw is a lenovo tmm m720q with a dual sfp+ card, and two switches, a cisco sx550x-12f (got a really good deal on it) and a sg550x-24p.

That's it. If you had the opportunity to start over, what would you do different?

My old setup was (still is currently) router on a stick. A little aliexpress Intel Celeron 1037u dual-nic machine that was perfect for years and years (no aes-ni). It has a lacp link to a cisco sg300. WAN comes into the sg300 switch, both pfsense nics are lacp link to the switch.

I was planning for the new setup to be a similar router on a stick. The hardware (switches) are overkill but the sx550x was stupid-cheap. I'm seeing a lot about transit networks/links between pfsense and a l3 switch. It's been a REALLY LONG time since I studied for ccna: probably 15 years and I haven't used any of it in the last 10. I feel like I'm more dangerous now with what I have forgotten than if I was ignorant and blindly following a tutorial.

It's just a house, not an AWS DC... But I do work from home and 1.2-1.7gbe speeds would be nice at times. I'd be thrilled with 2.5gbe all the time and I don't need anywhere near 10gbe. Am I foolish for thinking I can keep everything in L2 mode and let pfsense deal with the heavy lifting?
 

zer0sum

Active Member
Mar 8, 2013
527
228
43
First piece of advice is to dump PFsense and go with OPNsense :D

I find it's more fun to run a hypervisor like Proxmox or ESXi and then you can change firewalls in minutes if you feel like it.
If your network card supports SRIOV you can do hardware passthrough which makes things a little faster.

Keep things simple to begin with and add complexity once the basics are working out for you :)
 

Vesalius

Active Member
Nov 25, 2019
155
123
43
Any home network segmentation plans? Intervlan routing at 10g or even greater than 1g a need/want? Keeping the potential 10g traffic on the same lan/vlan makes things easy without l3, transit, pfsense dhcp issues to worry about. Will both of those switches be needed/used? A couple inexpensive 10g DAC cables between the 720q bonded to one or both of those switches would be a good start.
 

whiskytangofoxconn

New Member
Aug 17, 2021
27
5
3
First piece of advice is to dump PFsense and go with OPNsense :D

I find it's more fun to run a hypervisor like Proxmox or ESXi and then you can change firewalls in minutes if you feel like it.
If your network card supports SRIOV you can do hardware passthrough which makes things a little faster.

Keep things simple to begin with and add complexity once the basics are working out for you :)
OK when I started to type this it was "Literally about 5 minutes ago"... But now I've been distracted. I have OPNsense installed and I'm playing with it. I'm going to install pfsense and opnsense side-by-side and see what the differences are. It might not be fair because all the hardware is new and I know pf better than opn --but if opn feels more natural/intuitive then that will say A LOT.

You must be nuts installing a FW in a home hypervisor. Nothing good has come of that for me. (my mind thinks of the person who balances the checkbook standing over me breathing down my neck that the whole house internet is down and she wants to "watch her shows"...)

Ubiquiti AP's:
I have four SSID's/vlans: trusted, guest, management and "thingswedonottrust". This is a little more complicated than just new HW and pfsense (or opnsense) install. I want to do a captive portal. I run a virtual unifi controller (esxi, I'm not buying the cloud key). So do I run 4x vlans to the AP's or do I run one?? two?? and then break things out at the APs/controller? Do I do a pfsense captive portal or do I use the Unifi controller?

VLANS & associated L3 /24 networks: I've always done VLAN101= 10.x1.01/24, VLAN202= 10.x2.02/24, VLAN207= 10.x2.07/24, etc. I've always done one vlan = one /24. I guess it doesn't really matter but I'd like to REALLY clamp down on the number of VLANs and associated L3 networks. I have the typical networks: management, storage, I have 4x wifi ssid/vlans, IOT/things we do not trust, esxi, one for work, one for web/dmz, etc. Should I scrap all the VLANs and use rules to accomplish the same thing?
 

whiskytangofoxconn

New Member
Aug 17, 2021
27
5
3
Any home network segmentation plans? Intervlan routing at 10g or even greater than 1g a need/want? Keeping the potential 10g traffic on the same lan/vlan makes things easy without l3, transit, pfsense dhcp issues to worry about. Will both of those switches be needed/used? A couple inexpensive 10g DAC cables between the 720q bonded to one or both of those switches would be a good start.
I don't need 10G. Actually I'd be fine with 1.25gbe and thrilled with 2.5gbe. There are a few things I do that saturate (exhaust) the (LCAP) gigabit connections I have but other than that, I don't need 10G, don't need 10g wire speed, only have 300/300 external/WAN (and only need really 15-20 outbound reliable)... To be honest, I don't even care enough to do a speed-test or file transfer to see if I actually have 10g speeds. Also, I'm running most of this on the 5e I ran 6-ish years ago.

My old (still current) network was a cisco sg300 in L2 mode and pfsense with dual gig lacp router on stick. It was fine, until it wasn't. At that point I put the switch into L3 mode and offloaded some traffic. That made things much better. It's been basically untouched for 6-ish years --well 4 years with switch doing some of the L3 work. My only concern is that when I studied ccna stuff T1 & T3's were a thing and we were hand-lapping fiber terminations. I haven't used anything ccna related in 10+ years. A long time ago I used to have low voltage licenses so when I did my house (when I wired my house) I ran all smurf tubes. I'll run new connectivity if needed.

I'm mainly worried about keeping some of the networks isolated. I want to keep work/personal isolated as much as possible and I'd like to break out the outbound facing machines/hardware (mostly esxi vms) as much as I can without running dedicated hardware. Some of what I do is with files that are compressed at a rate of about 20:1. If I can figure out a way to work it in that they are transferred as compressed files then the 10gbe will mostly be for naught.

The way I think it'll fall into place is going to be 2x dac cables (redundant cables :) to the sx550x, then "stack" (with dac) the sx550x and the sg550x. I've never actually "stacked" Cisco switches before. I've never used this cisco software. Back when I used the Cisco network management software it was a standalone Windows application and we were doing Catalyst campus 10/100 switches.

I assume I can stack the sx550x and sg550x switches. (pretty sure they play nice together) I have a few sg300 series switches too. I never really thought about it until now: I don't have enough room (and it would be a waste of 10G ports) on the sx550x switch to bring everything down in a classic three-tier campus style network so will the SG300 switches be recognized and talk to the sg550x or cisco network manager if I only plug rj45 gigabit to gigabit?
 

jdnz

New Member
Apr 29, 2021
11
2
3
OK when I started to type this it was "Literally about 5 minutes ago"... But now I've been distracted. I have OPNsense installed and I'm playing with it. I'm going to install pfsense and opnsense side-by-side and see what the differences are. It might not be fair because all the hardware is new and I know pf better than opn --but if opn feels more natural/intuitive then that will say A LOT.
best reason to run OpnSense is Sensei - if you need to 'manage' family users internet access it's by far the most friendly tool I've found
 

zer0sum

Active Member
Mar 8, 2013
527
228
43
You must be nuts installing a FW in a home hypervisor. Nothing good has come of that for me. (my mind thinks of the person who balances the checkbook standing over me breathing down my neck that the whole house internet is down and she wants to "watch her shows"...)
Here's the trick...I have 5 x public IP addresses :p

ESXi almost never needs a reboot unless it's for a critical patch, so that is always up and hosting 2-3 firewalls.
So if I want to switch firewalls I just spin one up and get it working and tested just how I need it and then change the gateway address on my dhcp server.
I run my dns and dhcp through DietPi as that is always up and never changes
After ~24 hours all of my devices have changed to use the "new" firewall and I can decommission the "old" one.

I use 10G SRIOV interfaces as well so I have 8 x hardware passthrough NICs on both the "lan" and "wan" side to use with my virtual hosts.
This takes away the concern over having the hypervisor facing the internet through a vmxnet3 NIC
 
Last edited:
  • Like
Reactions: Vesalius

Vesalius

Active Member
Nov 25, 2019
155
123
43
Been running OPNsense and VyOS as VMs on my Proxmox as my primary firewalls for the last year or more. No issues with increased downtime. As @zer0sum mentioned, I don’t take Proxmox down much unless a new kernel is out, and then I do it one node at a time while others sleep. Helps to be an early bird.
 

Vesalius

Active Member
Nov 25, 2019
155
123
43
Is this a known vector of attack?
I’ll let @zer0sum answer for himself/herself, but IMO no … nothing in reality that I have found. I switched back to VirtIO for wan on Proxmox because it was easier to set up, migrate VM’s from one node to another and SR-IOV/passthrough did not offer a speed advantage on even my 1g symmetrical wan.
 
Last edited: