Will this work or am I being dumb?

[sadpanda]

After lurking here and elsewhere for a while I've decided to pull the trigger and do a home network overhaul with a sprinkling of 10g.

Main equipment is Aruba S2500, SMC 8708L2, Netgear GS108Ts and an underused pfSense box from old setup, Mellanox cards


Some things I have been considering:

• openVPN is slow
• VPNs seem to be fastest on native app (which needs updated periodically) followed by integration with routers/openWRT
• pfSense wireguard has gone from botched to experimental but no direct support by any vpn host like openWRT
• piHole has a slick/easy interface but that goes out the window with a bunch of chatty IoT devices clogging the logs etc
• firewall on stick is recommended for LAN wirespeed routing BUT I'm not sure I've seen anyone feed a pfSense box 2x10g bond from the switch
• OOB management

So what I'm thinking is reconfigure existing bare metal pfSense to Proxmox with 10g card, LXC/Dockers for pfSense/native vpn app/Omada app/ and piHole(s) for DNS and DHCP (separated by vlan). This way, if I change VPN provider or VPN provider changes protocol/servers all I have to do is change app. Repupose small 8 port switch for OOB. (orange is 10G)

 

[zunder1990]

I would make your pfsense its own box, it will be way easier to manage and be more stable if it was stand alone.

 

[ReturnedSword]

I would make your pfsense its own box, it will be way easier to manage and be more stable if it was stand alone.

Concur. Not on the more stable part, but from experience, much easier to manage/upgrade without potentially breaking another part of your lab stack.

 

[zunder1990]

Concur. Not on the more stable part, but from experience, much easier to manage/upgrade without potentially breaking another part of your lab stack.

more the the more stable part come from in my house I will be rebooting hosts all the time with no warning. If I take offline the internet connection i hear about it from the family within about 30 sec.

 

[PigLover]

Curious why running multiple instances of PiHole all on the same proxmox server? If its redundancy/availability you are after that really won't be much different than just running a single instance. Proxmox host down - all instances go with it. If it was a Proxmox cluster it would make sense.

 

[gregsachs]

One other thought-look at softether vpn server, i find very good performance and it can emulate openvpn servers

 

[ReturnedSword]

more the the more stable part come from in my house I will be rebooting hosts all the time with no warning. If I take offline the internet connection i hear about it from the family within about 30 sec.

This is so true. When I restart services or apply patches family members let me know they can't access the internet within 30 seconds of the service going down lol. Same when I'm fiddling with new pfBlockerNG rules that break some website.

 

[Mam89]

OP, I actually did exactly this when I first started my homelab as I only had a single 4u server. It worked just fine after a bit of set up and learning Proxmox a little better. Driver support is also much better than when I first attempted the task.

That being said, everyone else is 100% correct. Having the nested box was a real PITA when it came to maintenance or power outages (even with a ups). So suffice to say the first extra purchase I made was to get a dedicated 1u micro server with a low power processor. That was loaded bare metal with pfsense and the configuration loaded in no problem. Now when I want to update or change my stack, I can continue watching Netflix while it loads up.

That 1U server is actually the last man standing from my original setup. The 4u was converted to a NAS, I've added 2 compute nodes, and upgraded everything to 10G using the same Aruba switch as you. Everything but the 1U box is currently powered down now, but it doesn't effect the house

 

[oneplane]

Switch from pfSense to OpnSense for WireGuard support, and like others wrote, make it a separate box. Things like Intel QAT are also native and free and always enabled in OpnSense.

 

[Sean Ho]

I run OPNSense with wireguard-kmod on an old 4th gen TMM desktop, no issues. But you also need not feel constrained to run your VPN endpoint on your router; you could put it in a container on PVE if you like.

 

[oneplane]

That is definitely also an option: there is no reason you couldn't run a VPN server separately from your firewall/gateway device.

 

[sadpanda]

Sorry for delay.

I would make your pfsense its own box, it will be way easier to manage and be more stable if it was stand alone.

Concur. Not on the more stable part, but from experience, much easier to manage/upgrade without potentially breaking another part of your lab stack.

I transitioned from store bought routers with wrt to a dedicated pfsense box last year. After the initial setup, nothing ever changed. The only time I borked anything was related to vpn setup. The goal is to get things 'set' and and leave it alone instead of tinkering. My pfsense box sat at 2-5% utilization pretty much no matter what so tying on another box strictly for DNS/vpn gateway seems wasteful.

Curious why running multiple instances of PiHole all on the same proxmox server? If its redundancy/availability you are after that really won't be much different than just running a single instance. Proxmox host down - all instances go with it. If it was a Proxmox cluster it would make sense.

This is so true. When I restart services or apply patches family members let me know they can't access the internet within 30 seconds of the service going down lol. Same when I'm fiddling with new pfBlockerNG rules that break some website.

Again going back to answer 1, I can have different rules on different instances for different network segments. If I decide to tinker, I can do so on one segment and if something goes off the rails just revert to a backup instance. Running vms from a dedicated hypervisor seemed like it would be easier than 'side loading' on a bare metal pf install

OP, I actually did exactly this when I first started my homelab as I only had a single 4u server. It worked just fine after a bit of set up and learning Proxmox a little better. Driver support is also much better than when I first attempted the task.

That being said, everyone else is 100% correct. Having the nested box was a real PITA when it came to maintenance or power outages (even with a ups). So suffice to say the first extra purchase I made was to get a dedicated 1u micro server with a low power processor. That was loaded bare metal with pfsense and the configuration loaded in no problem. Now when I want to update or change my stack, I can continue watching Netflix while it loads up.

That 1U server is actually the last man standing from my original setup. The 4u was converted to a NAS, I've added 2 compute nodes, and upgraded everything to 10G using the same Aruba switch as you. Everything but the 1U box is currently powered down now, but it doesn't effect the house

I fought that urge early hence the dedicated boxes for router/storage/cameras approach. Again, the router box will only be doing 'router things' and no other services.

Switch from pfSense to OpnSense for WireGuard support, and like others wrote, make it a separate box. Things like Intel QAT are also native and free and always enabled in OpnSense.

I run OPNSense with wireguard-kmod on an old 4th gen TMM desktop, no issues. But you also need not feel constrained to run your VPN endpoint on your router; you could put it in a container on PVE if you like.

That is definitely also an option: there is no reason you couldn't run a VPN server separately from your firewall/gateway device.

This is the main reason for visualizing router functions.... I had a bare metal install with hardware crypto capable card/processor and doing openvpn on the router was always a pita. All it takes is VPN to change a server and my config is borked, then cause I'm only smart enough to be dangerous and don't do this stuff for a profession/daily I gotta shake the rust off, get on forums and figure out whats up etc. By contrast, VPN's update their own apps frequently and there is nothing to configure of fiddle with.[/QUOTE]

 

[sadpanda]

The main issue I'm having now is I have no clue what the hell is going on with the networking on proxmox ubuntu CT

Proxmox VE Network not configurable

Hi, After installation and reboot of the Proxmox VE (pve-manager/6.3-2/22f57405 (running kernel: 5.4.78-2-pve), same on 6.4 and 7) i try to add a bond device. i create the bonds I install ifupdown2. Empty bonds and bridges will be created. If i add a network eno3+eno4 to the bond or bridge i...

forum.proxmox.com

I think maybe just set ostype to unmanaged ?

How to disable auto-updates of /etc/hostsname, /etc/resolv.conf (& maybe others)?

PVE newbie here. How do I disable the following "auto-change-files upon startup" behavior? I have very-specific settings in /etc/hostname and /etc/resolv.conf (read: I know what I'm doing) in my Ubuntu 20.04 LXC container. PVE appears to be auto-changing values in these files upon every...

forum.proxmox.com

Modification of a file can be prevented by adding a .pve-ignore. file for it. For instance, if the file /etc/.pve-ignore.hosts exists then the /etc/hosts file will not be touched. This can be a simple empty file created via:

# touch /etc/.pve-ignore.hosts

Most modifications are OS dependent, so they differ between different distributions and versions. You can completely disable modifications by manually setting the ostype to unmanaged.

 

[sadpanda]

And I responded to all of you but its pending mod review triggered by editing the post....

 

[Sean Ho]

I'm not quite clear what the issue was with running VPN on your baremetal pfsense/opnsense box? Were you perhaps using it with a commercial VPN service (PIA, Mullvad, et al.), and when you needed to change the remote endpoint, you inadvertently messed up your pfsense config? There's nothing inherently harder about configuring VPN on pfsense/opnsense vs on ubuntu. Or are you referring to the frequency of VPN software updates, which has more to do with update channel (stable vs testing) than OS?

 

[sadpanda]

I'm not quite clear what the issue was with running VPN on your baremetal pfsense/opnsense box? Were you perhaps using it with a commercial VPN service (PIA, Mullvad, et al.), and when you needed to change the remote endpoint, you inadvertently messed up your pfsense config? There's nothing inherently harder about configuring VPN on pfsense/opnsense vs on ubuntu. Or are you referring to the frequency of VPN software updates, which has more to do with update channel (stable vs testing) than OS?

no wire guard, server IPs change, certificates expire

 

[Sean Ho]

Is it a private wireguard VPN, then? WG runs fine in pfsense with userland driver, or wireguard-kmod kernel driver using add-on package, or switch to OPNSense and use kernel driver. Your peers should be specified via DNS name rather than IP; if the IP changes, have the remote host use DDNS to update. Certs in wg are just pub/priv key pairs for each peer. Under normal circumstances you shouldn't need to rotate them, but if you do of course you'll need to update the config in all the peers; that's equally true for any WG implementation.

There are of course many ways to run a VPN, and you can choose whatever is most comfortable for you. But if you have a hesitation or concern about running wg on pfsense/opnsense, just know you have resources (the fine community right here) who can help you.

 

[oneplane]

I don't think we're all talking about the same meaning of a VPN here. A virtual private network is just a network connection crossing a wide area network that is not part of it. In server and RMM terms, a VPN is usually just for remote access or just remote management. The commercialised, commodified consumer meaning of 'VPN' is different in that it means "download illegally while trying to hide".