Will this work or am I being dumb?

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

sadpanda

New Member
Jan 19, 2022
11
0
1
After lurking here and elsewhere for a while I've decided to pull the trigger and do a home network overhaul with a sprinkling of 10g.

Main equipment is Aruba S2500, SMC 8708L2, Netgear GS108Ts and an underused pfSense box from old setup, Mellanox cards


Some things I have been considering:
  • openVPN is slow
  • VPNs seem to be fastest on native app (which needs updated periodically) followed by integration with routers/openWRT
  • pfSense wireguard has gone from botched to experimental but no direct support by any vpn host like openWRT
  • piHole has a slick/easy interface but that goes out the window with a bunch of chatty IoT devices clogging the logs etc
  • firewall on stick is recommended for LAN wirespeed routing BUT I'm not sure I've seen anyone feed a pfSense box 2x10g bond from the switch
  • OOB management

So what I'm thinking is reconfigure existing bare metal pfSense to Proxmox with 10g card, LXC/Dockers for pfSense/native vpn app/Omada app/ and piHole(s) for DNS and DHCP (separated by vlan). This way, if I change VPN provider or VPN provider changes protocol/servers all I have to do is change app. Repupose small 8 port switch for OOB. (orange is 10G)

Untitled.png
 

zunder1990

Active Member
Nov 15, 2012
209
71
28
Concur. Not on the more stable part, but from experience, much easier to manage/upgrade without potentially breaking another part of your lab stack.
more the the more stable part come from in my house I will be rebooting hosts all the time with no warning. If I take offline the internet connection i hear about it from the family within about 30 sec.
 
  • Like
Reactions: ReturnedSword

PigLover

Moderator
Jan 26, 2011
3,185
1,545
113
Curious why running multiple instances of PiHole all on the same proxmox server? If its redundancy/availability you are after that really won't be much different than just running a single instance. Proxmox host down - all instances go with it. If it was a Proxmox cluster it would make sense.
 

gregsachs

Active Member
Aug 14, 2018
562
192
43
One other thought-look at softether vpn server, i find very good performance and it can emulate openvpn servers
 

ReturnedSword

Active Member
Jun 15, 2018
526
235
43
Santa Monica, CA
more the the more stable part come from in my house I will be rebooting hosts all the time with no warning. If I take offline the internet connection i hear about it from the family within about 30 sec.
This is so true. When I restart services or apply patches family members let me know they can't access the internet within 30 seconds of the service going down lol. Same when I'm fiddling with new pfBlockerNG rules that break some website.
 

Mam89

Member
Jan 14, 2016
58
11
8
34
SoCal
OP, I actually did exactly this when I first started my homelab as I only had a single 4u server. It worked just fine after a bit of set up and learning Proxmox a little better. Driver support is also much better than when I first attempted the task.

That being said, everyone else is 100% correct. Having the nested box was a real PITA when it came to maintenance or power outages (even with a ups). So suffice to say the first extra purchase I made was to get a dedicated 1u micro server with a low power processor. That was loaded bare metal with pfsense and the configuration loaded in no problem. Now when I want to update or change my stack, I can continue watching Netflix while it loads up.

That 1U server is actually the last man standing from my original setup. The 4u was converted to a NAS, I've added 2 compute nodes, and upgraded everything to 10G using the same Aruba switch as you. Everything but the 1U box is currently powered down now, but it doesn't effect the house :)
 
  • Like
Reactions: zunder1990

oneplane

Well-Known Member
Jul 23, 2021
844
484
63
Switch from pfSense to OpnSense for WireGuard support, and like others wrote, make it a separate box. Things like Intel QAT are also native and free and always enabled in OpnSense.
 

Sean Ho

seanho.com
Nov 19, 2019
774
356
63
Vancouver, BC
seanho.com
I run OPNSense with wireguard-kmod on an old 4th gen TMM desktop, no issues. But you also need not feel constrained to run your VPN endpoint on your router; you could put it in a container on PVE if you like.
 

oneplane

Well-Known Member
Jul 23, 2021
844
484
63
That is definitely also an option: there is no reason you couldn't run a VPN server separately from your firewall/gateway device.
 

sadpanda

New Member
Jan 19, 2022
11
0
1
Sorry for delay.

I would make your pfsense its own box, it will be way easier to manage and be more stable if it was stand alone.
Concur. Not on the more stable part, but from experience, much easier to manage/upgrade without potentially breaking another part of your lab stack.
I transitioned from store bought routers with wrt to a dedicated pfsense box last year. After the initial setup, nothing ever changed. The only time I borked anything was related to vpn setup. The goal is to get things 'set' and and leave it alone instead of tinkering. My pfsense box sat at 2-5% utilization pretty much no matter what so tying on another box strictly for DNS/vpn gateway seems wasteful.

Curious why running multiple instances of PiHole all on the same proxmox server? If its redundancy/availability you are after that really won't be much different than just running a single instance. Proxmox host down - all instances go with it. If it was a Proxmox cluster it would make sense.
This is so true. When I restart services or apply patches family members let me know they can't access the internet within 30 seconds of the service going down lol. Same when I'm fiddling with new pfBlockerNG rules that break some website.
Again going back to answer 1, I can have different rules on different instances for different network segments. If I decide to tinker, I can do so on one segment and if something goes off the rails just revert to a backup instance. Running vms from a dedicated hypervisor seemed like it would be easier than 'side loading' on a bare metal pf install

OP, I actually did exactly this when I first started my homelab as I only had a single 4u server. It worked just fine after a bit of set up and learning Proxmox a little better. Driver support is also much better than when I first attempted the task.

That being said, everyone else is 100% correct. Having the nested box was a real PITA when it came to maintenance or power outages (even with a ups). So suffice to say the first extra purchase I made was to get a dedicated 1u micro server with a low power processor. That was loaded bare metal with pfsense and the configuration loaded in no problem. Now when I want to update or change my stack, I can continue watching Netflix while it loads up.

That 1U server is actually the last man standing from my original setup. The 4u was converted to a NAS, I've added 2 compute nodes, and upgraded everything to 10G using the same Aruba switch as you. Everything but the 1U box is currently powered down now, but it doesn't effect the house :)
I fought that urge early hence the dedicated boxes for router/storage/cameras approach. Again, the router box will only be doing 'router things' and no other services.


Switch from pfSense to OpnSense for WireGuard support, and like others wrote, make it a separate box. Things like Intel QAT are also native and free and always enabled in OpnSense.
I run OPNSense with wireguard-kmod on an old 4th gen TMM desktop, no issues. But you also need not feel constrained to run your VPN endpoint on your router; you could put it in a container on PVE if you like.
That is definitely also an option: there is no reason you couldn't run a VPN server separately from your firewall/gateway device.
This is the main reason for visualizing router functions.... I had a bare metal install with hardware crypto capable card/processor and doing openvpn on the router was always a pita. All it takes is VPN to change a server and my config is borked, then cause I'm only smart enough to be dangerous and don't do this stuff for a profession/daily I gotta shake the rust off, get on forums and figure out whats up etc. By contrast, VPN's update their own apps frequently and there is nothing to configure of fiddle with.[/QUOTE]
 

sadpanda

New Member
Jan 19, 2022
11
0
1
The main issue I'm having now is I have no clue what the hell is going on with the networking on proxmox ubuntu CT


I think maybe just set ostype to unmanaged ?

Modification of a file can be prevented by adding a .pve-ignore. file for it. For instance, if the file /etc/.pve-ignore.hosts exists then the /etc/hosts file will not be touched. This can be a simple empty file created via:

# touch /etc/.pve-ignore.hosts

Most modifications are OS dependent, so they differ between different distributions and versions. You can completely disable modifications by manually setting the ostype to unmanaged.
 

Sean Ho

seanho.com
Nov 19, 2019
774
356
63
Vancouver, BC
seanho.com
I'm not quite clear what the issue was with running VPN on your baremetal pfsense/opnsense box? Were you perhaps using it with a commercial VPN service (PIA, Mullvad, et al.), and when you needed to change the remote endpoint, you inadvertently messed up your pfsense config? There's nothing inherently harder about configuring VPN on pfsense/opnsense vs on ubuntu. Or are you referring to the frequency of VPN software updates, which has more to do with update channel (stable vs testing) than OS?
 
  • Like
Reactions: oneplane

sadpanda

New Member
Jan 19, 2022
11
0
1
I'm not quite clear what the issue was with running VPN on your baremetal pfsense/opnsense box? Were you perhaps using it with a commercial VPN service (PIA, Mullvad, et al.), and when you needed to change the remote endpoint, you inadvertently messed up your pfsense config? There's nothing inherently harder about configuring VPN on pfsense/opnsense vs on ubuntu. Or are you referring to the frequency of VPN software updates, which has more to do with update channel (stable vs testing) than OS?
no wire guard, server IPs change, certificates expire
 

Sean Ho

seanho.com
Nov 19, 2019
774
356
63
Vancouver, BC
seanho.com
Is it a private wireguard VPN, then? WG runs fine in pfsense with userland driver, or wireguard-kmod kernel driver using add-on package, or switch to OPNSense and use kernel driver. Your peers should be specified via DNS name rather than IP; if the IP changes, have the remote host use DDNS to update. Certs in wg are just pub/priv key pairs for each peer. Under normal circumstances you shouldn't need to rotate them, but if you do of course you'll need to update the config in all the peers; that's equally true for any WG implementation.

There are of course many ways to run a VPN, and you can choose whatever is most comfortable for you. But if you have a hesitation or concern about running wg on pfsense/opnsense, just know you have resources (the fine community right here) who can help you.
 

oneplane

Well-Known Member
Jul 23, 2021
844
484
63
I don't think we're all talking about the same meaning of a VPN here. A virtual private network is just a network connection crossing a wide area network that is not part of it. In server and RMM terms, a VPN is usually just for remote access or just remote management. The commercialised, commodified consumer meaning of 'VPN' is different in that it means "download illegally while trying to hide".
 
  • Like
Reactions: Fritz