My network is a mess, how can I make it better?

[Octopuss]

Our home network is made kind of... stupidly, but I couldn't think of any better way back then, and I still don't. I'm not a networking guy after all.
The major shortcoming is the damn virtualized server that runs pfSense in a VM, so obviously, when it goes down, everything stops working. Also when I need to take the server down to do whatever, there's no internet. I do have a router as a backup, but then there's the problem of weird choice of IP addresses.
There's a managed switch with static 192.168.0.2 address and 255.255.252.0 mask (which if I understand correctly lets me use a few subnets, I forgot the details).
The ESXi server is 192.168.2.1, and most non-computers (like printers, UPS, media player etc.) are in the 2.x range as well.
pfSense is 192.168.0.1.
DHCP range for LAN is 192.168.1.x, and there's also 4.x for wifi.
The more I look at it the less it makes any damn sense despite there being a concept.
One of the problems I repeatedly faced was losing access to all network when the server was down (or taken down) and I somehow couldn't connect to the switch anymore. I guess it has something to do with the default IPs when windows cannot connect to DHCP or something.


Can I get any recommendations what to change and why?

 

[altmind]

move your router to a more stable device. maybe hardware dev.

 

[Octopuss]

Out of the question. That was the whole point of the server, to have everything in one box. I know it's not ideal for a few reasons.

I was more curious about the IP addresses. Maybe there is a better setup for situations where I want to keep the local network working and plug in the backup router when the server is down and I need internet access, that sort of thing.

 

[elvisimprsntr]

@Octopuss

What is the price of your sanity?

Run pfSense on a bare metal appliance. You can pick up a used Protectli Vault off evilBay for $200

I like to keep my life less complicated by putting everything on the same sub-net and VLAN, but then I'm not running an enterprise network..

 

[Tech Junky]

@Octopuss

There's 3 private network classes....
10.x.x.x
172.16-32.x.x
192.168.x.x

Might he easier to negotiate things rather than messing with 255.255.x.0 subnets and keeping the 3rd octet straight in your mind. Splitting the LAN into the 3 groups makes it easier to spot things that might have cross talk between groups. Or keep it simple with the 10.x.x.x and assign say 10.0.0.0/24 to your core, throw wifi to 10.8.0.0/24, and everything else on 10.100.0.0/24. The IPs are all arbitrary and up to you as to what you want to pick. All of this is covered in RFC1918.

Now, the whole issue of maintenance / downtime.... you could duplicate the VM and use high availability where you put .1 as the gateway virtually and .2/.3 on the physical. This might work if you have a NAS laying around and spin up a copy of the VM onto it as a backup when you need to take the server down.

 

[Octopuss]

I think I'll just do two subnets, one for everything wired and one for wifi (maybe two for wifi so I can isolate those Xiaomi lights from the rest of the network). I guess I better make it as simple as possible.

Now do the IPs matter at all? Like what gets the 192.168.1.1? Should it be the switch or pfSense? I feel like it shouldn't at all, and if I lose internet, I'd still have access to the switch with the default Windows IP without having to manually editing the TCP/IP configuration.

 

[elvisimprsntr]

@Octopuss

Install pfSense on a dedicated low power Mini PC appliance. That way you won't loose internet when you take your server down. Seems to be the root of most of your frustration.

Protectli: Trusted Firewall Appliances with Firmware Protection

Secure your network with a trusted Protectli Firewall Appliance! Fully compatible with open-source software. US-based support, warranty and repairs.

protectli.com

Netgate

Netgate is an open-source driven secure networking company that provides appliance and software-based firewall, VPN and routing solutions including pfSense

www.netgate.com

Or if you like to gamble

Pfsense-Pfsense Manufacturers, Suppliers and Exporters on Alibaba.comFirewall & VPN

Pfsense Manufacturers & Pfsense Suppliers Directory - Find a Pfsense Manufacturer and Supplier. Choose Quality Pfsense Manufacturers, Suppliers, Exporters at Alibaba.com.Firewall & VPN

www.alibaba.com

Assuming you have your server connected to a UPS, you can share out the UPS status using NUT package as a UPS master and UPS slave on other clients. I have my UPS connect to my pfSense as a master and multiple TrueNAS servers as UPS slaves.

Packages — Nut package | pfSense Documentation

docs.netgate.com

 

[Octopuss]

No, I really do NOT want to buy another box, can we leave it there?

 

[elvisimprsntr]

No, I really do NOT want to buy another box, can we leave it there?

If pfSense is providing DHCP, DNS, and routing, your entire network will be inaccessible when pfSense is not running, as you have already discovered. You would have to move all those services to another box, so why not just install pfSense on its own dedicated low power appliance and call it a day? My pfSense appliance runs 24/7/365 and I never have to reboot except during infrequent updates.

 

[i386]

192.168.0.2 address and 255.255.252.0 mask

DHCP range for LAN is 192.168.1.x, and there's also 4.x for wifi.

252 dec = 1111 1100 bin
Unless I remember something wrong the 4.x range would be in another network?

 

[Octopuss]

Well yes, the 4.x is on a different VLAN. The wifi stuff is already separated.
I'm just trying to figure out an elegant way to recreate the wired stuff.

I shouldn't have mentioned the pfSense part, people are stuck on that and keep pushing the separate box idea It makes sense, but I'm not doing it, that was the entire point of the server and why I dumped all the money in it - to have it all in one machine. Yes it has obvious downsides, but it does run 24/7 without problems. It's just the edge cases that are annoying me that I'm trying to work around at least a little bit.

 

[i386]

After reading the thread I have questions
What's the default gateway for the 192.168.0.0 /22 network? 192.168.0.1?
Are there other gateways defined for the hosts? What's the ip address?
How are pfsense and the backup router connected to the wan(s)?
What do you mean with "weird choice of ip addresses" for the backup router?

 

[Octopuss]

I have already managed to mess it all up, great. Just like I expected, lol.
I thought I could just change the static DHCP mappings and then all the VMs would just get new IPs when I restart the server.
How naive.
Default gateway was originally 192.168.0.1, which was pfSense. I changed that to 192.168.1.2. The LAN interface is now 192.168.1.2/22.
Switch is now 192.168.1.1.
The backup router is not connected anywhere now, I'll deal with that later.

There is a seedbox VM that used to have 192.168.2.6 IP. I changed the static mapping to 192.168.1.8 in pfSense, and now I can't login to Webmin using the hostname anymore.
I can ping the new IP, but I can only access the webUI using the IP. Furthermore, when I log in, it still shows the machine having the old IP.
WTF happened there?
I'm scared to change the IP of TrueNAS now.

I tried rebooting the entire server and switch with no luck.

 

[Octopuss]

Hm, I tried anyway, and no, hostnames no longer work. But why?

 

[Tech Junky]

Hm, I tried anyway, and no, hostnames no longer work. But why?

Did you have them in a hosts file or did they appear in DNS? If they were in DNS then the clients timeout period might take awhile to purge them and update.

Either way there's a lot of different places things tend to hide when making changes. There might be a filter somewhere causing things not to update. There might be a route statement in a conf file that points to the old info. DHCP might be pointing to something different if you made a typo in the config.

 

[sic0048]

Our home network is made kind of... stupidly, but I couldn't think of any better way back then, and I still don't. I'm not a networking guy after all.
The major shortcoming is the damn virtualized server that runs pfSense in a VM, so obviously, when it goes down, everything stops working. Also when I need to take the server down to do whatever, there's no internet. I do have a router as a backup, but then there's the problem of weird choice of IP addresses.
......
One of the problems I repeatedly faced was losing access to all network when the server was down (or taken down) and I somehow couldn't connect to the switch anymore. I guess it has something to do with the default IPs when windows cannot connect to DHCP or something.


Can I get any recommendations what to change and why?

You have three choices.
1) Live with the system you have. Obviously what you are experiencing is a major downside to putting your firewall/router on a single VM machine.
2) Set up at least one more instance of your VM firewall/router in a "high availability" mode. This means putting the second VM on another machine so that when one goes down, the other can still function. There is obviously a lot more to this type of setup, but that is the basic idea.
3) Move your firewall router to it's own machine (ie "bare metal"). Putting it on it's own machine will ensure your network doesn't go down when you have to reboot your VM machine. You can easily get a machine for less than $200 to fit this need.

That's really about it. They all have their positives and negatives and only you can decide which option is best for your situation. Personally I think the way you currently have your firewall/router set up is the worst possible option because the negatives (regularly taking down your entire network and internet access to fix/update the VM machine) out weight the benefits (snapshots, etc), but people use that type of setup all the time.

 

[Octopuss]

Did you have them in a hosts file or did they appear in DNS? If they were in DNS then the clients timeout period might take awhile to purge them and update.

Either way there's a lot of different places things tend to hide when making changes. There might be a filter somewhere causing things not to update. There might be a route statement in a conf file that points to the old info. DHCP might be pointing to something different if you made a typo in the config.

What do you mean in DNS or hosts file? I just use pfSense. That's it.

Now everything stopped working out of the blue. I don't understand anything anymore. I keep restarting pfSense like an idiot and nothing happens.

 

[Tech Junky]

DNS

Well, when you setup DHCP you have to setup DNS options as well. I haven't used PF but, use just plain vanilla Ubuntu instead for a few reasons. This might be one reason though where the automated macros used within some setups obfuscate the underlying commands to make things work. Not to mention for me PF wouldn't be a happy experience with the HW issues I would end up fighting with some devices.

I split the functions and DHCP is one thing and DNS is pihole but, the DHCP service points the DNS sent to clients as the pihole IP. Now, either can do both functions except for the filtering part but, if you just use some public DNS IP's they can both issue that info to the clients all in one "program".

As to the hosts file.... This is something you might have done on the PC to make it easier to get to the devices but, usually on the local network the DHCP/DNS would be able to figure it out if the client is configured correctly. My lookups don't bring me to the server since it's not unique in terms of name but, I only have a couple of IP's I pinned as loopbacks to hit the main functions @ .50 and DNS @ .2

You could login to the server you're using and run a netstat to see what's being hosted / IP being used

 

[Octopuss]

I declared defeat and restored everything from a backup. I guess there are some setting dug deep in pfSense that I cannot find or whatever.

 

[StandbyPowerGuy]

Although I'm in agreement with what some others have said, that running pfSense on dedicated hardware is the simplest solution. I don't practice what I preach because I like a challenge. Like you, I have my router on a VM on my production server, with the same goal in mind, having one bok running. I do a few things differently, however...

First, I run OpenWRT, not pfSense. I used to run the latter on an old thin client, until BSD dropped x86 support. I switched to OpenWRT, which still supports 32 bit hardware today. I keep an instance on the thin client, configured with a full backup of the VM's configuration, minus the differences in network device names. If my VM craps out, I can just plug the thin client in its place, although these days my thin client is third string. I have a test server, identical to production in hardware and software, with a spare VM of OpenWRT set up. It's a matter of starting it up, loading the most recent backup from the production box, which is saved locally on the test server, and plugging in the WAN cable from my ISP's ONT. You could build a machine similar to my third string thin client, except 64-bit for pfSense, and jack it into your network at crunch time. It's bound to be a good deal less expensive and faster than a dedicated appliance. Check out the HP T610Plus, it's a solid 64-bit machine with a quad-core AMD CPU that's got hardware encryption for VPN and a PCIe (X8 IIRC) slot that can accommodate up to a full height quad-port NIC.

A backup device is great for long term failures or planned maintenance, but to have basic functionality such as access to the server's IPMI/BMC during updates requiring a reboot, I've got core devices configured with static IPs outside the DHCP range. Those devices include both the production and test servers' IPMI/BMC out-of-band management ports, the second ethernet interface of a workstation in my office, a laptop in a drawer with a wired connection, all three of my managed switches, and all three of my wireless access points. I can access all of the core devices, but I'm SOL on internet until I either fire up the VM on the test server or plug in the thin client.

My network is fairly elaborate and has managed switches so I use VLANs, but you could do the same thing on a single wired subnet, for simplicity. I'm curious why you're using multiple subnets. How many devices do you have on your network? Are you also using VLANs?