pfSense replacing a Cisco Router - not acting as expected

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

Jonthewise

New Member
Nov 22, 2017
15
1
3
42
Hey all. So I have just replaced my old Cisco 2921 with a little pfSense build on a desktop box with an added 4 port NIC card. All just GB connections, so nothing fancy there. I thought it would just be a case of remove and replace. That didn't seem to work though.

Firstly, my entire network lives on an Aruba S2500 with VLANs segregated off for various things, my home lab runs on VLAN10, IP Cams on VLAN11, my DMZ on VLAN13, and then the WiFi (Aruba access points, no fancy config - though I would like to be able to put different WiFi networks on different VLANs eventually, right now they're just sitting on access ports with no trunking) and desktops on VLAN15, and finally VLAN1000 (VLAN1 was already taken ;)) for the subnet that my router lives on.

I have a pretty straightforward subnet config, 192.168.(VLAN#).X/24 so lab on 192.168.10.X, etc. The router was at 192.168.0.2 (all of the .1 IPs are assigned to the Aruba VLAN interfaces) and the Aruba was set up with a default route of 192.168.0.2 to pass traffic to the router. DHCP was handled with an IP Helper pointing to my domain controller, and each subnet had the appropriate .1 gateway for that subnet configured. This all worked out swimmingly.

When I hooked up the pfSense box though, everything did not "just work." As a workaround, and to get the internet working again before the wife got home from work, I've basically turned on DHCP in pfSense, moved the WiFi access points, her PC's switch port, and my desktop to VLAN1000. First I had to add a LAN gateway for 192.168.0.0/24 pointing to 192.168.0.1 in pfSense to be able to even access the web console (before I moved everything to VLAN1000), and then I added a domain override to the DNS resolver for my local domain and an access list to allow 192.168.0.0/16 to do DNS lookups (though since the only thing connecting to pfSense is VLAN1000, don't know if the access list part was required) so that everything can still work (mainly the shared storage on my TrueNAS server for the windows file shares that my wife would be looking for).

Now I really want to get everything back onto the correct VLANs, but I'm not entirely sure where to start.

Right now, none of my lab systems can access the internet, even though the default route on the switch still points to the same IP (not sure if I mentioned that I just gave pfSense the same IP that the cisco was using) but nothing is routing. So I'd like to start by resolving that...

Should I create all the VLANs in pfSense, and then turn the port that it's connecting to into a trunk port? (I didn't do this previously with the router)

---

And once that is all done, I want to enable failover - maybe I'll start another thread for that after I get this working though - I have a Comcast internet gateway in bridge mode, and they gave me free Verizon LTE as a backup - I had the failover working on the Cisco using route maps, but I could never get failover and NAT working at the same time (so I had two configs that I had to switch between, basically making it a manual failover whenever I lose Comcast service) - I don't really expect incoming traffic from the LTE backup (though I do have Dynamic DNS, so it *could* maybe), but at least when it's on Comcast I like to have NAT into my DMZ to connect to my Horizon gateway and access my desktop remotely without having to change the config to make failover work.
 

Sean Ho

seanho.com
Nov 19, 2019
768
352
63
Vancouver, BC
seanho.com
Two common options: (1) do L3 routing on the S2500, and have only a single transit VLAN going between the switch and pfsense, and (2) trunk all VLANs to pfsense and do routing there, essentially using the S2500 only for L2. (1) is what you were originally trying to do, and is a perfectly valid option.

Did you add routes in pfsense so it knows how to reach devices on the VLANs hanging off the S2500? E.g., 192.168.10.0/24 via 192.168.0.1.
 

Jonthewise

New Member
Nov 22, 2017
15
1
3
42
Two common options: (1) do L3 routing on the S2500, and have only a single transit VLAN going between the switch and pfsense, and (2) trunk all VLANs to pfsense and do routing there, essentially using the S2500 only for L2. (1) is what you were originally trying to do, and is a perfectly valid option.

Did you add routes in pfsense so it knows how to reach devices on the VLANs hanging off the S2500? E.g., 192.168.10.0/24 via 192.168.0.1.
Option1 is what I was doing before. I added a static route for 192.168.0.0/16 to go through 192.168.0.1 on pfSense, but that doesn't seem to be working, so maybe I need to scale back and do /24 static routes for each VLAN?

Edit -

Created individual /24 static routes and disabled the /16 route. My internet is still working on the VLAN1000 devices, but still nothing on any other VLAN.

Edit2 -

Scratch that, somehow my webinar stayed running (probably already established connection?) while the rest of the internet stopped working until I turned the /16 route back on.
 
Last edited:

Jonthewise

New Member
Nov 22, 2017
15
1
3
42
Posting my Aruba config to see if anybody sees something glaringly out of line. As far as pfSense goes, it's basically a fresh installation, and if doing that over again is what it takes, I'm game. All I've done is assign a WAN, LAN, and OPT1 interfaces, gave the WAN DHCP, the LAN 192.168.0.2/24 with a gateway of 192.168.0.1 and route for 192.168.0.0/16 using the LAN GW, and then added the individual 192.168.10.0/24, 192.168.13.0/24, and 192.168.15.0/24 routes, all pointing to the LAN GW. Almost everything that resides on VLAN1000 now used to be VLAN15 when I had the Cisco router in place (basically all except ports 45 and 47, which were always on VLAN1000).

Code:
#
# Configuration file for ArubaOS
# ArubaOS Version 7.4.1.12 72393
version 7.4
enable secret "*"
hostname "ArubaCore"
clock timezone PST -8
controller config 10
ip access-list eth validuserethacl
  permit any
!
netservice svc-dhcp udp 67 68
netservice svc-dns udp 53
netservice svc-ftp tcp 21
netservice svc-h323-tcp tcp 1720
netservice svc-h323-udp udp 1718 1719
netservice svc-http tcp 80
netservice svc-https tcp 443
netservice svc-icmp 1
netservice svc-kerberos udp 88
netservice svc-natt udp 4500
netservice svc-ntp udp 123
netservice svc-sip-tcp tcp 5060
netservice svc-sip-udp udp 5060
netservice svc-sips tcp 5061
netservice svc-smtp tcp 25
netservice svc-ssh tcp 22
netservice svc-telnet tcp 23
netservice svc-tftp udp 69
netservice svc-vocera udp 5002
ip access-list stateless allowall-stateless
  any any any  permit
!
ip access-list session cplogout
  user   alias controller svc-https  dst-nat 8081
!
ip access-list stateless cplogout-stateless
  user   alias controller sys-svc-https  dst-nat 8081
!
ip access-list stateless dhcp-acl-stateless
  any any svc-dhcp  permit
!
ip access-list stateless dns-acl-stateless
  any any svc-dns  permit
!
ip access-list stateless http-acl-stateless
  any any svc-http  permit
!
ip access-list stateless https-acl-stateless
  any any svc-https  permit
!
ip access-list stateless icmp-acl-stateless
  any any svc-icmp  permit
!
ip access-list stateless logon-control-stateless
  any any svc-icmp  permit
  any any svc-dns  permit
  any any svc-dhcp  permit
  any any svc-natt  permit
!
ip access-list session validuser
  network 169.254.0.0 255.255.0.0 any any  deny
  any any any  permit
!
user-role ap-role
!
user-role authenticated
 access-list stateless allowall-stateless
!
user-role denyall
!
user-role denydhcp
!
user-role guest
 access-list stateless http-acl-stateless
 access-list stateless https-acl-stateless
 access-list stateless dhcp-acl-stateless
 access-list stateless icmp-acl-stateless
 access-list stateless dns-acl-stateless
!
user-role guest-logon
!
user-role logon
 access-list stateless logon-control-stateless
!
user-role preauth
!
user-role stateful-dot1x
!
!

crypto ipsec transform-set default-boc-bm-transform esp-aes256 esp-sha-hmac
crypto ipsec transform-set default-rap-transform esp-aes256 esp-sha-hmac


mgmt-user admin root maskingthelonghexstringthatwashere


ntp server 192.168.10.52
ntp server 192.168.10.51

firewall disable-stateful-h323-processing
!
ip domain lookup
!
aaa authentication mac "default"
!
aaa authentication dot1x "default"
!
aaa server-group "default"
 auth-server Internal
 set role condition role value-of
!
aaa profile "default"
!
aaa authentication captive-portal "default"
!
aaa authentication vpn "default"
!
aaa authentication mgmt
!
aaa authentication wired
!
web-server
!
papi-security
!
aaa password-policy mgmt
!
traceoptions
!
probe-profile "default"
   protocol icmp
!
qos-profile "default"
!
policer-profile "default"
!
ip-profile
   default-gateway 192.168.0.2
!
lcd-menu
!
interface-profile ospf-profile "default"
   area 0.0.0.0
!
interface-profile pim-profile "default"
!
interface-profile igmp-profile "default"
!
stack-profile
!
ipv6-profile
!
activate-service-firmware
!
aruba-central
!
rogue-ap-containment
!
interface-profile switching-profile "default"
!
interface-profile switching-profile "TrunkPort"
   switchport-mode trunk
!
interface-profile switching-profile "vlan10"
   access-vlan 10
   native-vlan 10
!
interface-profile switching-profile "vlan1000"
   access-vlan 1000
   native-vlan 1000
!
interface-profile switching-profile "vlan11"
   access-vlan 11
   native-vlan 11
!
interface-profile switching-profile "vlan13"
   access-vlan 13
   native-vlan 13
!
interface-profile switching-profile "vlan15"
   access-vlan 15
   native-vlan 15
!
interface-profile switching-profile "vlan9"
   access-vlan 9
   native-vlan 9
!
interface-profile tunneled-node-profile "default"
!
interface-profile poe-profile "default"
!
interface-profile poe-profile "poe-factory-initial"
   enable
!
interface-profile enet-link-profile "default"
!
interface-profile lacp-profile "FreeNAS"
   group-id 0
   mode active
!
interface-profile lldp-profile "default"
!
interface-profile lldp-profile "lldp-factory-initial"
   lldp transmit
   lldp receive
!
interface-profile mstp-profile "default"
!
interface-profile pvst-port-profile "default"
!
interface-profile dhcp-relay-profile "MainDHCP"
   helper-address 192.168.10.52
!
vlan-profile dhcp-snooping-profile "default"
!
vlan-profile mld-snooping-profile "default"
!
vlan-profile igmp-snooping-profile "default"
!
vlan-profile igmp-snooping-profile "igmp-snooping-factory-initial"
!
spanning-tree
   mode mstp
!
gvrp
!
mstp
!
lacp
!
vlan "1"
   igmp-snooping-profile "igmp-snooping-factory-initial"
!
vlan "9"
   description "VLAN0009"
!
vlan "10"
   description "VLAN0010"
!
vlan "11"
   description "VLAN0011"
!
vlan "13"
   description "VLAN0013"
!
vlan "15"
   description "VLAN0015"
!
vlan "1000"
   description "VLAN1000"
!
interface gigabitethernet "0/0/0"
   description "LivingRoomPort1"
   switching-profile "vlan1000"
!
interface gigabitethernet "0/0/1"
   description "FrontPorchCam1"
   switching-profile "vlan11"
!
interface gigabitethernet "0/0/2"
   description "OfficeWindowLeft1"
   switching-profile "vlan1000"
!
interface gigabitethernet "0/0/3"
   description "OfficeWindowLeft2"
   switching-profile "vlan1000"
!
interface gigabitethernet "0/0/4"
   description "OfficeWindowRight1"
   switching-profile "vlan1000"
!
interface gigabitethernet "0/0/5"
   description "OfficeWindowRight2"
   switching-profile "vlan1000"
!
interface gigabitethernet "0/0/6"
   description "VanessaOffice1"
   switching-profile "vlan1000"
!
interface gigabitethernet "0/0/7"
   description "GarageCam1"
   switching-profile "vlan11"
!
interface gigabitethernet "0/0/8"
   description "FrontCornerCam1"
   switching-profile "vlan11"
!
interface gigabitethernet "0/0/9"
   description "MasterBedroomPI"
   switching-profile "vlan1000"
!
interface gigabitethernet "0/0/10"
   description "OldRasPi"
   switching-profile "vlan1000"
!
interface gigabitethernet "0/0/11"
   description "MasterBedroom"
   switching-profile "vlan1000"
!
interface gigabitethernet "0/0/12"
   description "vSphere Mgmt"
   switching-profile "vlan10"
!
interface gigabitethernet "0/0/13"
   description "TempUPS"
   switching-profile "vlan10"
!
interface gigabitethernet "0/0/14"
   description "Physical Domain Controller"
   switching-profile "vlan10"
!
interface gigabitethernet "0/0/24"
   description "FreeNAS iLo"
   switching-profile "vlan15"
!
interface gigabitethernet "0/0/25"
   description "vSphere iLo"
   switching-profile "vlan15"
!
interface gigabitethernet "0/0/26"
!
interface gigabitethernet "0/0/27"
!
interface gigabitethernet "0/0/28"
!
interface gigabitethernet "0/0/29"
   description "AppleTimeCapsule"
   switching-profile "vlan15"
!
interface gigabitethernet "0/0/30"
   description "PhilipsHueBridge"
   switching-profile "vlan1000"
!
interface gigabitethernet "0/0/31"
   description "HDHomeRun"
   switching-profile "vlan10"
!
interface gigabitethernet "0/0/32"
!
interface gigabitethernet "0/0/33"
!
interface gigabitethernet "0/0/36"
!
interface gigabitethernet "0/0/37"
!
interface gigabitethernet "0/0/38"
!
interface gigabitethernet "0/0/39"
!
interface gigabitethernet "0/0/40"
!
interface gigabitethernet "0/0/41"
!
interface gigabitethernet "0/0/44"
   description "ArubaWiFi-2"
   switching-profile "vlan1000"
!
interface gigabitethernet "0/0/45"
   description "ComcastAdminConnection"
   switching-profile "vlan1000"
!
interface gigabitethernet "0/0/46"
   description "ArubaWiFi-1"
   switching-profile "vlan1000"
!
interface gigabitethernet "0/0/47"
   description "InternetUplink"
   switching-profile "vlan1000"
!
interface gigabitethernet "0/1/0"
   description "TrueNAS"
   switching-profile "vlan10"
!
interface gigabitethernet "0/1/1"
!
interface gigabitethernet "0/1/2"
   description "vSphere Trunk Port"
   switching-profile "TrunkPort"
!
interface gigabitethernet "0/1/3"
   description "vSphere Trunk Port"
   switching-profile "TrunkPort"
!
interface vlan "10"
   ip address 192.168.10.1 255.255.255.0
!
interface vlan "1000"
   ip address 192.168.0.1 255.255.255.0
!
interface vlan "11"
   dhcp-relay-profile "MainDHCP"
   ip address 192.168.11.1 255.255.255.0
!
interface vlan "13"
   dhcp-relay-profile "MainDHCP"
   ip address 192.168.13.1 255.255.255.0
!
interface vlan "15"
   dhcp-relay-profile "MainDHCP"
   ip address 192.168.15.1 255.255.255.0
!
interface vlan "9"
   ip address 192.168.9.1 255.255.255.0
!
interface mgmt
!
interface port-channel "0"
   switching-profile "vlan10"
   enet-link-profile pc_default
!
device-group ap
!
interface-group gigabitethernet "default"
   apply-to ALL
   lldp-profile "lldp-factory-initial"
   poe-profile "poe-factory-initial"
!

snmp-server view ALL oid-tree iso included
snmp-server group ALLPRIV v1 read ALL notify ALL
snmp-server group ALLPRIV v2c read ALL notify ALL
snmp-server group ALLPRIV v3 noauth read ALL notify ALL
snmp-server group AUTHPRIV v3 priv read ALL notify ALL
snmp-server group AUTHNOPRIV v3 auth read ALL notify ALL

snmp-server enable trap

process monitor log
end
 

Jonthewise

New Member
Nov 22, 2017
15
1
3
42
I also tried replacing the default-gateway with a default route:
Before:
Code:
ip-profile
   default-gateway 192.168.0.2
After:
Code:
ip-profile
   route 0.0.0.0 0.0.0.0 192.168.0.2 0
Still doesn't seem to be working. From one of the machines I left on VLAN15:

Code:
Windows IP Configuration


Ethernet adapter Ethernet0 2:

   Connection-specific DNS Suffix  . :
   IPv4 Address. . . . . . . . . . . : 192.168.15.42
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   Default Gateway . . . . . . . . . : 192.168.15.1

C:\Users\jon.HOME>ping 192.168.0.1

Pinging 192.168.0.1 with 32 bytes of data:
Reply from 192.168.0.1: bytes=32 time=1ms TTL=64
Reply from 192.168.0.1: bytes=32 time=1ms TTL=64
Reply from 192.168.0.1: bytes=32 time=1ms TTL=64
Reply from 192.168.0.1: bytes=32 time<1ms TTL=64

Ping statistics for 192.168.0.1:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 0ms, Maximum = 1ms, Average = 0ms

C:\Users\jon.HOME>ping 192.168.0.2

Pinging 192.168.0.2 with 32 bytes of data:
Request timed out.
Request timed out.
Request timed out.
Request timed out.

Ping statistics for 192.168.0.2:
    Packets: Sent = 4, Received = 0, Lost = 4 (100% loss),
And tracert hits the gateway at 192.168.15.1, and then the rest of the hops time out forever. It really feels like an issue on my Aruba, which CAN ping the pfSense box... but I don't understand why it was working fine with the cisco router and now with pfsense it just doesn't want to play at all.

Also, pfSense box can ping 192.168.15.42 just fine... which makes everything even more confusing to me, because it's only aware of the 192.168.0.1 interface on the Aruba, so that tells me the Aruba is doing the VLAN routing okay... at least in one direction.

I'm happy to post any logs or other command output that could help figure this out.
 
Last edited:

LodeRunner

Active Member
Apr 27, 2019
540
227
43
All I've done is assign a WAN, LAN, and OPT1 interfaces, gave the WAN DHCP, the LAN 192.168.0.2/24 with a gateway of 192.168.0.1 and route for 192.168.0.0/16 using the LAN GW, and then added the individual 192.168.10.0/24, 192.168.13.0/24, and 192.168.15.0/24 routes, all pointing to the LAN GW. Almost everything that resides on VLAN1000 now used to be VLAN15 when I had the Cisco router in place (basically all except ports 45 and 47, which were always on VLAN1000).
The part I bolded, you did that on pfSense? If so, there's a problem right there. You've told pfSense to route a network for which it already has a local address. That's not going to end well.

Pick a different subnet for your transit VLAN, like 172.31.254.0/30. pfSense, gets .2, Aruba gets .1. Then the route on pfSense would be 192.168.0.0/16 via 172.31.254.1. The route on Aruba would be 0.0.0.0/0 via 172.1.254.2.
 

Jonthewise

New Member
Nov 22, 2017
15
1
3
42
The part I bolded, you did that on pfSense? If so, there's a problem right there. You've told pfSense to route a network for which it already has a local address. That's not going to end well.

Pick a different subnet for your transit VLAN, like 172.31.254.0/30. pfSense, gets .2, Aruba gets .1. Then the route on pfSense would be 192.168.0.0/16 via 172.31.254.1. The route on Aruba would be 0.0.0.0/0 via 172.1.254.2.
The Cisco router went away, and the pfSense server took it's place. See the diagram I just uploaded right before you posted. I used the Cisco router's LAN IP address for the LAN address in pfSense. The WAN port is DHCP on both. The default route on the Aruba is 192.168.0.2 (the ip of the router before, and pfsense now) and pfSense does have a route of 192.168.0.0/16 via 192.168.0.1(the Aruba's VLAN1000 address) configured.
 

Jonthewise

New Member
Nov 22, 2017
15
1
3
42
The Cisco router went away, and the pfSense server took it's place. See the diagram I just uploaded right before you posted. I used the Cisco router's LAN IP address for the LAN address in pfSense. The WAN port is DHCP on both. The default route on the Aruba is 192.168.0.2 (the ip of the router before, and pfsense now) and pfSense does have a route of 192.168.0.0/16 via 192.168.0.1(the Aruba's VLAN1000 address) configured.
OH!

Is it because of the overlapping route?!? Why didn't the cisco ever care about it? So you're saying if I change the IP subnet of VLAN1000 to the 172.1.254.0 range then everything will cooperate?
 

LodeRunner

Active Member
Apr 27, 2019
540
227
43
OH!

Is it because of the overlapping route?!? Why didn't the cisco ever care about it? So you're saying if I change the IP subnet of VLAN1000 to the 172.1.254.0 range then everything will cooperate?
Probably. Technically, you're not supposed to have overlapping routes. Some routers will still choose the most specific route (a /24 being more specific than a /16). I've always kept my transit VLAN/subnet in a completely separate RFC1918 space, so I've never had occasion to test how pfSense would handle overlapping routes.

I assume 172.1.254.0 was a type, but so you don't discover another fun routing issue, be sure the address falls in 10.0.0.0/8 or 172.16.0.0/12. 172.0.0.0/12 is publicly routable.
 
  • Like
Reactions: dswartz

dswartz

Active Member
Jul 14, 2011
610
79
28
Probably. Technically, you're not supposed to have overlapping routes. Some routers will still choose the most specific route (a /24 being more specific than a /16). I've always kept my transit VLAN/subnet in a completely separate RFC1918 space, so I've never had occasion to test how pfSense would handle overlapping routes.

I assume 172.1.254.0 was a type, but so you don't discover another fun routing issue, be sure the address falls in 10.0.0.0/8 or 172.16.0.0/12. 172.0.0.0/12 is publicly routable.
back in the day, I had to work with cisco IOS. i vaguely remember that you could overlay routes of different widths and it just worked. iirc, the intent was to have a wider route pointing to the bit bucket or a lower-priority interface or some such.
 

LodeRunner

Active Member
Apr 27, 2019
540
227
43
Yes, Cisco will do that, a lot of routers will. However there's no RFC that says a router has to do it. I found this, albeit from 2016, so the exact reference may no longer be correct.

When subnets in different locations overlap there addresses, a router's routing table entries overlap. As a result, hosts in different locations can be assigned the same IP address. Router's clearly cannot route packets correctly in these cases. In short, a design that uses overlapping subnets is considered to be an incorrect design and should not be used.
– Page 566: ICND1 100-101
 

Jonthewise

New Member
Nov 22, 2017
15
1
3
42
Probably. Technically, you're not supposed to have overlapping routes. Some routers will still choose the most specific route (a /24 being more specific than a /16). I've always kept my transit VLAN/subnet in a completely separate RFC1918 space, so I've never had occasion to test how pfSense would handle overlapping routes.

I assume 172.1.254.0 was a type, but so you don't discover another fun routing issue, be sure the address falls in 10.0.0.0/8 or 172.16.0.0/12. 172.0.0.0/12 is publicly routable.
okay, yeah, I was just copying the subnet range you used in the aruba gateway (which appears to be a typo, because the rest of your example used 172.31.254.0.

Anyway, so I went ahead and made my transit VLAN use 10.0.0.0/24; Set the IP on the Aruba to 10.0.0.1 and reloaded pfSense from scratch, set the LAN to 10.0.0.2 with a gateway of 10.0.0.1 and created a route of 192.168.0.0/16 pointed at 10.0.0.1

And to connect to the internet to post this, I had to move my laptop to 10.0.0.0/24 with 10.0.0.2 manually set as the GW and DNS. Nothing in my lan can get out.
 

Jonthewise

New Member
Nov 22, 2017
15
1
3
42
As it turns out, I had to tell pfSense to allow traffic from the 192.168.0.0/16 network as well. The default allow rule says LAN net, but it's not actually allowing anything from the interface. I added a rule for 192.168.0.0/16 and bam, the internet started working!

Thanks for all the assistance troubleshooting!
 

LodeRunner

Active Member
Apr 27, 2019
540
227
43
As it turns out, I had to tell pfSense to allow traffic from the 192.168.0.0/16 network as well. The default allow rule says LAN net, but it's not actually allowing anything from the interface. I added a rule for 192.168.0.0/16 and bam, the internet started working!

Thanks for all the assistance troubleshooting!
Glad you pinned it down! I was thinking that might have been it, but was not in a position to test, was planning on dragging out an unused L3 switch and building a lab for it.
 

Jonthewise

New Member
Nov 22, 2017
15
1
3
42
Glad you pinned it down! I was thinking that might have been it, but was not in a position to test, was planning on dragging out an unused L3 switch and building a lab for it.
Wow, that would have been above and beyond! Thanks again for the help, can't tell you how much I appreciate it :)
 

LodeRunner

Active Member
Apr 27, 2019
540
227
43
I used to run a routed VLAN setup but it cause me more trouble than it was worth (e.g. failed the WAF test) so I flattened my network. The config I ran was stolen replicated from the office I worked at at the time as a learning experience.
 
  • Like
Reactions: dswartz

Jonthewise

New Member
Nov 22, 2017
15
1
3
42
LOL, I built my config using Google, but I know all about the WAF; I can't tell you how many times mine's complained about the internet going down because of this or that - my setup includes DHCP and DNS on a windows server that manages my domain, and was in a VM inside my ESXi cluster. Any time ESX would crash (which happened far more after the upgrade to 7 then it ever did on 6.7) she'd lose DNS resolution, which basically meant no internet. I've since put a domain controller on a desktop box in the top of the rack and moved DHCP over to it so that I can manage my vCenter environment without breaking the internet. Now that the pfSense box is working, I think I've gotten it down to the point that I can shut down everything in my lab except the TrueNAS server without her knowing (and if I have to reboot TrueNAS then she only loses a few network drives - I *used* to even keep our profile folders on the NAS so that I could back them up nightly to back blaze :D now she has to copy anything she wants backed up to one of the network shares, but she preferred that option;))
 
  • Like
Reactions: dswartz

LodeRunner

Active Member
Apr 27, 2019
540
227
43
In my case, it was the way routing and mDNS/Bonjour/Avahi/etc didn't get along. I think it may ahve also affected uPnP which some games we play together use and caused issues joining the same game. Some of that I fixed with a VM with interfaces in both VLANs and an mDNS proxy/reflector running on it. But it would occasionally crash.

And honestly, the isolation didn't buy me any additional security (in my specific use cases) so I flattened it. It was fun and useful experience to build it. Now that I have spare L3 switches I might setup a lab environment again, but for the most part, if it's not trusted, it goes in a different VLAN with a different interface to my firewall and strict rules are enforced there.

I run a pair of R720XDs with Server 2019 Datacenter and Starwind VSAN. Only thing we lose during server maintenance now is the Plex server because it has a GPU pinned to it via DDA, so it can't migrate (also the VHD containing all the media is stuck on that host anyway; boot volumes only are in the VSAN because it's a pair of FusionIO 1.2 TB SSDs).
 

dswartz

Active Member
Jul 14, 2011
610
79
28
Speaking of WAF? I used to run a virtualized router/gateway, but switched to HW because my wife telecommutes, and having her go offline because I was dicking with something? Not good, bruh!