networking help

[marcoi]

so i need to pick the brains of the network gurus out there.

I have an ATT fiber at home with 8 static IP block with 5 usable. I have two ESXI hosts that run home services and dev lab. I setup a third esxi host to be used for external services (minecraft/blog, etc).

I have the ATT modem setup with 2 IP. One IP address goes to a sophos utm 9 VM on first two ESXI hosts and allows access to home prod services like vpn. The second IP goes to a pfsense VM running on third box. I want to make sure the esxi 3 host VMs have no access to the home network.

So quick diagram

ip1 --> sophos UTM 9 -- home services. -- esxi host 1
ip2 --> pfsense -- public services --- esxi host 3

I want to be able to access the public services from my home esxi host 1&2. I dont know the best way to get the communication setup.

IE do I setup sophos to pass IP range to pfsense by adding a third nic to pfsense that has access to home network?

Hopefully this makes sense. So far I can access my hosted services from external internet IE cell phone but not from my home network.

 

[marcoi]

picture to maybe help what i am trying to do.


I want to be able to access the MC server on 10.25 from my desktop on 0.100. I dont know the best way of doing this.

 

[marcoi]

if i setup an ip address on interface opt1 on pfsense for my home lan, say 0.101 I can ping that ip from pfsense. But i cant ping say 0.1 etc. so i can get communication across, its just not allowing for accessing other ip addresses.

i setup a status route on sophos UTM for pfsense ip range to see if it helps.
anyone have ideas? better setup vs solving this issue -assuming it solvable.

 

[marcoi]

No idea? At least let me know if this is a good approach.

 

[Rand__]

So why did you not access the services via the external ip? Or do you need more than the exposed services?

 

[marcoi]

I have the esxi method to get to the hosts. So that is covered.

But when I use the external IP to access it doesn't work correctly from with the lan. If I access using a cell phone internet connection, the services work as expected.

I don't know if or how to get it working via external IP

If I had a second ISP it would not be an issue.

 

[Rand__]

you traced the connection? maybe its taking a shortcut and thus not being properly NAT/De-NATed

 

[marcoi]

I don't recall what it does, tonight I'll revert my changes and teat to see what happens.

What is considered best practices for this kind design?

 

[Rand__]

No idea

But if you have multiple public IPs then usually it would make sense to only use the public IPs to bind your services to and not internally as well. So I'd try to fix up the public IP routing issue rather than buidling a second path.

One issue I always had when diagnosing these kind of issues was that it was the way back that did not work properly, not the way *to* the other box. So make sure that you debug also the other way around (where does the MC server send packets to and can it reach your internal box)...

 

[marcoi]

seems like i cant access any of static ip from the lan.
The ATT modem cant talk back to itself when i try to ping/tracert to the different IP address in my static block.

 

[Terry Wallace]

That pretty standard the at&t handoff is not a switch its a route. So your ip block of 8 public ips' is your data.. At&t won't route it out and back into another port.. From their point of view that would be like asking the mailman to deliver mail from your front door to your back door.

Here's what should happen. You route the block of IP's to a device (normally a router) since you have pfsense you can use that. From pfsense you forward or bind services to different IP's and servers internally. Its alot different than the usual home router one ip setup your probably familiar with.

If you need a diagram or some help setting it up let me know.

 

[marcoi]

@Terry Wallace - That is what I figured. I was trying to avoid the single router option, but i think is the only real option to properly set this up the way i want it. I might setup another pc just to act as router layer vs running a vm. That way when i bring down hosts, i dont need to worry about moving VMs around.

Here is a possible design, based on the one main router with many sub routers, any thoughts?

• The ATT modem doesnt do a real Bridge, but i can disable the FW and pass the block down to a main router.
  
  • Strict NAT? Bridge Mode? What is IP Passthrough? Can I enable on my Arris BGW210 or like router?
• For Main Router - thinking of using pfsense then routing a specific IP to sophos home VM and pfsense WS VM.
  • I am hoping pfsense can handle all internet FW needs and be secure. I assume it can but not a strong user of pfsense so i dont know feature sets, etc.
  • I think i want to keep the two routers (sophos/pfsense) separate so each can be configured for thier specific need.
• ESXI Hosts management will be all on the home network under Sophos UTM
• pfsense Dev Lab for now will get IP from Sophos (currently configured that way). It allows me to run dhcp for dev boxes without using all IP for Sophos which has a 50 ip limit.
  • I may change the source IP and get a static IP from Main Router in case i ever want to expose a dev VM to internet.
• I want me home network to be able to access the Web Service VMs, but not the Web Service VMs to access the home network.

 

[Terry Wallace]

I was thinking something more along these lines. Then your not tied to a specific external ip being tied to a host. But rather directed through to an internal location at the top firewall.

if you want to run internal firewalls after that you can. but most of your access controls should be able to be implemented up top with firewall rules.

Oh and for simplicity of editing:

this was in Flowchart Maker & Online Diagram Software

 

[marcoi]

Here is my first design - still a work in progress.

I will either be using pfsense or Sophos XG as first FW- no special options turned on, just acting as a FW to allow traffic in/out and route to right location. (working on testing speed of both FW softwares)

I want to keep vmotion an option so that is what the iWAN vSwitch.

I will still be using Sophos UTM for home protection, i have too much configuration in it to deal with migrating off to something else right now.

I have two sets of WAPs. First will have no UTM rules and allow full access to Internet. I do this for IOT devices and for wife so he FB isnt blocked by Sophos UTM rules lol.

I'm still not sure if i need to another FW in esxi host 1 to work with VMs there that will need be exposed to Internet on Port .66
Port .65 is already in use by Sophos UTM to allow access to Family MC server, plus VPN ,etc..

Also im still not sure how a VM on the home network will access a service exposed on .66

 

[marcoi]

so i am redoing my network, working through a lot of changes. I removed the att modem by doing a bridge mode with pfsense. Details in this thread:https://forums.servethehome.com/ind...ter-advice-before-moving-to-at-t-fiber.24847/

I currently have bare bones running. PFSense VM running with 4 intel nic card in pass-through mode is the providing connection to the Fiber with the att router just acting as authentication. All functions of the ATT RW are off at this point.

Right now pfsnese has LAN which is providing my home network ip addresses and access to internet. I setup a vlan 10 with dhcp to provide ip address to IOT device i dont want talking or seeing my home network. I finally got that working last night but it was a hassle. I have two dell powerconnect 5524 switches. which were blocking the vlan 10 from getting to my TP-link WIFI APs. I run 4 EAP245(US) (two v1 and two v3). I have two WIFI points (one for home network and other on vlan 10 for IOT).

I want to replace my 2 5524 switches with something newer that easier to work with. Last night i had to run the cmd line to get trunking working. I dont know if i even set it up correctly lol. Basically since my APs are on various ports and some ports lead to other switches I cant control what else is on the ports, i needed to have all vlan traffic passed though the network.

Vlan 1 - default


vlan10 trunk


sample of config


For new switch(s) i still need the 10GB SFP+, but that could be another switch which i use for storage traffic.

 

[marcoi]

At a high level is this model possible?

FW1 - provides main internet routing.
FW2 - connects to FW1 and uses static IP .65 as WAN for traffic in and out.
FW2 - provides DHCP range of 0.1/24

FW1 - Lan provides access to dev vms, using dhcp on 1.1/24
FW1 - pushes static ip traffic of .66 to .69 to various servers for hosted services.

Workstation on FW2 has access to FW1 Lan and static ip addressed servers .66 to .69
FW1 Lan does not access FW2 lan
Static IP servers does not access FW1 or Fw2 Lans.

 

[m_b]

Is there a reason you feel you need two separate firewalls, as opposed to having one firewall just segment your network (you could set up a HA pair if you're worried about single point of failure)? One firewall with multiple interfaces (one per network segment), running SNAT seems like it would simplify everything a little.

 

[marcoi]

I'm using pfsense for fw1, but want to use sophos utm as fw2. I don't feel like pfsense is upto level of filtering out stuff as good as sophos.

 

[m_b]

Why not just use UTM? Or do you think you'll hit the 50 device limit?

 

[marcoi]

I'm using pfsense to act as a replacement for att giga router which doesn't have real bridge mode. There is another thread here about att wpa bypass