Sanity check on ESXI host networking

[kerrigansuperfan]

Hey all. Newbie here, decided to start playing around in the homelab world so here I am. I picked up 2x R620s, a Brocade 6610 L3 switch, and a VMUG membership. I want to play around with vCenter, but when I went to install it, was notified that NTP needed to be set up. No problem right? Just ensure there's connectivity to the internet, right??

I want to be secure and play around with OPNsense / Grafana so I decided on trying to route all VM traffic (to include ESXI management traffic) through a OPNsense VM running in ESXI. Not sure if this is a good idea or possible but here we are. Anyway that idea led me to this network diagram. I mostly followed this guide (thank you for getting me this far Kapone). I feel like I'm almost there, OPNsense can ping 8.8.8.8 from both WAN and LAN interfaces, and it can see my ESXI host IP address 10.0.30.3. When I SSH into ESXI at 10.0.30.3 I can ping both OPNsense router interfaces (WAN and LAN), but not 8.8.8.8. In fact when viewing the live log in OPNsense, when I ping 8.8.8.8 (or 192.168.1.1) from my ESXI host IP, I see the packet going out passing through the firewall but nothing comes back. I can only assume 8.8.8.8 receives the ping but for some reason it's not getting routed correctly back to ESXI host all the way back through the network. There's gotta be a simple solution here. Here's my routing table in the brocade switch:

brocade#show ip route
Total number of IP routes: 6
Type Codes - B:BGP D:Connected O:OSPF R:RIP S:Static; Cost - Dist/Metric
BGP Codes - i:iBGP e:eBGP
OSPF Codes - i:Inter Area 1:External Type 1 2:External Type 2
Destination Gateway Port Cost Type Uptime
1 0.0.0.0/0 10.10.10.2 ve 2 1/1 S 1h7m
2 10.0.20.0/24 DIRECT ve 2 0/0 D 1h7m
3 10.0.30.0/24 DIRECT ve 3 0/0 D 1h11m
4 10.0.40.0/24 DIRECT ve 4 0/0 D 1h7m
5 172.16.0.0/24 DIRECT ve 1 0/0 D 1h7m
6 192.168.1.0/24 10.10.10.2 ve 2 1/1 S 1h7m

Here are my gateways on OPNsense:

Name Priority Gateway Monitor IP
WAN_DHCP (active) 254 (upstream) 192.168.1.1
LANINT 255 (upstream) 10.0.20.2
LAN_GW 255 (upstream) 10.0.20.1
WAN_DHCP6 (active) 254

Here's my routing table on OPNsense:

Network Gateway Description
10.0.30.0/24 LAN_GW - 10.0.20.1 MGMT to WAN
0.0.0.0/0 WAN_DHCP - 192.168.1.1

Here is my vSwitch topology for my LAN / Transit network

The only other thing I can think of is changing my TCP/IP stack on ESXI...but then again I feel like I'd break everything, because as of now ESXI can see both OPNsense interfaces...

Please for the love of god help me. I've been working on this for weeks. I've scraped this sub, servethehome.com forums, brocade forums, vmware help pages and forums, youtube videos...literally everything. I know my networking skills are lacking, but I feel like I'm *almost* there...thanks all.

 

[joeribl]

What NAT rules do you have in your firewall?

 

[kerrigansuperfan]

What NAT rules do you have in your firewall?

I was just looking at that - as of now they're set to automatic. I'm guessing I need to go for hybrid rules and add each VLAN IP to it?

 

[joeribl]

I was just looking at that - as of now they're set to automatic. I'm guessing I need to go for hybrid rules and add each VLAN IP to it?

Most likely. I don't have any particular experience with OPNsense, maybe you can post a screenshot of your outbound NAT rules?

 

[u4096]

Your setup seems to have some unnecessary complexity. The diagram is also hard to read. I would rethink the strategy a bit and simplify.

1. Is your cable modem in bridge mode? If not, you're dual/triple NATng from ESXi -> OPNSense -> Netgear -> Cable modem.
2. Is your Netgear router used for Wireless access as well?
3. Why is your desktop dual homed for VLAN3 and the Netgear subnet?
4. Overall, do you have a segmentation strategy in mind e.g. VLAN 3 = ESXi management, but what is the purpose of VLANs 2, 4 and 5? They all show a VLAN ID of 0 in ESXi screenshot which is incorrect.
5. Virtual firewall...there is an argument to be made for hosting it inside of ESXi. But I would steer against it since if ESXi went down, your whole network would go down with it. Except for the desktop connected to the Netgear router. Which seems to defeat the purpose of having everything go through OPNSense.
6. If the goal is to make OPNSense the gateway, why is the Netgear router connected directly to the cable modem?

 

[kerrigansuperfan]

Most likely. I don't have any particular experience with OPNsense, maybe you can post a screenshot of your outbound NAT rules?

Went on a work trip so wasn't able to respond until now - it was 100% this...I feel so dumb, I literally spent weeks trying to figure this out. All working as intended now - thank you!!!

Yes, there is some complexity here. Couple reasons for this - I don't want to mess with my existing home network. These servers / switch are loud, I have a small space with an open floor plan, and the wife gets a vote too. So I can only run it when I'm playing around with different technologies, just wanted to learn stuff from linux, to networking, to a bit of software development, etc etc. For this reason, I didn't want the rest of my home network to be dependent on my lab running 24/7. At the same time, I wanted OPNSense to be the gateway for everything within the lab environment.

Desktop was just dual homed to make inital setup easier, I'll probably remove that soon. Really appreciate the feedback, hope my layout makes more sense, but I'm sure it's still a little crazy. With the above in mind, would you make any different recommendations? Or is there a better way to display my layout that you'd recommend? I just used powerpoint, but I hear there are some good programs out there to better display network architecture

 

[u4096]

No problem. One of the bigger questions you might want to figure out first is, do you want the switch to do the L3 routing/filtering or OPNSense?

While there are many variations of the 2 options below, these are what come to mind that might help simplify your topology:

Option 1: AP (trunk port) -> Brocade (VLAN 1/ VLAN2 / VLAN 3 with ACLs controlling traffic between them) -> simple port to OPNSense as the default GW -> ISP Router

Option 2: AP (trunk port) -> Brocade ports lumped together with tagged/untagged VLAN traffic -> out of a single trunk port on Brocade -> OPNSense Virtual Interfaces i.e. VLAN 1/ VLAN 2 / VLAN 3 etc. with fw rules between them -> ISP

Option 2 is also known as router on a stick and requires that VLANs exist on both the switch and OPNSense. OPNSense would be responsible for the VLAN termination i.e. anytime you plug a regular device in an untagged port or have a phone connect to your AP, the request will ultimately land in OPNSense for DHCP requests, routing, etc. You need both the switch and the FW running to get traffic moving and out to the internet.

Option 1 is a bit simpler overall and easier to troubleshoot as the FW functions and the routing functions would be separate reducing the overhead on the FW. However, there is the initial complexity of configuring the ve (virtual ethernet) interfaces on the switch and learning the ACL syntax.