First of thanks for letting me know DCUI (yellow config screen) did not even knew it was there. If I knew about it it would save me countless reinstalls.Yeah you don't need a vSwitch1 (you have no phys nics assigned/avail for it which equals completely isolated storage unless you ONLY want this host to connect to it then yes you could take this route but I am not EVEN muddying the waters w/ that discussion right now. Should be no issue to have vmk0 (mgmt) and vmk1 (vmkernel IP SAN conn) IP's assigned on same vSwitch0. If you lose mgmt, you can always reset very quickly via DCUI (yellow config screen). No need to re-install.
So I found my old PCI slot Intel PRO/1000 PT Dual Port Server Adapter that I use to use with pfSense and its recognized in ESXi:
I know its old and should used something newer like PCIe but I will worry about that once I get it working.
I tried to follow your above explanations on what to do when I have two physical nics but I kinda lost you at the end because I have nothing to reference it to.
Here's what I came up with. Do I need to physically connect chosen port on my Intel it to my router ?