Thanks for the info! Yes windows bridge is a joke. I was not able to figure out how to set up a bridge with PfSense and I read PfSense performance is also dismal so I am giving up on PfSense.
I believe Marsh reported speeds close to 10Gb with vyos so I am now trying to make VYOS + HyperV work. I was able to set up VYOS as the router mode now if I can figure out how to configure VYOS with 5 port as a switch for the LAN and one port for the WAN. If anyone has already figured that, it would be great if I can take a peek. I am using vyos 1.1.8 as I couldn't figure out how to configure vyos 1.3.
Running ESXi 7.0 U1 between 2 VM with a VyOS Router between them and x540-AT2 10GB nics, my servers are also the same as this 1U, running sys-6028u-tr4+, might pick one of theses U1 for FW/Routing purpose also running ESXi.
[root@iperf02 ~]# iperf3 -c x.x.x.x
Connecting to host 10.0.1.185, port 5201
[ 4] local x.x.x.x port 52064 connected to x.x.x.x port 5201
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 4] 0.00-1.00 sec 1.01 GBytes 8.70 Gbits/sec 147 789 KBytes
[ 4] 1.00-2.00 sec 1.10 GBytes 9.42 Gbits/sec 0 1.50 MBytes
[ 4] 2.00-3.00 sec 1.10 GBytes 9.42 Gbits/sec 0 1.97 MBytes
[ 4] 3.00-4.00 sec 1.09 GBytes 9.41 Gbits/sec 0 2.36 MBytes
[ 4] 4.00-5.00 sec 1.09 GBytes 9.41 Gbits/sec 474 1.57 MBytes
[ 4] 5.00-6.00 sec 1.05 GBytes 9.04 Gbits/sec 777 1.23 MBytes
[ 4] 6.00-7.00 sec 1.10 GBytes 9.42 Gbits/sec 204 953 KBytes
[ 4] 7.00-8.00 sec 1.07 GBytes 9.22 Gbits/sec 0 1.57 MBytes
[ 4] 8.00-9.00 sec 1.09 GBytes 9.39 Gbits/sec 0 2.02 MBytes
[ 4] 9.00-10.00 sec 1.09 GBytes 9.35 Gbits/sec 1356 1.44 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-10.00 sec 10.8 GBytes 9.28 Gbits/sec 2958 sender
[ 4] 0.00-10.00 sec 10.8 GBytes 9.27 Gbits/sec receiver
iperf Done.