pfSense VMXNET3 bad performance

[thatsysadmin]

I'm having a problem where pfSense on ESXi 7u2 can't push more than half a gigabit through using VMXNET3 adapters inside pfSense with 4 vCPUs, but I can't get gigabit speeds. Only half.

I tried disabling Kernel PTI mitigations, disabling network card offloading, raising the queues on the VMXNET3 adapters as said on the Netgate Docs, to moving all the cores into a single vsocket. No luck. Using the emulated Intel E1000 network adapters seems to solve the problem which leads me to believe there's a bug in the VMXNET3 driver for FreeBSD 12, but I want to know what could be causing the issue.

On the same machine, using Hyper-V, I could push gigabit speeds no problem with the same configuration (4vCPUs, Hyper-V synthetic NICs).

Is anyone aware of a bug in the VMXNET3 drivers in pfSense at the moment?
Thanks!

 

[zer0sum]

How are you measuring your speeds?

Using internet based speed tests like Internet Speed Test | Fast.com I can get 1+ Gbps through my OPNsense VM.
And that's with just 2 vCPU's and VMXNET3 adapters without any special tuning

 

[BoredSysadmin]

Read this related thread: https://forums.servethehome.com/index.php?threads/pfsense-under-esxi-slow-wan-speed.34166/
Also, did you installed OpenVM tools on the pfSense VM?

 

[mbze430]

This is interesting. I am about to also virtualize a pfSense to get rid of my ASUS router, and was just asking the same thing over at r/pfSense about VMXNET3 because I plan on using a X540-T2 on the WAN and LAN side. Worse come to worse there is always SR-IOV or DirectPath I/O

VMWare virtualized pfSense deployment question regarding WAN 10gb : PFSENSE (reddit.com)

 

[thatsysadmin]

How are you measuring your speeds?

Using internet based speed tests like Internet Speed Test | Fast.com I can get 1+ Gbps through my OPNsense VM.
And that's with just 2 vCPU's and VMXNET3 adapters without any special tuning

I was using Google's speed test and Speedtest.net.

 

[thatsysadmin]

Read this related thread: https://forums.servethehome.com/index.php?threads/pfsense-under-esxi-slow-wan-speed.34166/
Also, did you installed OpenVM tools on the pfSense VM?

He swapped over to PVE. I did install Open VM Tools. No luck.

 

[zer0sum]

I was using Google's speed test and Speedtest.net.

As I mentioned I have an OPNsense FW virtual machine running with 2 vCPU's, 8gb ram and VMXNET3 adapters.
This is on ESXi 6.7, on an AsrockRack X470D4U, with a Ryzen 5600X, and Mellanox ConnectX-3 cards

Speedtest.net has speeds that stay a bit more consistent


I found Google Fiber speedtest to be slow with uploads every time

 

[thatsysadmin]

It seems like disabling all the hardware offload gets it to full speed. Thanks for your help everyone!

 

[zer0sum]

It seems like disabling all the hardware offload gets it to full speed. Thanks for your help everyone!

I definitely have all of that turned off

 

[thatsysadmin]

I definitely have all of that turned off
View attachment 20075

Are you running OPNsense?