Pfsense under ESXI slow WAN speed

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

maxermaxer

Active Member
Oct 28, 2016
289
48
28
48
Problem/issue I have: the WAN speed under Pfsense on EXSI is too low. My WAN speed should be nearly 1000Mbps. Without the Pfsense router I am getting somewhere between 960-980Mbps.

speed test.PNG

I have been using Pfsense as my home network router and firewall for a long time. I haven't really looked deeper into this problem that I have for a long time. I have done many online search about this topic and tried different ways including "Disable hardware TCP segmentation offload", "Disable hardware large receive offload" in Advanced --> Network Interfaces. I also tried E1000 and VMXNET3. They don't help. I don't have suricata installed. Now I don't have a clue what the real problem is ...

My current hardware/software is:
Supermicro A1SAi-2750F - Intel(R) Atom(TM) CPU C2750 @ 2.40GHz
32GB RAM
I use 2 Nic ports of the 4, one as WAN and the other as LAN
Pfsense 2.5.2
EXSI 6.5

Anyone can shed some lights on this? Is it the common problem of running Pfsense on VM and cannot be easily fixed?
 
Last edited:

gea

Well-Known Member
Dec 31, 2010
3,141
1,182
113
DE
Do you use vmxnet3 vnics?
They are much faster than e1000.
 

mrossco

New Member
Sep 20, 2021
10
5
3
Specifically, what motherboard are you running, and are you using the onboard NICs or did you plug in a PCIE card?
 

mrossco

New Member
Sep 20, 2021
10
5
3
Any reason you disabled offloading? I don’t have direct experience with this hardware, but the first thing you want to do is make sure you have the proper driver installed. As old as this platform is, I suspect no special configuration should be necessary.

when you disable offloading, this makes these operations cpu bound, which could explain slow speeds. First thing you should do is find guides for configuring an Intel i354 nic to work with ESXi.

Next, you will want to look at your PFSense configuration. Excessive rules or scanning engines could also tax the cpu and slow your speeds down.
 

RTM

Well-Known Member
Jan 26, 2014
956
359
63
Usually disabling offloads is required when doing virtualization, as far as I can remember (though I can't really find a source) it is due to pf needing checksums to be computed, and virtual NICs don't compute checksums (because it is not necessary on a virtual switch, where you are really just copying data in memory). It can lead to very odd behavior when you have checksum offloading enabled, as I remember smaller packets will pass through whereas larger ones will not.

I suggest three things, in the following order (after 1, things get a little far fetched):
  1. Use iperf3 to test your actual bandwidth, without using what I assume is an internet based service someone else provides
  2. Try with a more recent version of ESXi or a different hypervisor if the newer ESXi versions will not support your board
  3. Try without virtualization
 

BoredSysadmin

Not affiliated with Maxell
Mar 2, 2019
1,050
437
83

nickf1227

Active Member
Sep 23, 2015
197
128
43
33
A few things.
Are the port groups in promiscuous mode? I've had issues with pfsense if they are not.
1632523711209.png

Did you install Open-VM-Tools?
1632523648141.png

You should then be able to re-enable offloading.
1632523952383.png
 

zer0sum

Well-Known Member
Mar 8, 2013
849
473
63
Have you tried different speed test websites?

This is an OPNsense FW on ESXi 6.7 with 2 cores of i5-8500, 8G ram, and just VMXNET3 adapters on a tiny M920Q

1632605549644.png

Offload disabled
1632605669838.png
 
Last edited:

maxermaxer

Active Member
Oct 28, 2016
289
48
28
48
Hi guys! After hours of testing I want to come back on this. Now I can finally reach the following close-to-ideal speed through pfSense running on VM.

What did I do? I have tried many different ways of improving the speed of the pfSense in a VM on the Supermicro A1SAi-2750F [Intel(R) Atom(TM) CPU C2750 @ 2.40GHz] system. No matter what I tried it still stayed in 550-650MBps. After doing some more research online I finally concluded that it is due to the ATOM C2750 CPU not supporting the VT-d, hence the performance on VM is dragging the WAN speed down. I have tried Proxmox and EXSI. The results are the same.

Then I changed the system. I used my ASROCK X99E-ITX/ac motherboard sitting at the corner and installed on it a E5-2630L-V3 CPU purchased from local 2nd hand market (@approximately US$23). I installed Proxmox and setup a VM for pfSense. Now I can get 950Mbps DL speed and 872Mbps UL speed. Very nice!

So the conclusion is if you want to use pfSense on VM make sure you use CPU that support VT-D. Otherwise the speed will suffer!

Thanks for all the help from you guys!
@BoredSysadmin , special thanks to you. What you said had led me to find out the importance of VT-D for pfSense-on-VM!

SpeedMeter.PNG

HGC.JPG

Speed_from_FastDotCom2.Jpg
 
Last edited:
  • Like
Reactions: cesmith9999

zer0sum

Well-Known Member
Mar 8, 2013
849
473
63
It would be interesting to see your results if you installed OPNsense on the same hardware :)
 

BoredSysadmin

Not affiliated with Maxell
Mar 2, 2019
1,050
437
83
Not sure if I to take any credit here. VT-D might have helped you out to offload network processing due to slower CPU.
Since you replaced to a much faster cpu is the likely reason for performance improvement. Now your cpu support VT-D you could consider start using a 10gig networking and passthrough the network card to pfsense VM. or just leave everything as is if you're already happy with it.
 

zer0sum

Well-Known Member
Mar 8, 2013
849
473
63
Thank you! I will find a time to setup a VM for OPNsense to test it. What do you expect to see? Faster speed? :)
Might be identical, but I've moved to OPNsense a long time ago, and I'm not looking back at PFsense at all
It's just soo much better and there's none of the crazy bullshit going on with the devs, unlike with the PFsense jackwangs!
 

maxermaxer

Active Member
Oct 28, 2016
289
48
28
48
Not sure if I to take any credit here. VT-D might have helped you out to offload network processing due to slower CPU.
Since you replaced to a much faster cpu is the likely reason for performance improvement. Now your cpu support VT-D you could consider start using a 10gig networking and passthrough the network card to pfsense VM. or just leave everything as is if you're already happy with it.
Good idea! In order not to waste the CPU I have already installed Nextcloud in another VM. It would be nice to use 10gig NIC on it. :)