Minisforum MS-01 opnsense VM: WAN & LAN low performances

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
F

Fazio

New Member
Dec 20, 2022
28
21
3
Hello,

I received my MS-01 today and I'm using a opnsense VM in Proxmox.
Below the VM config:

1708027195708.png
1708027211228.png

opnsense is vanilla, no custom configurations made yet.
I'm noticing very low throughput, mostly in UL towards the opnsense box.
Code: 
VM (1Gbps NIC) to opnsense (1Gbps NIC)
root@deb:~# iperf3 -c 192.168.1.1 -t 10 -p 24798
Connecting to host 192.168.1.1, port 24798
[  5] local 192.168.1.31 port 35106 connected to 192.168.1.1 port 24798
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   115 MBytes   964 Mbits/sec    0   2.10 MBytes       
[  5]   1.00-2.00   sec   112 MBytes   944 Mbits/sec    0   2.10 MBytes       
[  5]   2.00-3.00   sec   112 MBytes   944 Mbits/sec    0   2.10 MBytes       
[  5]   3.00-4.00   sec   111 MBytes   933 Mbits/sec    0   2.10 MBytes       
[  5]   4.00-5.00   sec   112 MBytes   944 Mbits/sec    0   2.10 MBytes       
[  5]   5.00-6.00   sec   112 MBytes   944 Mbits/sec    0   2.10 MBytes       
[  5]   6.00-7.00   sec   112 MBytes   944 Mbits/sec    0   2.10 MBytes       
[  5]   7.00-8.00   sec   111 MBytes   933 Mbits/sec    0   2.10 MBytes       
[  5]   8.00-9.00   sec   112 MBytes   944 Mbits/sec    0   2.10 MBytes       
[  5]   9.00-10.00  sec   112 MBytes   944 Mbits/sec    0   2.10 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.10 GBytes   944 Mbits/sec    0             sender
[  5]   0.00-10.02  sec  1.10 GBytes   941 Mbits/sec                  receiver

iperf Done.

opnsense to VM

root@deb:~# iperf3 -c 192.168.1.1 -t 10 -p 4861 -R
Connecting to host 192.168.1.1, port 4861
Reverse mode, remote host 192.168.1.1 is sending
[  5] local 192.168.1.31 port 58598 connected to 192.168.1.1 port 4861
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  23.1 MBytes   194 Mbits/sec                  
[  5]   1.00-2.00   sec  25.3 MBytes   212 Mbits/sec                  
[  5]   2.00-3.00   sec  46.7 MBytes   392 Mbits/sec                  
[  5]   3.00-4.00   sec  35.3 MBytes   296 Mbits/sec                  
[  5]   4.00-5.00   sec  36.8 MBytes   309 Mbits/sec                  
[  5]   5.00-6.00   sec  35.4 MBytes   297 Mbits/sec                  
[  5]   6.00-7.00   sec  19.3 MBytes   162 Mbits/sec                  
[  5]   7.00-8.00   sec  28.4 MBytes   238 Mbits/sec                  
[  5]   8.00-9.00   sec  35.9 MBytes   301 Mbits/sec                  
[  5]   9.00-10.00  sec  33.6 MBytes   282 Mbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   320 MBytes   268 Mbits/sec  4458             sender
[  5]   0.00-10.00  sec   320 MBytes   268 Mbits/sec                  receiver

iperf Done.

Same tests from opnsense GUI

Result 1

Interface    ixl1
Start Time    2024-02-15 20:56:13 +0100
Port    24798
General

Time    Thu, 15 Feb 2024 19:56:54 UTC
Duration    10
Block Size    131072
Connection

Local Host    192.168.1.1
Local Port    24798
Remote Host    192.168.1.31
Remote Port    35106
CPU Usage

Host Total    10.14
Host User    2.65
Host System    7.49
Remote Total    0.00
Remote User    0.00
Remote System    0.00
Performance Data

Start    0    0
End    10.017673    10.017673
Seconds    10.017673    10.017673
Bytes    0    1177944064
Bits Per Second    0    940692764.8766335


opnsense to VM

Result 2

Interface    ixl1
Start Time    2024-02-15 20:57:45 +0100
Port    4861
General

Time    Thu, 15 Feb 2024 19:57:57 UTC
Duration    10
Block Size    131072
Connection

Local Host    192.168.1.1
Local Port    4861
Remote Host    192.168.1.31
Remote Port    58598
CPU Usage

Host Total    2.73
Host User    0.01
Host System    2.80
Remote Total    0.00
Remote User    0.00
Remote System    0.00
Performance Data

Start    0    0
End    10.000421    10.000421
Seconds    10.000421    10.000421
Bytes    335282176    0
Bits Per Second    268214448.9716983    0

Same results are seen also with Speedtests, almost 1Gbps in DL and 0.3Gbps in UL.
The 2nd iperf is reporting 4458 retries, while the DL one isn't. CPU is 10% while testing DL and 3% while doing UL.

I reinstalled PVE and opnsense from scratch twice to exclude misconfiguration on my end.
The same performances are seen using PCI passthrough of the Intel X710 AND using the vmbr bridge in Proxmox.

Any hint or idea? Considering that I'm waiting for a 10Gbps switch (my line is 8/2Gbps), I would like to squeeze this machine to max the line.
Thank you!
 

Attachments

F

Fazio

New Member
Dec 20, 2022
28
21
3
The issue seems related to opnsense, iperf between VM (on different host) and Proxmox (where opnsense is hosted) is OK (Proxmox IP is on 3rd NIC, management/LAN only).
Code: 
root@deb:~# iperf3 -c 192.168.1.11 -p 5201 -t 10


Connecting to host 192.168.1.11, port 5201


[  5] local 192.168.1.31 port 35924 connected to 192.168.1.11 port 5201


[ ID] Interval           Transfer     Bitrate         Retr  Cwnd


[  5]   0.00-1.00   sec   116 MBytes   971 Mbits/sec    0   3.14 MBytes       


[  5]   1.00-2.00   sec   111 MBytes   933 Mbits/sec    0   3.14 MBytes       


[  5]   2.00-3.00   sec   112 MBytes   944 Mbits/sec    0   3.14 MBytes       


[  5]   3.00-4.00   sec   112 MBytes   944 Mbits/sec    0   3.14 MBytes       


[  5]   4.00-5.00   sec   111 MBytes   933 Mbits/sec    0   3.14 MBytes       


[  5]   5.00-6.00   sec   112 MBytes   944 Mbits/sec    0   3.14 MBytes       


[  5]   6.00-7.00   sec   112 MBytes   944 Mbits/sec    0   3.14 MBytes       


[  5]   7.00-8.00   sec   112 MBytes   944 Mbits/sec    0   3.14 MBytes       


[  5]   8.00-9.00   sec   111 MBytes   933 Mbits/sec    0   3.14 MBytes       


[  5]   9.00-10.00  sec   112 MBytes   944 Mbits/sec    0   3.14 MBytes       


- - - - - - - - - - - - - - - - - - - - - - - - -


[ ID] Interval           Transfer     Bitrate         Retr


[  5]   0.00-10.00  sec  1.10 GBytes   943 Mbits/sec    0             sender


[  5]   0.00-10.00  sec  1.10 GBytes   941 Mbits/sec                  receiver





iperf Done.


root@deb:~# iperf3 -c 192.168.1.11 -p 5201 -t 10 -R


Connecting to host 192.168.1.11, port 5201


Reverse mode, remote host 192.168.1.11 is sending


[  5] local 192.168.1.31 port 36734 connected to 192.168.1.11 port 5201


[ ID] Interval           Transfer     Bitrate


[  5]   0.00-1.00   sec   112 MBytes   940 Mbits/sec                  


[  5]   1.00-2.00   sec   112 MBytes   941 Mbits/sec                  


[  5]   2.00-3.00   sec   112 MBytes   941 Mbits/sec                  


[  5]   3.00-4.00   sec   112 MBytes   941 Mbits/sec                  


[  5]   4.00-5.00   sec   112 MBytes   941 Mbits/sec                  


[  5]   5.00-6.00   sec   112 MBytes   941 Mbits/sec                  


[  5]   6.00-7.00   sec   112 MBytes   941 Mbits/sec                  


[  5]   7.00-8.00   sec   112 MBytes   941 Mbits/sec                  


[  5]   8.00-9.00   sec   112 MBytes   941 Mbits/sec                  


[  5]   9.00-10.00  sec   112 MBytes   941 Mbits/sec                  


- - - - - - - - - - - - - - - - - - - - - - - - -


[ ID] Interval           Transfer     Bitrate         Retr


[  5]   0.00-10.00  sec  1.10 GBytes   943 Mbits/sec  106             sender


[  5]   0.00-10.00  sec  1.10 GBytes   941 Mbits/sec                  receiver





iperf Done.
 
F

Fazio

New Member
Dec 20, 2022
28
21
3
The issue might be related to Proxmox VMs and not to opnsense itself..
Using a Debian LXC

Code: 
Debian via VMBR to bridged port of ISP router

Idle Latency:     2.26 ms   (jitter: 0.13ms, low: 2.16ms, high: 2.33ms)
    Download:   949.16 Mbps (data used: 428.2 MB)                                                   
                  8.36 ms   (jitter: 0.47ms, low: 2.17ms, high: 10.35ms)
      Upload:   948.09 Mbps (data used: 450.4 MB)                                                   
                  6.15 ms   (jitter: 1.47ms, low: 2.69ms, high: 8.72ms)
 Packet Loss:     0.0%
 
 Debian via VMBR to RouterOS VM on same Proxmox host
 
 Idle Latency:     2.53 ms   (jitter: 0.16ms, low: 2.36ms, high: 2.65ms)
    Download:   234.66 Mbps (data used: 287.2 MB)                                                   
               1148.81 ms   (jitter: 116.76ms, low: 2.15ms, high: 1930.23ms)
      Upload:     1.10 Mbps (data used: 1.6 MB)                                                   
                836.03 ms   (jitter: 83.31ms, low: 5.35ms, high: 2448.14ms)
 Packet Loss:    62.3%
 
S

SmashedSqwurl

New Member
Jun 3, 2019
11
1
3
I've been having some similar issues with an X710-DA2 PCIe card in a Supermicro system. iperf3 tests to and from my Vyos VM are all fine, but running from a client through the firewall to an upstream server gives terrible upload speeds.
 
F

Fazio

New Member
Dec 20, 2022
28
21
3
Currently using CHR, with vmbr bridges and I’m now getting full ISP performances, 8.2/2Gbps in WAN.
Guess something is not working as expected in my opnsense setup/Proxmox..
 
F

Fazio

New Member
Dec 20, 2022
28
21
3
SmashedSqwurl said:
Is that CHR under Proxmox, baremetal, or something else?
Click to expand...
I changed nothing on the host, nor on PVE. I just spin up a new CHR VM instead of opnsense, applied the same config (on PVE side) and everything was working at full speeds.
 
R

RTM

Well-Known Member
Jan 26, 2014
956
359
63
This is just grasping for straws here, but you could look into playing around with the hardware offloading features inside OPNsense:

Settings — OPNsense documentation

docs.opnsense.org docs.opnsense.org

If I am reading the docs correctly, they are disabling most features (which afaik is generally advisable), so you may want to play around with enabling them again.

You might also find inspiration in this article from Teklager.se:
teklager.se

OPNsense performance optimization

Optimization of OPNsense performance on low power devices (APU2, APU3, APU4)
teklager.se teklager.se

On the subject of grasping for straws: you might also get better performance if you can pin the CPU cores of the VM to p-cores on the CPU. Not that I have any idea on how to do this in Proxmox.
 
F

Fazio

New Member
Dec 20, 2022
28
21
3
the-last-englishman said:
You should only need to pasthrough 0000:02:00 to get both nics to show in opnsense

View attachment 34681
Click to expand...
Weird, I had the same config but got just 400Mbps in LAN/WAN..


RTM said:
This is just grasping for straws here, but you could look into playing around with the hardware offloading features inside OPNsense:

Settings — OPNsense documentation

docs.opnsense.org docs.opnsense.org

If I am reading the docs correctly, they are disabling most features (which afaik is generally advisable), so you may want to play around with enabling them again.

You might also find inspiration in this article from Teklager.se:
teklager.se

OPNsense performance optimization

Optimization of OPNsense performance on low power devices (APU2, APU3, APU4)
teklager.se teklager.se

On the subject of grasping for straws: you might also get better performance if you can pin the CPU cores of the VM to p-cores on the CPU. Not that I have any idea on how to do this in Proxmox.
Click to expand...
I disabled the hardware offload as advised, however the CPU load was very low and I don’t think was the cause, as the same config is performing flawlessly on Debian VM
 
T

the-last-englishman

Member
Aug 20, 2023
81
42
18
This is the default network interface settings I use in PFsense, maybe you have similar in OPNsense ?

1708286399782.png

Here is the options on the interface in the console:

1708286614603.png
And some more information on the pcie device from the console:

1708286844633.png
 
Last edited:
You must log in or register to reply here.
Top Bottom