Different Speeds from opposite ends of a VPN [Diagram]?

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

IamSpartacus

Well-Known Member
Mar 14, 2016
2,515
650
113
-u is correct. but you need to remove the max bitrate limit by setting to 0, take a look at the --bitrate option. even if you don't fill the pipe, look for asymmetry/symmetry when comparing in both directions.

if the asymmetry is not seen in udp and only in tcp, then you've narrowed the problem to something tcp related. if you're seeing it both in udp and tcp, then it is something layer 3 and below...
Are you referring to -b --bandwidth?
 

IamSpartacus

Well-Known Member
Mar 14, 2016
2,515
650
113
OK very interesting result with the UDP test.

Unraid2 to Unraid1 was as expected, pretty much full line speed of 956Mbps and 31% lost datagrams.. However, Unraid1 to Unraid2 came back with 3.16Gbps (3 times line speed) with 83% lost datagrams so something is clearly wrong.

Also I'm seeing a lot of OUT OF ORDER packets in both directions.
 
Last edited:

Blinky 42

Active Member
Aug 6, 2015
615
232
43
48
PA, USA
Hmm... Try the UDP test on the individual links in the unraid1 -> unraid2 direction and then the longer links in that direction to see if the packet loss appears on the local network or transitionsing on or both proxmox hosts. If you have a 3rd host in the internet as an you can do a udp test to from unraidX -> proxmoxX -> aws instance or whatever to double check there isn't any problems out to a 3rd party. I would hope that lets you zero in on the combination of links/hosts that are the culprit.
 

IamSpartacus

Well-Known Member
Mar 14, 2016
2,515
650
113
Hmm... Try the UDP test on the individual links in the unraid1 -> unraid2 direction and then the longer links in that direction to see if the packet loss appears on the local network or transitionsing on or both proxmox hosts. If you have a 3rd host in the internet as an you can do a udp test to from unraidX -> proxmoxX -> aws instance or whatever to double check there isn't any problems out to a 3rd party. I would hope that lets you zero in on the combination of links/hosts that are the culprit.
Unraid1 to pfsense1 is showing the same behavior. Though this time the client and server are showing different results. When I do the UDP test from Unraid 1 to pfSense 1 they each show the following result:

Unraid1: 3.38Gbps (72% lost datagrams)
pfSense1: 954Mbps (72% lost datagrams)


pfSense1 to pfsense2 I get the following:

pfSense1: iperf3: error - unable to read from stream socket: Resource temporarily unavailable
pfSense2: 0 bytes/sec - iperf3: error - select failed: Bad file descriptor


pfsense2 to Unraid2 I get the following:

pfsense2: 9Mbps (99% datagrams lost)
Unraid2: 0Mbps (99% datagrams lost, 5736 datagrams received out of order)


I don't really know what to make of these results.
 
Last edited:

Samir

Post Liker and Deal Hunter Extraordinaire!
Jul 21, 2017
3,257
1,447
113
49
HSV and SFO
Those checkboxes do disable the features. Are you suggesting to disable the disabling of those features?
Even though we know what the exepected behavior of what disabling these should do, you can test it and see if it reduces the slow link or leaves it the same--giving some more clues to what might be going on.

Those udp results are totally odd to me, but I don't have enough experience to know what it could be. :(
 

IamSpartacus

Well-Known Member
Mar 14, 2016
2,515
650
113
OK...I've finally isolated the issue. The slowdown only occurs if the client in Site 1 is connected to the LAN via 10Gb. Both my Unraiid01 server and my everyday PC in Site 1 have 10Gb (server is fiber, PC is Ethernet) connectivity and when I do transfers over their 10Gb connections, I only get roughly 1/3 line speed (200Mb). If I connect my PC to the network via 1Gb, I get the full VPN speed (600+Mb).

Now the obvious culprit would be jumbo frames but I have checked, double checked and tripled checked every NIC on both my PC, Unraid server, 10Gb switch, and pfSense and all are using the standard 1500MTU.

If I manually set the link speed of my PC to LAN connection from auto (10Gb) to 1Gb, I get full speed across the VPN.

Anyone have any other ideas as to why clients connected via 10Gb to the LAN would be seeing this kind of behavior?
 

BLinux

cat lover server enthusiast
Jul 7, 2016
2,669
1,081
113
artofserver.com
@IamSpartacus I would go back to the pcap trace and see if you can identify the issue in the pcap. you said you saw a lot of retransmits and out of order? what do the interface statistic counters look like?
 

Samir

Post Liker and Deal Hunter Extraordinaire!
Jul 21, 2017
3,257
1,447
113
49
HSV and SFO
I'm not exactly which links you changed from 10gb to 1gb so this might be something you already tried, but I would try setting the link on site 1 from the pfsense to the unraid from 10gb to 1gb and see if that fixes it. If it does, I'd suspect both nics and the interconnect and start by diagnosing those components.
 

IamSpartacus

Well-Known Member
Mar 14, 2016
2,515
650
113
I'm not exactly which links you changed from 10gb to 1gb so this might be something you already tried, but I would try setting the link on site 1 from the pfsense to the unraid from 10gb to 1gb and see if that fixes it. If it does, I'd suspect both nics and the interconnect and start by diagnosing those components.
I can't do any testing with regard to Unraid because that server only has dual SFP+ ports and thus I can't connect that to pfSense.

The testing I've been doing between 10Gb and 1Gb is with my main PC. The tests are using the same NIC (10Gb Ethernet) but I just manually change the link speed from 10Gb to 1Gb for the test. Changing the speed from 10Gb to 1Gb gets my full speed.
 

Samir

Post Liker and Deal Hunter Extraordinaire!
Jul 21, 2017
3,257
1,447
113
49
HSV and SFO
I can't do any testing with regard to Unraid because that server only has dual SFP+ ports and thus I can't connect that to pfSense.

The testing I've been doing between 10Gb and 1Gb is with my main PC. The tests are using the same NIC (10Gb Ethernet) but I just manually change the link speed from 10Gb to 1Gb for the test. Changing the speed from 10Gb to 1Gb gets my full speed.
Got it.

Do you have another PC on the same network as your own where you could repeat the test? Different hardware manufacturer would be a bonus.

Also, in the diagram you provided, it seems like the unraid is connected to the pfsense as well as win10 vm via a dac--is this correct? There is no 10Gb switch and all traffic bound for the pfsense must pass through the unraid?
 

IamSpartacus

Well-Known Member
Mar 14, 2016
2,515
650
113
Got it.

Do you have another PC on the same network as your own where you could repeat the test? Different hardware manufacturer would be a bonus.

Also, in the diagram you provided, it seems like the unraid is connected to the pfsense as well as win10 vm via a dac--is this correct? There is no 10Gb switch and all traffic bound for the pfsense must pass through the unraid?
I do have a 10Gb switch, I just didn't put it on the diagram for simplicity but I have just updated the diagram so check the OP again.

I have done the same test with laptop connected to the switch via 1Gb and it also gets full speed. Unfortunately I don't have another piece of hardware that has a 10Gb NIC to test with.
 

Samir

Post Liker and Deal Hunter Extraordinaire!
Jul 21, 2017
3,257
1,447
113
49
HSV and SFO
I do have a 10Gb switch, I just didn't put it on the diagram for simplicity but I have just updated the diagram so check the OP again.

I have done the same test with laptop connected via 1Gb and it also gets full speed. Unfortunately I don't have another piece of hardware that has a 10Gb NIC to test with.
Thank you. Does your system connect directly to the switch or to the unraid?

If it's connected directly to the switch, try another port. If it's connected directly to the unraid, try connecting it directly to the switch.
 

IamSpartacus

Well-Known Member
Mar 14, 2016
2,515
650
113
Thank you. Does your system connect directly to the switch or to the unraid?

If it's connected directly to the switch, try another port. If it's connected directly to the unraid, try connecting it directly to the switch.
Everything is connected to the switch, no direct connections. I have tried many different ports.
 

Samir

Post Liker and Deal Hunter Extraordinaire!
Jul 21, 2017
3,257
1,447
113
49
HSV and SFO
Everything is connected to the switch, no direct connections. I have tried many different ports.
Has this problem always existed? Or did you only recently notice or discover it? When was the switch last rebooted?

Are you able to test to see if you're getting 10g speeds to the local unraid?

On site 2, how are the 10Gb VMs connected to the unraid and pfsense?
 

IamSpartacus

Well-Known Member
Mar 14, 2016
2,515
650
113
Has this problem always existed? Or did you only recently notice or discover it? When was the switch last rebooted?

It's existed for as long as I've had 1Gbps fiber on both ends of the VPN (a little over a year). But I always chalked it up to VPN overhead. With the latest version of pfsense including some speed optimizations for IPsec, I decided to do some more testing to see if I could get more speed. This was the first time I tried transferring from Site 2 to Site 1 as a test and noticed the must higher speed. I have no real need to do that kind of transfer which is why I never did so in the past.

I rebooted the switch last week while troubleshooting this issue.


Are you able to test to see if you're getting 10g speeds to the local unraid?

Yes. I have full 10Gb connectivity between my PC and local Unraid. I always have as I do lots of transfers from my PC to Unraid. I just tested it again just now and get 1.09GB/s.

On site 2, how are the 10Gb VMs connected to the unraid and pfsense?

The 10Gb connection is just the virtual NIC connection between the VM and host hypervisor. It's not a physical 10Gb connection.
See answers in bold.
 

mstone

Active Member
Mar 11, 2015
505
118
43
46
I have done the same test with laptop connected to the switch via 1Gb and it also gets full speed. Unfortunately I don't have another piece of hardware that has a 10Gb NIC to test with.
If it only affects 10Gbe NICs it sounds like something can't handle receiving at that data rate and drops packets. This would trigger a backoff in transmission rates. This should converge at something near the available bandwidth, but it sounds like your congestion control is screwed up. You might also check your ethernet flow control settings and make sure they're consistent. If they're all currently on turn them off, and vice versa. (Flow control is often a bad idea, but sometimes helps.)
 

IamSpartacus

Well-Known Member
Mar 14, 2016
2,515
650
113
Any of you familiar with pfSense traffic shaping. I think if i setup a limiter from my 10Gb hosts (my server and PC) when connecting across the VPN, it may fix my issue.