Different Speeds from opposite ends of a VPN [Diagram]?

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

IamSpartacus

Well-Known Member
Mar 14, 2016
2,515
650
113


I'm noticing some weird traffic behavior between these two LAN networks on opposite ends of a Site-to-Site IPsec VPN (routed IPsec in pfSense).

If I run iperf3 or initiate a samba transfer from Site 1 to Site 2, I'm getting a throughput of about 200Mbps. However, if I initiate either iperf3 or a samba file transfer from Site 2 to Site 1, I'm getting over 600Mbps. Both pfsense boxes are running identical hardware and identical packages (snort and pfblockerng turned off for these tests). Both c2758 CPUs are at the same CPU usage during the transfers.

I drew out the above diagram mainly just for myself to confirm if everything looks good and I'm not seeing a networking issue. Nor do I believe a networking either would cause this slowdown for just one direction of the connection. Am I crazy here? And if not, can someone point me in a direction to start looking for the culprit to my issue?
 
Last edited:

Tom5051

Active Member
Jan 18, 2017
359
79
28
46
could the uplink speed at site 1 be slower than site 2? Is the pfsense device servicing the VPN traffic using more CPU at site 1?
 

IamSpartacus

Well-Known Member
Mar 14, 2016
2,515
650
113
CPU usage is equal on both ends during the transfers. Speedtests to the internet show full bandwidth and my iperf3 testing results are posted below. All tests were done using the WAN interface to eliminate VPN overhead being the issue.



pfSense 1 --> pfSense 2 - 607Mbps
pfSense 2--> pfSense 1 - 595Mbps

Unraid 1 --> Unraid 2 - 154Mbps
Unraid 2 --> Unraid 1 - 897Mbps

Unraid 1 --> pfsense 1 - 907Mbps
pfSense 1 --> Unraid 1 - 815Mbps

Unraid 2 --> pfsense 2 - 888Mbps
pfSense 2 --> Unraid 2 - 703Mbps
 

BLinux

cat lover server enthusiast
Jul 7, 2016
2,669
1,081
113
artofserver.com
is the link between unraid2 and pfsense2 only 1Gbps like in your diagram? vs at site 1, it is 10Gbps?

i'm wondering if there's some TCP window size thing going on... unraid1->pfsense1 can handle larger window, but by the time it gets to pfsense2->unraid2, it needs to break it down, adding latency. just speculation... easiest way to see what's going on is to run tcpdump or wireshark while running the test. can you also run a UDP test?

when there's asymmetry, i usually look for asymmetry in the network, so i noticed the 1Gbps @ site2 vs 10Gbps @ site1.

what do you get if you run the test from:

unraid1->pfsense2?
unraid2->pfsense1?
 

IamSpartacus

Well-Known Member
Mar 14, 2016
2,515
650
113
Yes, the link between unraid2 and pfsense 2 is only 1Gbps. Site 2 is my offsite backup location so I don't need more than 1Gbps on that size.

unraid1 -> pfsense2 = 668Mbps
unraid2 -> pfsense1 = 687Mbps

I will run some more extensive tests with wireshark late tonight and some UDP testing.
 

IamSpartacus

Well-Known Member
Mar 14, 2016
2,515
650
113
Alright, ran some more tests today and I've definitely narrowed the the issue down to being between 10.10.50.x and 192.168.20.x. It doesn't matter what host I run testing from on 10.10.50.x, the speed loss is the same. I then ran wireshark on my PC (10.10.50.50) while doing an iperf3 test from my PC to Unraid2 (192.168.20.30) and I'm getting a bunch of TCP Retransmissions and TCP Out-of-Order errors. I don't have much experience troubleshooting with wireshark so not really sure where to go from here.
 

IamSpartacus

Well-Known Member
Mar 14, 2016
2,515
650
113
Do you have an MTU mismatch?
Not that I can see. Jumbo frames is off on my switch in Site 1 (Site 2 just has a dummy switch) and both pfSenses are using the default MTU's for their interfaces. Both Unraid boxes are also using default MTU.
 

mstone

Active Member
Mar 11, 2015
505
118
43
46
I'd guess routing problem where something's looping around unnecessarily
 

kapone

Well-Known Member
May 23, 2015
1,095
642
113
I see that both sites have FIOS. Are the fios speeds identical (or close to) in both sites?

Run a dslreports speed test per site, and see if the results are similar. The bufferbloat results may be interesting as well.

Edit: I see that you ran speedtests already, but humor me and run the dslreports test.
 

BLinux

cat lover server enthusiast
Jul 7, 2016
2,669
1,081
113
artofserver.com
Not that I can see. Jumbo frames is off on my switch in Site 1 (Site 2 just has a dummy switch) and both pfSenses are using the default MTU's for their interfaces. Both Unraid boxes are also using default MTU.
are you able to check duplex settings? check if there's a mismatch some how if auto didn't negotiate correctly.

also, i know you said you saw tcp re-trans, but did you run the udp test? if the problem mostly manifest in tcp, i would look closer at tcp window...
 

vanfawx

Active Member
Jan 4, 2015
365
67
28
45
Vancouver, Canada
as far as I know, everything from gigabit up is full duplex. I don't think you can even force half-duplex on a gigabit link. If you look at ethtool, you'll notice that only 10/100 offers full/half duplex options:

Settings for eth0:
Supported ports: [ TP ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Supported pause frame use: Symmetric
Supports auto-negotiation: Yes
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Advertised pause frame use: Symmetric
Advertised auto-negotiation: Yes
Speed: 1000Mb/s
Duplex: Full
Port: Twisted Pair
PHYAD: 1
Transceiver: internal
Auto-negotiation: on
MDI-X: on (auto)
Supports Wake-on: pumbg
Wake-on: g
Current message level: 0x00000007 (7)
drv probe link
Link detected: yes

For fun, have you tried disabling any of the offload features on your NICs on pfsense? It's possible you could be running into an edge case there.
 

IamSpartacus

Well-Known Member
Mar 14, 2016
2,515
650
113
I see that both sites have FIOS. Are the fios speeds identical (or close to) in both sites?

Run a dslreports speed test per site, and see if the results are similar. The bufferbloat results may be interesting as well.

Edit: I see that you ran speedtests already, but humor me and run the dslreports test.
Site 1: Speed test result
Site 2: Speed test result


are you able to check duplex settings? check if there's a mismatch some how if auto didn't negotiate correctly.

also, i know you said you saw tcp re-trans, but did you run the udp test? if the problem mostly manifest in tcp, i would look closer at tcp window...
I've been looking for good examples for a udp test with regard to what switches to use. Can you recommend one? I tried simply adding -u and it gave me terribly poor results (under 1Mbps).


For fun, have you tried disabling any of the offload features on your NICs on pfsense? It's possible you could be running into an edge case there.
How is this done?
 

kapone

Well-Known Member
May 23, 2015
1,095
642
113
Hmm...while there is some variance in the speed tests, it can't account for the difference you're seeing. There's something else going on..
 

vanfawx

Active Member
Jan 4, 2015
365
67
28
45
Vancouver, Canada
Those are the defaults. Try disabling TCP Segment offload and LRO. This usually isn't an issue on bare metal hardware, but you've checked everything else, so why not?
 

IamSpartacus

Well-Known Member
Mar 14, 2016
2,515
650
113
Those are the defaults. Try disabling TCP Segment offload and LRO. This usually isn't an issue on bare metal hardware, but you've checked everything else, so why not?
Those checkboxes do disable the features. Are you suggesting to disable the disabling of those features?
 

BLinux

cat lover server enthusiast
Jul 7, 2016
2,669
1,081
113
artofserver.com
I've been looking for good examples for a udp test with regard to what switches to use. Can you recommend one? I tried simply adding -u and it gave me terribly poor results (under 1Mbps).
-u is correct. but you need to remove the max bitrate limit by setting to 0, take a look at the --bitrate option. even if you don't fill the pipe, look for asymmetry/symmetry when comparing in both directions.

if the asymmetry is not seen in udp and only in tcp, then you've narrowed the problem to something tcp related. if you're seeing it both in udp and tcp, then it is something layer 3 and below...