Routing is driving me nuts

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

Switchback028

Member
May 18, 2018
61
26
18
So I've been working on this problem for two weeks now and I'm completely stumped.

Basically, I moved to the west coast and have Webpass for my internet. However, Webpass has a really poor connection from San Diego to New York City, where I have my server colocated.

My original plan was to bypass this speed limitation by renting a VPS and routing my traffic manually. This works because the Webpass -> VPS link doesn't go via the Hurricane Electric link that is slow.

So here's the dilemma. When I've got everything in place, and traffic is routing via the VPS my speeds are crap (15Mbit/s). However, going on either leg of the journey the speeds are much better (data below). I'ved tried multiple protocols (IPSec over GRE, Straight GRE, Wireguard) with the same result. The firewalls on both the SAN1 and NYC1 end are PFsense 2.3.2.

ISP Speeds:
Colo - Gig Symmetric
VPS - Gig Symmetric
Webpass - Alleged Gig Symmetric (Never really gotten a full gig, more like 500Mb/s up/down)

VPS OSs tested:
HE/OVH Transit- Debian 10 (GRE, IPSec over GRE)
Cogent Transit- Debian 11 (GRE, IPSec over GRE, Wireguard)
Cogent Transit- VyOS 1.1.7 (GRE)

I've tested this three ways now:
IPSec over GRE
Straight GRE
Wireguard to VPS and VPS to Colo

Every time:
SAN1 -> VPS is good (300Mb/s)
VPS -> NYC1 is good (800-900Mb/s)
NYC1 -> VPS -> SAN1 is bad (5-15Mb/s)

Does anyone know or have any insight as to why this is not functioning as it should?
 

edge

Active Member
Apr 22, 2013
203
71
28
What is the cpu utilization of the pfsense firewalls when VPS is and is not used?
 

Switchback028

Member
May 18, 2018
61
26
18
While running an iperf test both firewalls are under 10% utilization. For reference specs are:
SAN1: i5-5200U (Physical System)
NYC1: CARP instances, 2x virtual machines with 4vcpus each (e5-2680v4)