Poor network performance after jumping to Fiber

[c2cahoon]

Hello All,

I recently jumped to symmetrical gig fiber and am having issues with my guest vm’s getting very subpar network performance.
My setup is

pfsense router running latest version (whitebox)
6600K, 16GB RAM, Intel 82576 NIC.

vmware host running 7.0
Xeon D-1541, 64GB ram, Intel x552 and 82599 NIC’s connected with 10GbE SFP+ adapters.

I have my ONT connected to the pfsense box, the ATT router on another port and then an unmanaged gigabit switch connected to disperse the LAN side. Two connections go to my VMware host and are both configured as an uplink.

If I connect my laptop to the switch directly I am able to get Speedtests around 800+ both up and down. From a Windows based VM I am seeing 300/20. Linux based I see 300/300.

I’ve replicated the network performance issue in Proxmox so I don't think it's hypervisor related,l making me think I have a hardware problem, or just poor configuration.

Any thoughts on how I can further troubleshoot this?

Thanks

 

[j_h_o]

I'm afraid I don't understand the way the ONT is connected. Can you clarify the physical connections you have? How is the SFP module connected to the pfSense 1000Base-T port? What is connected to the router?

How does your ISP handle authentication? Is there PPPoE, or do you just get an IP via DHCP?

 

[c2cahoon]

This should clear up some of the confusion.

GitHub - uchagani/pfatt: enable true bridge mode for AT&T U-Verse and pfSense

enable true bridge mode for AT&T U-Verse and pfSense - uchagani/pfatt

github.com

Physically the ONT from ATT connects to port 0 of my pfsense NIC.
Port 1 is connected to the LAN switch
Port 2 is connected to the ATT Router they provide which is required for the 802.1X authentication.

The LAN switch has two connections to my hypervisor host using sfp+ to 10GbE adapters.

I believe the router is fine as I was able to directly connect to the attached switch and get near full speed(at least expected speeds). Its only the hypervisor and it's guests that seem to exhibit the poor performance.

 

[c2cahoon]

Another discovery

I was using the CLI tool for speedtest.net, I couldn’t find an iperf server to target that was capable of gigabit transfers.

I recently switched to SpeedTest++ after stumbling upon it in another post

Testing download speed (32) ..... Download: 1003.14 Mbit/s
Testing upload speed (12) .... Upload: 315.20 Mbit/s

This seems that only the upload is suffering degraded performance. Downloading a large file I was able to confirm that I can see near expected downloads but the upload appears to be accurate at around 300 Mbit/s.

 

[Blinky 42]

I would caution against trusting any of the speedtest sites's results as gospel without testing a number of servers and across all times of day/night/overnight. Quick experiments with the cli version on my 1G Fios would show 900/900 one day, and 300/300 the next against the same server, and in the span of 15 min the other afternoon trying different servers I got
505/533, 561/543, 320/332, 544/481, 300/591, 313/605, 743/499, 655/598, 672/668, 731/352
The last site was a cable system I know the admins at and know for sure they only have a 1G transit link that the speedtest traffic could go over so I was using all the b/w that others in the headend were not using at that moment.

oh - And I at the time I could confirm that I get 900 both ways hitting my own servers at colos with >1G connections available.

 

[c2cahoon]

Sorry It took me a minute to get back to this.

On the host running a live CD I am able to see 800 down and 125 up. I believe the upload limit might be limited by the boot media. I'm going to get a spare SSD with linux on it to test again. I don't believe there is going to be an issue with the hardware achieving the full gigabit speed.

My next immediate test is to rent a server capable of gigabit and run some iperf tests to have something more accurate.

 

[Kev]

Check all those offloads in both pfsense and VMware. Disable and enable them one by one to test.