Problem with disk or network performance - server 2012 R2 with hyper-v role

[Neandertaler]

Hello everybody,

Have been reading for some time and have now signed in and looking forward to exchanging about our server hobby. Sorry for any weird language as english is not my native tounge.I am just setting up a small “playserver” homelab environment for hyper-v. Server is a supermicro based box with a dual core haswell 3,2 Ghz pentium, 8 GB ram and 2 on board intel NICs (plus IPMI). I have installed Windows Server 2012 R2 Standard with hyper-v role onto a software mirror on two very old laptop drives as I have read that the management OS does not need high IOPS or disk bandwidth. This software mirror is host drive c:

Furthermore I have 4 very old 1TB WD green drives, set up as a 2-way mirror storage space with 2 columns, wich is kinda like a raid 10. That SS mirror is host drive F: where all VHDX files are located.

When benchmarking the SS drive F: from within the physical host (the server R2 standard with hyper-v role enabled), I get the following results:



Not that bad for 7 years old 5400 rpm consumer drives. Then I have set up a generation 2 VM with Windows server 2012 R2 with the essentials role enabled. This VM has a C: drive vhdx for the OS and an E: drive vhdx for the data. Both vhdx files reside on the SS mirror. When benchmarking the drives from within the virtual R2 server I get the following results:

Drive C: OS


Drive E: DATA


A certain loss for virtualisation, but acceptable none the less.

I have teamed the two host NICs and connected the team to a virtual external switch. VMs are connected to this one and only virtual external switch. The team is switch dependent and I have configurated my GS110TP switch accordingly.

When benchmarking from an external physical client like a laptop connected to my switch onto a fileshare on the virtual servers E: drive, I get very slow results:



This is a loss of performance of up to 66%! The cpu runs appr. At 65-75% within the VM and 15-45% on the hyper-v host while benchmarking.

I have set up a second VM which is a Generation 1 Windows 7 pro VM with only 1 OS disk, which is drive C:. The VHDX also resides on the host´s F: drive which is the SS mirror. When measuring from within the WIN 7 VM, I get the following results:



Somewhat comparable to the data of the server VM and OK. But when I benchmark a fileshare on the virtual server from the win 7 VM, bandwidth totally drops to unacceptable levels:



And now comes the weird part: I have set up a fileshare on the hyper-v host on the F: drive, where the VM files are located in order to check what happens when I access the fileshare on the physical host from the win 7 VM. That way I hoped to see if the problem is low disk performance or low network performance. I get the following results:



Now I am not shure what to think and where to look. I was of the opinion that putting 4 old drives in a striped mirror would give me enough power to “almost” saturate a gigabit network in most cases. But somewhere in this setup the performance gets lost and I do not know where……I know the hardware is very weakish…….but can this be the only reason? I could just solve the problem with money, better hardware and SSDs, but inmho this does not make sense without understanding what is happening in the first place.

Any tips or insight regarding this issue…. ?

 

[Neandertaler]

short update:

i have checked the NIC Drivers on the host and the guests and everything is current. Most current integartion services are installed. Furthermore I have set up a second Generation 2 Server 2012 R2 guest in order to check bandwidth between two virtual 2012 R2 Servers in order to rule out Generation 1 Windows 7 VM as a cause. Same result: 20-30 mb/s sequential .. :-(

 

[ultradense]

First, have you tried to unplug one networkcable from your team? It might be team-related.

Also, with old nics/drivers in combination with 2012, some new speed-features only create problems. I often play with these settings (first on the host, if it helps then on the guest too). Try one at a time and test in between. It might also be a combination of 2 (so you can try to disable all).

netsh int tcp set global chimney=Disabled
netsh int tcp set global autotuninglevel=Disabled
netsh int tcp set global congestionprovider=None
netsh int tcp set global ecncapability=Disabled
netsh int ip set global taskoffload=disabled
netsh int tcp set global timestamps=Disabled

To validate type:
netsh int tcp show global
Or check the advanced properties of the physical nics in the host.

Sorry, don't have time to explain them all, but Google should help.

 

[ultradense]

Specifically, try to disable LSO (large send offload) for ipv4 and ipv6.

 

[Neandertaler]

first of all thanks for the Feedback. Although I do not think it has anything to do with the teaming as I had the same low Speed before teaming, I was actually planning to break the team and use one virtual Switch per NIC and use one physical NIC per VM to see what happens then. The NICs are on-board (supermicro X10sll-l, Intel I217-LM and I210)..so fairly new and non broadcom. I will nevertheless try out your suggestions.....

 

[Neandertaler]

pulled one cable of the Team: no difference. disabled all the mentioned stuff on th ephysical NICs on the host: no difference. But:

I have found a hyper-v warning in the Hosts log:

MAC conflict: A port on the virtual switch has the same MAC as one of the underlying team members on Team Nic Microsoft Network Adapter Multiplexor Driver

Hmmmm, having the same MAC twice seems like that could cause Problems. I just checked: MAC adress is set to "dynamic" in the guest Setting (probably Default). should i set this to static and use different MACs for each VM? And what is MAC spoofing, which is another Option?

 

[Neandertaler]

broke up the Team which solved the MAC address Problem. Still no better bandwidth. Now i will try to use two virtual Switches....

 

[Neandertaler]

OK....tried all the different Settings and found that the single Setting causing the low Speed is the detailsetting of the "side scaling" in the NIC Settings. I have no idea why, but the CPU Setting for side scaling was set to 8 cpus, while I only have a dual core. Now set to 2 cpu´s, I get 75 mb/s read and 65mb/s write. No matter what other Settings i was playing around with, especially with all the offloading Features,.....it did not get any faster than that.

 

[ultradense]

Yep, RSS often creates problems... Glad you found some culprit.

 

[Neandertaler]

Well thanks again for the tips you gave me. I have performed numerous tests and finally got >100 mb/s between VMs. The X10SLL-F board I am using has one Intel I210 and an I217LM NIC. I have cerated 1 virtual Switch "vSwitch_210" and 1 virtual Switch "vSwitch_217" connected to the matching physical NIC. I only get the full Gigabit Speed when attaching the VMs to the vSwitch_217, which is weird as i always thought that Transfer between VMs only runs "in Software" and the Features and actual Performance of the physical NIC does not influence that bandwidth. When attaching the VMs to the vSwitch_210, I am back to 75 mb/s. I will now test in more detail:

a) 1 VM on vswitch_210 and one VM on vSwitch_217
b) both VMs on vSwitch_210
c) both VMs on vSwitch_217 (>100 mb/s)
d) variant a reversed

and test the resulting Speeds in several runs........I assume I will have to find the appropiate NIC Setting for each NIC seperately in order to maximize Speed. When that is all finalized I will again go the NIC teaming route..... Now I am thinking about getting a dual Intel I350 NIV additionally (as I want to fiddle around with a virtual pfsense in the future; I know....virtual firwall is suboptimal... ;-)