Windows Server 2012 R2 + Hyper-V and 3 x 1Gb NICS in a team with poor network speeds

[TallGraham]

Thanks again to dba and britinpdx for the feedback. I think we can confirm this is the issue regarding digitally signing SMB shares on a domain controller.

I don't really want to leave my settings like this dba as it lessens security. I'll see if I can find a workaround, and if not try and virtualise the DC I think. I will keep you all posted here.

 

[mrkrad]

DC's disable write-back caching no matte what!

Also make sure you check your flow control settings o your NIC/SWITCHES! By default it is on (along with IPV6).

You cannot disable IPV6 properly on DC's in 2008R2 any more that is moot make sure your vm's are not dropping packets as tall, you should see a mic defer packets on a port a few times a day but ZERO drops per day is tolerable!.

Power management inside the VM also as well as on the hpervisor make a huge difference too!.

Also if you are using any advanced stack tcp/ip features make sure you plenty of vcpu to handle the traffic as well.

These same problems magnifies themselves in 10ge ethernet as well! if not more so!

 

[Lost-Benji]

Ok, I have come into the end of this thread but in the OP, I already got bad feelings and feel they should be addressed sooner rather than later. My comments will be in RED for the quote:

It is a domain controller running DNS and DHCP as well so probably breaking all the rules! Whats wrong with that? It is doing what is supposed to do...

3 x RAID arrays. See further down for the ideas on your chosen RAID
- 1 for OS (RAID5 4 x SSD)
- 1 for Data (RAID6 8 x 1TB),
- 1 for VMs (RAID6 8 x 1TB)

Supermicro X9SCM-iiF motherboard with two onboard Intel 82574L Gb NICs on board
Also 1 x quad port Intel Pro 1000 PT network card

Everything is using standard Windows 2012 R2 drivers. I then created an Ethernet Team using the "Windows teaming". This uses one of the onboard NICs and two from the Quad port PT card. Why do this? It is just looking messy straight up.
This is set to Switch Dependant and Dynamic as they seemed the best options. I have also created a LAG on my Cisco SG300-28 switch that all 3 plug into. It is a static LAG and not using LACP. Try LACP, has worked best with my S2012 and Win8 Teams.

Now in the Host OS I can copy between the two RAID6 arrays and get speeds of 500MB/s+. So that is great, the disk speed is OK. No, you haven't proven anything. No mention of the HW RAID card or what areas you are testing from and to, these speeds are what I would expect on the SSD's but not with the HDD's, especially with the choice of RAID-5 & 6. Speaking of which, not sure why you have chosen parity but I feel this wasn't a great choice. RAID-10 for ALL of the arrays. Parity, especially RAID-6 takes a lot of grunt and usually has large write performance hit. I am not stating this to argue as it will just end a pissing match, what I am saying is, make sure your reasoning is sound and that your drive-arrays actually write faster than the network speeds.

I have a lot of experience in the past using LAG on physical servers and getting better transfer speeds between them. Always used to use either the Intel ProSET software or the Broadcom equivalent to get the job done. Used the Windows one here as I thought it would be OK (or maybe not) I wouldn't blame it yet.

Now I am aware that virtualization is a bit more complex and I think I am just confusing myself now with RSS, vRSS, and VMQ. So want to try and get some help from here please.

I have two VMs set up on the server.
- Windows 2012 R2 server
- Windows 7 Professional SP1

Now if I copy about 14GB, several large ISO files, of files between them then I get transfer rates of almost almost 1Gb, or about 100MB+. Very nice, but I would expect more as the virtual NICs and the virtual switch all state they are 10Gb. I have the team of 3 x 1GB cards going into the virtual switch, but it seems like I only ever get the speed of one of them. These speeds between the VM's should run at the drive speeds as the 10GbE Virtual NIC won't be the limit. What will kill this is that you are moving files from one VHD to another on the same array which as I mentioned already, feel that you have poor write speeds to start with.

Now is gets a bit more complex. The Host OS shares the team as well and connects through a virtual NIC to the virtual switch. Again the OS says this is 10Gb. You are probably all cringing now and wondering why I have done this like this. Sorry if it is wrong. I was already cringing, now I am just mellowed. Please put this right before you go any further. Put the Host on a single on-board NIC and the team the quad-port for the HyperV. If you need faster Host speeds, team it. Avoid mixing your team across different hardware and buses at this stage.

If I try to copy the 14GB from the Windows 7 VM to a share on the Host OS then I get speeds of between 20MB/s - 60MB/s if I am lucky. Also when I do this copy to the host OS the CPU on the VM maxes out at 100% for the duration of the copy. I tested this in the Windows 2012 Server VM as well and the same thing happens with CPU usage and network speed. Again, poor choices in RAID and arrays needs to be cleared out of the equation.

RAID setup suggestion:
Host - Mirrored pair of HDD's unless you are low on RAM and it needs to page off a lot.
VM's - RAID-10 SSD's after you confirm that you can TRIM the drives (unlikely) or use SSD's that have foreground GC (Samsung Pros, not Intels that are background, background won't get a chance if the drives are in use all the time). bulk data should not be on here, use the Data arrays for that.
Data - RAID-10 volumes of up to 8 drives a piece. RAID-5 or 6 parity is fine too, just keep in mind that the parity calcs take balls to get real speeds, if you are going to have high I/O then RAID-10 arrays best.

NIC's:
2x On-board - Host & Management, not shared with the HyperV.
4x Quad-port NICs in Team and assigned to the HyperV dedicated.

If you test from Host to VM's, traffic now should pass out to the switch before returning so keep this in mind however, if the arrays are sorted out, then drive speed on the Host may limit before the NIC speeds.


There has been some really good testing and ideas from Brit and DBA and feel they too have got the right ideas. I just see that there is bigger issues that need to be sorted before fussing over claimed network issues.

 

[TallGraham]

Hi Lost-Benji

Thanks for the reply. Input is always welcome but I think you should have read the rest of the thread and checked out the server build thread over in the DIY servers section too.

Just to answer some of your concerns/questions:

The Hyper-V host also being a domain controller is what actually turned out to be the problem. There is a link above to a blog where it is explained. There are 2 group policies that get implemented by default on domain controllers and involve digitally signing SMB traffic. Once these were disabled the network speed bottlenecks disappeared totally. dba very kindly backed up my findings of slow speeds by replicating it on his Hyper-V host which is also a domain controller as well. britinpdx could not replicate the issue because his system was in a workgroup environment.

The server is a project and proof of concept for me about the new technologies and the way they work.

All the RAID arrays are on their own separate Adaptec hardware RAID controllers. The OS array is on a 6405, and both the VM and Data arrays are on 5805 cards. These are more than capable of dealing with the parity requirements and overheads. I would even doubt with these mechanical drives there would be any difference in speed at all between a RAID10 and a RAID6 array. Quite possibly with the added speeds of SSDs though.

The whole point of the disk copy which clocked at over 500MB/s did actually prove a lot. It was copied from the array where the VM VHD resides to the array where the the data shares reside. This showed that the underlying hardware was actually more than capable of sending this data out at a high enough rate that would saturate the 1Gbe connection. It is all well and good having a 1Gbe NIC that can theoretically transfer at 125MB/s but if you are moving data from an old IDE hard disk that can only max out at 50MB/s then you are never going to achieve full network throughput as the underlying hardware simply cannot supply the data that quickly.

The reason for using 2 ports from the Intel PT Quad port cards and one of the onboard NICs in a team is simple. Redundancy. If the Intel PT card fails I still have some connectivity. If I had two switches in a logical switch stack then I would also spread the connections between the switches for the same reasons. All the speed tests that were done later on were using just 1Gbe NIC and no teaming, so the spread was irrelevant then as I had gone back to basics.

There are clearly several schools of thought now on using NIC teams now in Windows Server 2012 and vNICs on the top of that. You can team everything together on the vSwitch and then have separate vNICs for Host, Management, Hyper-V, iSCSI, etc with a quality of service (QoS) applied to the vSwitch. This solution then means that you can lose all bar one of your physical NICs and you will still have connectivity to every service that you have a vNIC for. It might be a bit slow but you will have connectivity to all services.

There do appear to be various issues with this though surrounding which type of teaming you choose. Also how would MPIO work with iSCSI on the vSwitch and just 1 x 10Gbe vNIC? Do you even need it with a 10Gbe vNIC? Should you just enable more than one of them and do it like that? It is all quite complex when you first read it all and set it up. Add in RSS, vRSS, and VMQ and my brain melted the other day and fell out of my ear.

I have a couple of network designs that I will be posting up in the build section to ask for comments from other users. Please feel free to take a look and add you thoughts. As I said it is a proof of concept for me and all input is welcome

 

[Lost-Benji]

No worries. I only skimmed the thread and when it was the same stuff over and over then combined with the RAID/server setup, it screamed back to basics.

I come across servers on a daily basis that are under performing and 98% of the time its thanks to poor implementation of the RAID and Disk structures hence my suggestion and concerns for the chosen uses currently.


I will look for your other threads,
Cheers.



LB