Hyper-V 2012 network design questions

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

wuffers

New Member
Dec 24, 2012
19
0
0
So I haven't gotten around to posting any details on the SAN / infrastructure upgrade project I've been working on. All the hardware is now in (Supermicro substituted two sets of parts adding to the delay, and I had to wait quite awhile for my low profile Infiniband HBA brackets - but that's another story) and I'm now in the midst of racking (actually racking is done) and cabling. And let me apologize for the somewhat long post. :p

Are there any Hyper-V experts lurking around to help me revise/tune my network design for a Hyper-V Cluster? I have read the following and still undecided on how best to proceed:
Hyper-V : Network Design, Configuration and Prioritization : Guidance - TechNet Articles - United States (English) - TechNet Wiki (I realize this is not for Hyper-V 2012 but the requirement is the same)
10Gbps | Working Hard In IT (4 part series on 10GbE and Hyper-V)

The diagram (connectivity per Hyper-V host):

Notes and environment details:
- I only have 21 ports on the access panel in the rack for RJ-45 connections (which can be patched to my stack of 3750s). I'd like to use only as many RJ-45 ports as necessary.
- The PowerConnect 8024F have 4 combo ports to support both SFP+ and 10GBase-T (so 8 in total)
- Both 10GbE media types are Intel cards, so are SR-IOV capable
- Cluster of 3 Hyper-V hosts for now, may expand later
- Used for TFS Lab Management (lots of VMs and most likely many VM created/destroyed)
- Blue connectivity boxes are done - reds are the ones I have questions about
- I'd like to only use one 10GbE combo port on the 8024F per host
- SAN traffic will stay segregated on the Infiniband side

Questions:
- What type of traffic does the Hyper-V management contain? If I need to deploy lots of VMs is 1GbE enough?
- Live Migration traffic seems to be dormant most of the time. Can I just piggyback on the teamed VM Network trunk and assign an IP address with a private native VLAN? Partition using SR-IOV?
- Or an alternative is to put Live Migration on its own private VLAN on a X540 10GbE port.
- I've put the Cluster Heartbeat (failover) and CSV together. I understand that Redirected I/O could be important. Is 1GbE enough? Maybe put this in a partition on the VM Network team?

I know there are a lot of options and I'm going slightly crazy thinking of them. I just need some injection of sanity by someone who's been through it.
 

markpower28

Active Member
Apr 9, 2013
413
104
43
Check the following link for reference Q and A: I only have two NICs on my Hyper-V host. Should I team them or not? - Jose Barreto's Blog - Site Home - TechNet Blogs

I will leave the management NIC on the build-in NIC. Management Network is for SCVMM to manage Hyper-V. 1 GE NIC is more than enough.
For CSV and Live Migration, I will created additional vLans for them on 10 GB NIC (both CSV and Live Migration can benefit from 10GB NIC)
8024 is a L3 managed switch, after LACP/LAG, I will do one vLan for VM traffic(public traffic), one vLan for CSV(private between hyper-v), one vLan for HB(private between hyper-v) and one vLAN for Live Migration(private between hyper-v).

What kind of storage you plan to use? IB network on SMB 3.0?
 
Last edited:

wuffers

New Member
Dec 24, 2012
19
0
0
Check the following link for reference Q and A: I only have two NICs on my Hyper-V host. Should I team them or not? - Jose Barreto's Blog - Site Home - TechNet Blogs

I will leave the management NIC on the build-in NIC. Management Network is for SCVMM to manage Hyper-V. 1 GE NIC is more than enough.
For CSV and Live Migration, I will created additional vLans for them on 10 GB NIC (both CSV and Live Migration can benefit from 10GB NIC)
8024 is a L3 managed switch, after LACP/LAG, I will do one vLan for VM traffic(public traffic), one vLan for CSV(private between hyper-v), one vLan for HB(private between hyper-v) and one vLAN for Live Migration(private between hyper-v).

What kind of storage you plan to use? IB network on SMB 3.0?
From what you're saying:
3 distinct private VLANs for each of CSV, HB and Live Migration. I can probably combine CSV+HB in one VLAN though, right?
VM Network will be trunked (have accces to all VLANs), instead of access to only one VLAN.

Still doesn't answer how to wire them physically though. My idea is to have some redundancy if my X520 NIC goes, even if it's at a slower speed.

The hosts will connect back to a HA ZFS cluster. I think I'm stuck with iSCSI over IPoIB but I haven't gotten that far yet.
 

dba

Moderator
Feb 20, 2012
1,477
184
63
San Francisco Bay Area, California, USA
Have you though about using a "converged" network architecture instead of lots of separate NIC cards? Conceptually, I really like the "small number of big pipes sliced up into smaller pipes virtually" approach versus "one NIC for each network".

For example:
http://en.community.dell.com/techce...n-12th-generation-dell-poweredge-servers.aspx
NIC Teaming, Hyper-V switch, QoS and actual performance | part 1 ? Theory

Also, while Windows 2012 enables converged networks, I had serious performance problems with IPoIB on Windows 2012. when you benchmark your SAN over IB, let us know how it performs.
 

mrkrad

Well-Known Member
Oct 13, 2012
1,244
52
48
1. How do you honor switch flow control with nic convergence? Do you not honor pause frames (nor send them?) What happens if the switches choke both of your nic's? All vlan's choke and die because one slow host throws a fit?

2. Can you switch on the nic's? You most certainly don't want to have to send packets to a switch to have it loopback into the same port. Some CNA's can do this when the lighting is just right, some cannot. SR-IOV on some nic's bypasses all virtual switching (great for serving storage).

So what's a perfect storm? Bad cable, server dropped a nic, 10% packet loss at gigabit speed with flow control throwing tons of pause at your server? Buffers start choking up. From my experience, folks design for perfect condition, and this leads to a sh**storm when things go wrong. Could be a bad day, say the aircon starts flaking and those super-fast nic's and switches start overheating and self-throttling to prevent a meltdown.


I just scored some ancient broadcom 10gbase-T nic's (160 bucks for 4) , to go with QLE8152's and OCE11102-FM, and some solarflare 6125F and some older Intel AT2 adapters.

If this all falls apart, I'm going to pickup one of those $900 voltaire 4096 and give that 32gbp/s IB a shot. But what I need is the ability to maintain 90% line rate virtualized with 4 to 16 vm's per host. At any time, 1 VM should be able to utilize 90% line rate, whilst the other 15 utilize the other 5-10%.
 

markpower28

Active Member
Apr 9, 2013
413
104
43
For physical connection, PC 8024 is stackable, looks like you have already made them as a single logical unit. That's how you able to team 2 x t520, right?

If 2 port t520 goes(which is very unlikely), I would have another 2 port t520 installed on the system and add to the team. mix between 10 GE and 1 GE may not be a good idea.

if create additional vlan is not an issues i will separate CSV and HB if possible.

I have deployed hyper-v cluster using HP flex10 and UCS using converged virtual nic. It works very well but come with a price tag. Not sure which vendor cisco using but HP use emulex. The only issues is buggy firmware and potential PXE booting hang

From what you're saying:
3 distinct private VLANs for each of CSV, HB and Live Migration. I can probably combine CSV+HB in one VLAN though, right?
VM Network will be trunked (have accces to all VLANs), instead of access to only one VLAN.

Still doesn't answer how to wire them physically though. My idea is to have some redundancy if my X520 NIC goes, even if it's at a slower speed.

The hosts will connect back to a HA ZFS cluster. I think I'm stuck with iSCSI over IPoIB but I haven't gotten that far yet.
 

wuffers

New Member
Dec 24, 2012
19
0
0
Thanks for all the replies guys. I'll also be burning some of our unused hours we have for a MS Partner Technical Consultant (since we're a Gold Partner ISV) who specializes in Hyper-V and can hopefully make some recommendations based on our situation.

Have you though about using a "converged" network architecture instead of lots of separate NIC cards? Conceptually, I really like the "small number of big pipes sliced up into smaller pipes virtually" approach versus "one NIC for each network".
Well, I could just have my team of X520 do all the duties of Management, Cluster Failover HB, CSV and Live Migration but I think that's just asking for trouble if the X520-DA2 fails. I'm not looking for the perfect design where everything is segregated but a balance of performance and redundancy. The NICs are available (1 for IPMI, 2 onboard i350 GbE, 2 onboard X540 10GbE) so why not use them?

Also, while Windows 2012 enables converged networks, I had serious performance problems with IPoIB on Windows 2012. when you benchmark your SAN over IB, let us know how it performs.
I guess I'll find out when I get there. I just need to get through finishing physical cabling so I can get to the software configuration part. What benchmarks should I be using? IOMeter, some file transfers? Is IOZone even available for Windows Server 2012?

1. How do you honor switch flow control with nic convergence? Do you not honor pause frames (nor send them?) What happens if the switches choke both of your nic's? All vlan's choke and die because one slow host throws a fit?

2. Can you switch on the nic's? You most certainly don't want to have to send packets to a switch to have it loopback into the same port. Some CNA's can do this when the lighting is just right, some cannot. SR-IOV on some nic's bypasses all virtual switching (great for serving storage).

So what's a perfect storm? Bad cable, server dropped a nic, 10% packet loss at gigabit speed with flow control throwing tons of pause at your server? Buffers start choking up. From my experience, folks design for perfect condition, and this leads to a sh**storm when things go wrong. Could be a bad day, say the aircon starts flaking and those super-fast nic's and switches start overheating and self-throttling to prevent a meltdown.
The concept is great, but yes, I'd rather factor in some redundancy in the design. Hyper-V's Redirected I/O is there for a reason.

For physical connection, PC 8024 is stackable, looks like you have already made them as a single logical unit. That's how you able to team 2 x t520, right?

If 2 port t520 goes(which is very unlikely), I would have another 2 port t520 installed on the system and add to the team. mix between 10 GE and 1 GE may not be a good idea.
Yes, I've stacked the PC8024Fs together and it is working as one logical unit. My 3750s are also managed as a stack. I won't be mixing media speeds in a team (is that even possible?). As failure rates on NICs are so uncommon, I did not design my hardware choice to have a redundant one. I had to keep my budget to a number and getting 5 extra X520s would not have fit anyways. Since it's designed as a Hyper-V cluster, I can have one host down while I get a replacement if it does fail.

if create additional vlan is not an issues i will separate CSV and HB if possible.

I have deployed hyper-v cluster using HP flex10 and UCS using converged virtual nic. It works very well but come with a price tag. Not sure which vendor cisco using but HP use emulex. The only issues is buggy firmware and potential PXE booting hang
No big issue with creating another VLAN. Just thought that if I use one physical NIC for both of these I wouldn't need to partition or configure separate IPs, etc. just to keep it simpler.

My own thoughts on this has changed a bit, I'm thinking:
- Teamed X520s would be VM Network and Live Migration
- IPMI port will just be for remote KVM (I could probably do some tagged/untagged VLAN to use this as IPMI and Management both)
- Management on 1GbE. I'm told that SCVMM doesn't use that much to manage the hosts.
- Onboard 10GbE for Cluster Failover and CSV

I'll see what the PTC has to recommend and give an update then.