Hyper-V Web Hosting Cluster Exploration

Patrick

Administrator
Staff member
Dec 21, 2010
11,906
4,868
113
The purpose of this thread is to discuss the idea of moving the STH colocation to a Hyper-V cluster for web hosting.

Here is the current colocation installation:


Current configuration
Nodes 1-4 each have 48GB of DDR3 and 160GB Intel SSDs
Nodes 1 and 3 each have a 3TB WD Red and a 120GB scratch SSD
Nodes 1, 2 and 4 have dual L5520 and node 3 has dual L5638

Currently the main STH VM is running on node 3 and the main forums VM is running on node 1. All nodes are currently running Proxmox VE 3.1

My current thoughts on how this would work:
Nodes 2 & 4 would be simple Hyper-V Server 2012 R2 "compute" nodes
Nodes 1 & 3 would be Windows Server 2012 R2 (Essentials/ DC/ Standard?) with Hyper-V and host, active directory, SMB 3.0 shares, also with the 3TB drives for backups

Thoughts on the migration:
Move raw disk images from Proxmox Nodes 1 and 3 to Node 4
Install Windows Server 2012 R2 with Hyper-V in Nodes 1 and 3
Install Hyper-V Server 2012 R2 on Node 2
Migrate images from Proxmox Node 4 to Nodes 1 and 3
Change Proxmox Node 4 to Hyper-V Server 2012 R2
 

nitrobass24

Moderator
Dec 26, 2010
1,083
127
63
TX
As I mentioned in the other thread...although maybe not very clearly. I think it would be best to leave all 4 nodes as pure Hyper-v Hosts and put the AD component into a VM, so it is highly available. Similar concept to people putting their VMware vCenter Server into the VM farm.

Do you have existing licensing for Windows Server 2012 R2? I would use that to dictate which version you use. Technically Standard would do just about anything you want. The nice thing about Datacenter version is it includes unlimited guest vm licenses, when the guest is used on the host using the same license.
 

Patrick

Administrator
Staff member
Dec 21, 2010
11,906
4,868
113
Understood. So the Hyper-V server boots and no issue with the Ad server coming up after? Might make sense then.

The other item at that point is figuring out storage options, and if I can do the above. Maybe I can try with ServeThe.biz
 

dba

Moderator
Feb 20, 2012
1,478
181
63
San Francisco Bay Area, California, USA
As I mentioned in the other thread...although maybe not very clearly. I think it would be best to leave all 4 nodes as pure Hyper-v Hosts and put the AD component into a VM, so it is highly available. Similar concept to people putting their VMware vCenter Server into the VM farm.

Do you have existing licensing for Windows Server 2012 R2? I would use that to dictate which version you use. Technically Standard would do just about anything you want. The nice thing about Datacenter version is it includes unlimited guest vm licenses, when the guest is used on the host using the same license.
One other option, the one that I am using: Use Windows 2012 with the Hyper-V role installed instead of Hyper-V 2012 and add the AD role to each Hyper-V host. That way you get multiple AD servers for redundancy and, because the role runs on the host and not a VM, you don't have to worry about startup order. The downside is the need for 2012 server licenses, but if you happen to have some already then it might not be an issue.

Are you thinking of using remote storage via SMB3 and clustering?
 

Patrick

Administrator
Staff member
Dec 21, 2010
11,906
4,868
113
Licenses are on the Christmas list hence why Essentials would be preferred.

Yes dba thinking of doing SMB 3 and clustering.
 

sboesch

Active Member
Aug 3, 2012
394
31
28
Columbus, OH
My Hyper-V installations boot fine with out the Domain Controllers running, it will just used cached mode until the DC comes online. Standard Edition should be fine, I don't see you running dozens of VMs. Your plan for shuffling-converting the virtual disks and hosts looks sound. Remote storage with clustering as DBA mentioned should be solid, are you planning on changing the hardware at all?
 

dba

Moderator
Feb 20, 2012
1,478
181
63
San Francisco Bay Area, California, USA
Licenses are on the Christmas list hence why Essentials would be preferred.

Yes dba thinking of doing SMB 3 and clustering.
Your need for reliability is much higher than I have for my development boxes, of course. For real clustering, I really do like the SMB3 option for its speed without complexity. But for those who are using VMs and can stand a few minutes of potential downtime, I will contribute this: Replication!

I built up and used a Hyper-V cluster based on a shared SAS 6G SAN with redundant SAN nodes, each of which had redundant controllers - lots of gear costing more than the servers they served. It worked very well, but it was complex, a bit expensive, and the speed was only middling by my standards, limited by the capabilities of the RAID controllers.
I replaced it with a setup that is far simpler and much faster: VM disks on directly attached SSD drives, with disaster recovery provided by replicating the VMs every five minutes to a standby Hyper-V host. Talk about cheap and easy compared to clustering! On a day-to-day basis, I get the excellent throughput and IOPS of local SSD drives, and if something fails I can spin up a replica in about 15 seconds. I wouldn't use the setup for a tran$actional system, of course, but most workloads can deal with a brief outage, and for those a cluster is - in my opinion - overkill.
 
Last edited:

Mike

Member
May 29, 2012
482
16
18
EU
Not to be the asshole here all the time but... You could forget about fail over, hot standby and what not by just looking a bit higher up. LBs, webservers and replicating database. Keep the images consistent with some automated configuration. I think Windows is over complicating stuff right now... ?
 

Patrick

Administrator
Staff member
Dec 21, 2010
11,906
4,868
113
Thanks for all the excellent input.

sboesch - was not planning to change the hardware. If adding a few SSDs into the mix would make life significantly easier, I might take a trip to Fiberhub to add them.

dba - Great points. The clustering option does not look that bad at all. Does replication look only for changes like rsync or does it copy the whole volume each time? The volumes are only ~40GB with relatively minor changes each day in terms of size (not growing by even 100MB/ day). Replicating the differences would work well since that is not write heavy. Replicating the entire volume would incur heavy daily writes.

Mike - HAproxy on the front end pfsense boxes is trivial to add. They are running 12GB RAM and quad core L5520's so more than enough power for pfsense + HAproxy. Adding load balancers is not a huge issue and I do have the ability to have extra VMs setup for replication. What did you have in mind for an architecture? Proxmox VE has me just in a load of management headache and I am basically at the point where I want to move to something that has easier/ better management. The option of doing the replication in the VMs is not a bad one at all. I am looking to a year or so down the road when I will need to replace the C6100's and do want something that is relatively painless to migrate. Proxmox had that promise but am becoming less than a believer in it.
 

dba

Moderator
Feb 20, 2012
1,478
181
63
San Francisco Bay Area, California, USA
Thanks for all the excellent input.
...
dba - Great points. The clustering option does not look that bad at all. Does replication look only for changes like rsync or does it copy the whole volume each time? The volumes are only ~40GB with relatively minor changes each day in terms of size (not growing by even 100MB/ day). Replicating the differences would work well since that is not write heavy. Replicating the entire volume would incur heavy daily writes...
Hyper-V replication is asynchronous and block-level change based - very low overhead. The VMs track changes locally and then push them on a schedule - I replicate every five minutes, with an app-consistent replica every hour just for fun. By having Hyper-V automatically retain the last n replication snapshots, I also get some protection from human error... not that I've ever had to use it of course. At any given time I have my daily backups, the last 24 hours worth of hourly replication points, and the last few five-minute replication points. Why so much attention to retaining replication snapshots in case of human error? I plead the 5th.
 
Last edited:

Patrick

Administrator
Staff member
Dec 21, 2010
11,906
4,868
113
That is exactly what I want and why I am looking at Hyper-V. Snapshots are easy to make and handy. Would save me lots of time if ever I had a plug-in update blow up the WP site (not that I have done that either.)
 

mrkrad

Well-Known Member
Oct 13, 2012
1,244
52
48
I'm getting my popcorn..

Just curious how hyper-v has come up.. How bout management of hyper-v nodes not in a DC? as easy as vcenter (root/password)?
 

Patrick

Administrator
Staff member
Dec 21, 2010
11,906
4,868
113
Probably good to get the popcorn.

The genesis of Hyper-V is basically the idea that I want to move to either Hyper-V or ESXi. I will certainly give you that vCenter is a strong contender still.

Also, the forums have moved from node 1 to node 4 so there is now a single and a three disk node available.
 

Fzdog2

Member
Sep 21, 2012
92
14
8
I must say I am liking the all-in-one contained vCenter cluster serving up VSAN datastore to run everything on. I keep an iSCSI datastore for backups as well.
 

DolphinsDan

Member
Sep 17, 2013
90
6
8
Patrick here is why I like your new thought on this: Windows + Linux = Win

I've been looking into OpenStack but Windows is still 100x easier to install. 99.9% of Windows is GUI administration. Proxmox from what I have seen has a good enough UI but that UI does not do simple things well so you end up having to look up obscure commands. You have sets of GlusterFS or DRBD commands to setup clustered storage. You then need to get Proxmox to use. Mounting requires CLI input. Just a big PITA.

I like VMware better. For what you are doing, I think this concept is sound.

Also, last I tried Hyper-V performance, it is actually a lot better than my VMware UG friends suggest. Microsoft had a clunky product in 2008 but 5-6 years later it is respectable.
 

Patrick

Administrator
Staff member
Dec 21, 2010
11,906
4,868
113
Feeling comfortable to the point that I am installing Hyper-V Server 2012 R2 on one of the extra boxes (single SSD only)
 

Patrick

Administrator
Staff member
Dec 21, 2010
11,906
4,868
113
I've got some virtual ISCSI if you get bored ;)
Hmmm... what is this?!?

I do think Hyper-V is going to be the project in 2014 though. I'm also "considering" moving to 1x C6100 with 4x compute nodes, then using 2x C2750/ C2758 boxes. Would likely use significantly less power.
 

Xeppo

New Member
Nov 13, 2013
28
11
3
Hmmm... what is this?!?

I do think Hyper-V is going to be the project in 2014 though. I'm also "considering" moving to 1x C6100 with 4x compute nodes, then using 2x C2750/ C2758 boxes. Would likely use significantly less power.
While Hyper-V is a great platform for virtualization, I feel like the majority of its value is lost if you do not already own (or need) licenses for the datacenter. While I'm not 100% familiar with the ins-and-outs of your setup here, I feel as though you have essentially zero need for Microsoft outside of Hyper-V management here on the site. By the time you purchase SCVMM (Which is basically required if you want to use any of the more useful features of Hyper-V), you're looking at a cost on par with or greater than VMWare.

My question is this: Why leave Linux? KVM is a very evolved, documented, and supported hypervisor. It supports (almost) all of the major enterprise features, and does so at a price point that won't break the bank. I recommend that you move off of Proxmox, and move into something less GUI-dependent (Debian, CentOS, etc.)