GlusterFS shared storage on Hyper-V

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

Jeggs101

Well-Known Member
Dec 29, 2010
1,529
241
63
This weekend I was sketching something out. I want to have shared storage for Hyper-V but I'm nervous about license costs. If MS makes desktop Windows free, I'm worried what that'll do to the next version of Server. Also, I have a few very small clusters of <10 machines so setting up and maintaining Windows Server Clustering and AD is a bigger burden.

Here's what I'm thinking:
- Mirrored small boot OS SSDs
- Pass-through 4-6 SSDs or HDDs to a Linux VM
- Run GlusterFS on the Linux VM on each server
- Set replication to 2 to limit how many copies I have
- Setup some sort of shared IP so that storage is available from a common IP address
- Have the Hyper-V hosts boot VMs off of the GlusterFS storage

Has anyone tried something similar? Is this way too hard for what I'm trying to do?
 

Mike

Member
May 29, 2012
482
16
18
EU
Sounds interesting, although I'm not sure at what point you would rather 'centralize' storage instead of running it locally on every node. I use drbd between two nodes to be enabled for HA storage and you could use a shared FS on that too. If 2 or 3 of your nodes out of ten provide storage (vms) to the rest that may be performant enough, and you could migrate those instances if you feel like it.
 

cesmith9999

Well-Known Member
Mar 26, 2013
1,417
468
83
I am kinda confused here. what is your end goal.

if you are willing to deal with a minimal GUI. just use the free Hyper-V sku and use that to do SOFS and spaces
 

Patrick

Administrator
Staff member
Dec 21, 2010
12,511
5,792
113
I am kinda confused here. what is your end goal.

if you are willing to deal with a minimal GUI. just use the free Hyper-V sku and use that to do SOFS and spaces
I think @cesmith9999 is right, Hyper-V Server 2012 R2 (free) has storage roles that can be added. Now, @Jeggs101 if I am reading your comments correctly, you may be concerned with the availability of those features in the next version hence why you want to go Linux.

@PigLover great link. I may give that a try myself.
 

PigLover

Moderator
Jan 26, 2011
3,184
1,545
113
Its also possible that you'll run into this:

Hyper V over SMB does not work on non-Microsoft Samba servers

As recently as September people were still reporting problems serving Hyper-V .vhdx over Linux/Samba. This is the relevant comment:

It's not MSFT problem some people understand SMB spec too "freely"... There are some known working SMB3 non-MSFT implementations (EMC, NetApp and maybe Nutanix) and at least few of them are indeed BASED on an open source SAMBA. But these companies did a huge amount of work to make their fork-outs from SAMBA actually work. Unfortunately nobody bothered to back-contribute their fixes to original code base. Again, it's not MSFT problem.
 
  • Like
Reactions: Patrick

Patrick

Administrator
Staff member
Dec 21, 2010
12,511
5,792
113
That is really interesting. I really like how easy replication, migration, snapshots and dynamic memory are in Hyper-V. Stability for me has been great as has been the ability to literally download and run images on my desktop/ laptop without other software.

At the same time I get nervous about thinks like you mentioned @PigLover - those are the scary problems to me.
 

PigLover

Moderator
Jan 26, 2011
3,184
1,545
113
I am kinda confused here. what is your end goal.

if you are willing to deal with a minimal GUI. just use the free Hyper-V sku and use that to do SOFS and spaces
There might be two problems with this:

First, SOFS prerequisite is that the nodes presenting the SOFS must be part of a Failover Cluster. If I remember correctly, Failover Cluster cannot be activated on a Hyper-V Server node. It requires "Standard" or "Datacenter" install of Windows Server to activate the Failover Cluster role.

Second, SOFS isn't really "Share Nothing" the way GlusterFS is. It depends on each node in the SOFS being able to access ALL of the disks directly - either via a shared SAN or multi-homed JBOD.

MSFT has said they will be including "Shared Nothing" storage in Server 10. The details aren't released (or even fully leaked) yet. But the OPs concerns are valid: there is no reason to believe this will be availble in a way that it can be used with reasonable license fees/terms.
 

markarr

Active Member
Oct 31, 2013
421
122
43
Failover cluster is a feature not a role, it can be installed on the free version of the Hyper-V. I have a 3 node cluster using the Hyper-V cluster 2012 R2.
 

TuxDude

Well-Known Member
Sep 17, 2011
616
338
63
Its also possible that you'll run into this:

Hyper V over SMB does not work on non-Microsoft Samba servers

As recently as September people were still reporting problems serving Hyper-V .vhdx over Linux/Samba. This is the relevant comment:

It's not MSFT problem some people understand SMB spec too "freely"... There are some known working SMB3 non-MSFT implementations (EMC, NetApp and maybe Nutanix) and at least few of them are indeed BASED on an open source SAMBA. But these companies did a huge amount of work to make their fork-outs from SAMBA actually work. Unfortunately nobody bothered to back-contribute their fixes to original code base. Again, it's not MSFT problem.
People insulting Samba doesn't help anyone, its just spreading more pro-MS FUD. If that provided quote was true, the companies who released products based on a modified Samba without making their contributions available in source-form would be sued for violating the GPL.

It is far more likely the Samba's implementation of SMB3 is perfectly fine, though it is also incomplete. There are many features in SMB3 that MS has labelled optional and that Samba has not (yet) implemented. It could be any of those missing features that is causing the Hyper-V issues.
 

PigLover

Moderator
Jan 26, 2011
3,184
1,545
113
I don't think that anyone in that thread was "insulting" Samba. It was really pretty fact-based:

- Hyper-V has certain requirements of an SMB3 filesystem in order to support VMs
- Those include a number of features not currently implemented in the OpenSource Samba (at least as of Samba 4.12)
- Some commercial systems based on Samba have implemented the required features and do work
- Those features have not been backported into OpenSource Samba.

Pretty black and white, really. The closest thing to "insulting" Samba in that thread was the statement that "it is not MSFT fault" that the Samba project has not implemented all of the required option. While not the most diplomatic wording - it is completely true.
 

TuxDude

Well-Known Member
Sep 17, 2011
616
338
63
Actually it was this statement that I thought was insulting towards Samba: "It's not MSFT problem some people understand SMB spec too "freely"" - I think they are doing a very good job with their project, doing their best to be fully compatible with very few resources available, while that line seems to imply they are taking freedoms with the protocol that should not be allowed.

And I do believe MS has at least a little bit of blame to carry here, as their documentation states only that "You can also use non-Microsoft file servers that implement the SMB 3.0 protocol." If there is a requirement for an optional feature of SMB3 that Samba does not implement, then that is poor documentation on MS's part.
 

PigLover

Moderator
Jan 26, 2011
3,184
1,545
113
Hyper-v does work with "non-Microsoft filerservers that implement the SMB 3.0 protocol". It works with EMC. It works with NetApp. There are a few others. So Microsoft's claim that offends you is actually true, fact based and defensible. At no point does their documentation state that it works with ALL non-Microsoft implementations. At no point does it claim to work with any specific implementation (e.g., Opensource Samba). It is hardly rational to "blame" Microsoft for saying true and correct things in their documentation.

Its also true that the Samba project team knows exactly which feature of SMB 3.0 are missing that prevents Hyper-V from working with their code, primarily the lack of "resilient file handles". This is clearly noted in several presentations from the Samba project specifically written about what it will take to eventually support Hyper-V. It is not implemented because it is hard. While perhaps impolitic to say it as directly as they did, the fact that the Samba team has not chosen to implement this feature - optional or not - is in fact not Microsoft's fault.

I also have a lot of respect and admiration for the work from the Samba project. I use it every day, both for AD and SMB. It is getting better with every point release .

But I also understand its limitations. There are functions that are not complete yet. That is a fact and that fact prevents using Samba in many applications where it would be highly desirable (like this one for the OP). I think it is highly disingenuous for anyone to blame Microsoft for this or to claim that it is somehow a problem that Microsoft's documentation is "poor" when the Samba team actually knows what is required and has made a (completely rational) choice to work on other priorities like finishing AD support.
 

PigLover

Moderator
Jan 26, 2011
3,184
1,545
113
Dialing down the rhetoric a bit and addressing @Jeggs101 directly:

- The configuration you suggest is rational
- Unfortunately, you currently can't support homing Hyper-V to an OpenSource Samba filesystem
- I don't think FC/SOFS does what you described
- There is hope that Windows "Share Nothing" fileserver in Server-10 will do what you want (when released)

You could probably do a hybrid using GlusterFS/Samba like you describe for user-data filesystems and Hyper-V replication to protect the VMs.
 

Jeggs101

Well-Known Member
Dec 29, 2010
1,529
241
63
Thanks for all of the advice. I will give the guide @PigLover linked a try in a week when the new equipment gets installed.
 

seang86s

Member
Feb 19, 2013
164
16
18
Failover cluster is a feature not a role, it can be installed on the free version of the Hyper-V. I have a 3 node cluster using the Hyper-V cluster 2012 R2.
Your shared storage for the Hyper-V cluster is a SOFS running on these nodes as well? What kind of storage hardware are you running?
 

markarr

Active Member
Oct 31, 2013
421
122
43
Not currently using host based SOFS. Currently using combination of local raid and csv in a MS cluster with two hosts, and then replicating to a third host with fat storage for instant fail-over. My csv is served by a zfs nas box.
 

gigatexal

I'm here to learn
Nov 25, 2012
2,913
607
113
Portland, Oregon
alexandarnarayan.com
A free edition of MS server 10 with a gui would be awesome. A lot more developers might start working on Windows, it could create much more in the way of market share for MS and take it from the *Nix side. I'm loving Server 2012 R2 and hyper-V right now. But my 2012 license (trial) will expire soon.