Crazy Storage Spaces Question...

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

b-rex

Member
Aug 14, 2020
59
13
8
So, I've been trying to figure out a problem with my lab design. I currently have a 2-node ESXi vSAN setup with 4x 365GB FIO2, 4x 500GB Cons S.2, and 4x Intel S3610 400GB. Great. But...I'm expanding and based on what I could get my hands on, I will be expanding using Hyper-V and SCVMM. Also great. The problem comes in with what I have to create my S2D cluster. Just a disclaimer...what I'm about to discuss I would never do in a production environment (if it can even be done). I have a lot of experience with S2D and it's been great for most of the people I've worked with. I always set it up with a traditional all-flash or hybrid using physical disks. I've never ventured away from that. For my home lab...I'm looking to make the most out of my setup (and money) while still maintaining performance AND redundancy. I have two physical servers that will act as clustered storage. Each has 3x 1.6TB P3608 (w/ essentially 2x 765GB Intel NVME), 6x Intel 1.6TB S3610, 12x 500 GB Cons S.2). For the most part, that will do fine. I can lose an entire server and still have storage available. However...I've been looking at doing something kinda radical and strange but I think technically it would work and I would gain higher storage efficiency and still maintain resiliency and most of the performance. I want to create 16 virtual machines spread across the two current ESX nodes (2 VM each) and S2D nodes (6 VM each) for a total of 16 VM-based SOFS with 1.0-1.4 TB each. The reason I'm doing this is that I need to be able to lose an entire server for maintenance at a given time...and I was going to use parity+mirroring which gives me more efficiency but requires a minimum of 4 nodes in order to work correctly. The diagram below (thrown together quick so try to ignore some of the naming issues) highlights what I'm after...

1.png

The goal with this is so that I can gain some redundancy. For example...if I lose a host (or two even)...one hypervisor, in theory could handle most of the additional virtual machines (although with much heavier load placed on the underlying disks). The VM disk sizes will be lowered to make sure that in the event of a failure, I have storage available to handle the additional machines. My question is...would this work? I have seen many multi-failure scenarios where a 16 node cluster could tolerate some pretty crazy stuff but those are built to support Microsoft configurations. This would not be for many reasons. I'm looking at it and at first glance it would make sense and that it would work. I know it's crazy...and might just be stupid but I'm curious if anyone else has any thoughts about it.
 

b-rex

Member
Aug 14, 2020
59
13
8
So, I've been trying to figure out a problem with my lab design. I currently have a 2-node ESXi vSAN setup with 4x 365GB FIO2, 4x 500GB Cons S.2, and 4x Intel S3610 400GB. Great. But...I'm expanding and based on what I could get my hands on, I will be expanding using Hyper-V and SCVMM. Also great. The problem comes in with what I have to create my S2D cluster. Just a disclaimer...what I'm about to discuss I would never do in a production environment (if it can even be done). I have a lot of experience with S2D and it's been great for most of the people I've worked with. I always set it up with a traditional all-flash or hybrid using physical disks. I've never ventured away from that. For my home lab...I'm looking to make the most out of my setup (and money) while still maintaining performance AND redundancy. I have two physical servers that will act as clustered storage. Each has 3x 1.6TB P3608 (w/ essentially 2x 765GB Intel NVME), 6x Intel 1.6TB S3610, 12x 500 GB Cons S.2). For the most part, that will do fine. I can lose an entire server and still have storage available. However...I've been looking at doing something kinda radical and strange but I think technically it would work and I would gain higher storage efficiency and still maintain resiliency and most of the performance. I want to create 16 virtual machines spread across the two current ESX nodes (2 VM each) and S2D nodes (6 VM each) for a total of 16 VM-based SOFS with 1.0-1.4 TB each. The reason I'm doing this is that I need to be able to lose an entire server for maintenance at a given time...and I was going to use parity+mirroring which gives me more efficiency but requires a minimum of 4 nodes in order to work correctly. The diagram below (thrown together quick so try to ignore some of the naming issues) highlights what I'm after...

View attachment 15476

The goal with this is so that I can gain some redundancy. For example...if I lose a host (or two even)...one hypervisor, in theory could handle most of the additional virtual machines (although with much heavier load placed on the underlying disks). The VM disk sizes will be lowered to make sure that in the event of a failure, I have storage available to handle the additional machines. My question is...would this work? I have seen many multi-failure scenarios where a 16 node cluster could tolerate some pretty crazy stuff but those are built to support Microsoft configurations. This would not be for many reasons. I'm looking at it and at first glance it would make sense and that it would work. I know it's crazy...and might just be stupid but I'm curious if anyone else has any thoughts about it.
And then...as quickly as I came up with this, I realized it would make more sense to show what a failure could look like...
2.png
 

b-rex

Member
Aug 14, 2020
59
13
8
That's partially incorrect. The two node configuration does support nested resiliency with nested mirror-accelerated parity and nested two-way mirror. Even so, my solution above would use virtual storage hosted on the physical devices to provide storage for the storage pools. This would allow me up to 16 nodes (the max available from storage spaces). Six guests on each Hyper-V node, and two on each ESX node. Each would have pooled/tiered storage to act as volume/datastore for virtual disks provided to the guests. The guests would be my SOFS servers, not the two physical servers. I know there's something I'm missing in this design and I'm having a hard time pinpointing it at the moment which is why I posted this but it's not that I don't understand the restrictions of having a two-node cluster (which is why I'm proposing what I'm proposing).
 

dswartz

Active Member
Jul 14, 2011
610
79
28
It's not clear what his statement meant: could mean '2 node only supports mirror' or 'mirror can only be done on 2 node'.
 

cesmith9999

Well-Known Member
Mar 26, 2013
1,417
468
83
Once you lose a physical node , the failover of the parity volume(s) on the VM's will cause your virtual SOFS to fail because of the time it takes to recover the disks from the VM's booting up on the remaining node. You will most likely have to reboot your entire virtual SOFS cluster on the 1 remaining node to bring the Virtual SOFS back online. then wait for things to shake out once it is back online.

like you said. "I would not do this in Production", so why do it here? SOFS and S2D are very fragile. I would expect this design to be a nightmare to keep running.

Chris
 

b-rex

Member
Aug 14, 2020
59
13
8
I thought about the time that it would take for VMs to migrate and while substantial, it would only matter during an actual hardware or power failure (I'll get back to this in a second). The VMs would never be shutdown, they would be live migrated. The live migration would take a lot of time, I grant that.

The reality is that I only have two boxes for storage. To get mixed 3-way mirror and parity (which boasts at least 50% efficiency), the fault tolerance I'm looking for, performance, etc. is with 4+ nodes. I have two and they both cost a fortune. It's all I can afford so I'm trying to get the most I can out of it. I would never do less than 4 nodes in production...but I'm usually dealing with 10s-100s of thousands of dollars when designing solutions that are enterprise-grade. I built this with a few thousand. Two-way mirroring is resilient enough, three-way mirroring is better, three-way mirror-accelerated with parity is even better. I was going for the the latter. Live migrations aside (which I think could be mitigated with decent network performance), I think the fatal flaw that I was missing was how the individual storage is configured at the physical layer. The read/write cache the NVMe provides for S2D would not be available for the VMs. It wouldn't be available on the underlying storage pools either. I did some checking and confirmed that the caching is where this all falls apart even if I manually classify storage in each the physical and virtual storage pools. Storage Spaces on its own on a single node (the physical nodes in this case) does not provide the same caching abilities as Storage Spaces Direct. This is basic and something I'm somewhat disappointed that I forgot about it. I think I'm just trying too hard to get as much as possible from what I have. I went for performance but capacity matters too and I'm running a little low on both. If not for the caching issues, I'd do this setup at home to make the most of the storage I have available to me.

Also, SOFS and S2D are not fragile. I'd take S2D over several other SDS solutions any day due to the wealth of features, its purpose-built integration with Hyper-V, performance, and resiliency. Aside from commercially-built SAN solutions, S2D is one of the best out there especially if you're doing Hyper-V virtualization with ReFS.
 

b-rex

Member
Aug 14, 2020
59
13
8
It's not clear what his statement meant: could mean '2 node only supports mirror' or 'mirror can only be done on 2 node'.
Both are wrong. Nested resiliency is a mix of mirror and parity and is supported on 2-node clusters...so "2 node only supports mirror" is not correct. And mirror can be done on any configuration, S2 or S2D, 1+ nodes all the way to 16.