Storage Spaces (Server2016) with lots of disks

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

Cutha

Member
Sep 24, 2016
75
7
8
50
Canada
Hi,

I have about 40-50 3TB drives and my plan was to have them connected in a giant pool with much of it being used as 20-40TB virtual disks created as dual parity. I was planning on leaving about 10TB RAW free on the storage space so that on a disk failure storage spaces would have enough room to auto repair.

I can't seem to find a recommendation for the max number of drives to have in a pool to maintain a healthy safety margin. I know, or at least think I know, that this does not really have an analogous comparison with a HW RAID controller. Does anyone have a recommendation for this?

Also, I have 5x500GB SATA SSD's that I want to use with this pool. I was going to create tiered spaces with a small 3 way mirrored for each virtual disk but maybe it's better if I just set them as Journal drives? I think as Journal drives they will server as a write buffer to all the virtual disks in the storage space.(?)

This entire pool is primarily for media storage. At one point I was thinking I was just going to make a single massive virtual disk but I figured it would be a pain to work with. But maybe not...

Thanks for taking the time to read this. I hope it's reasonably coherent; you would think I would learn about creating posts at 4:30AM.
 

i386

Well-Known Member
Mar 18, 2016
4,220
1,540
113
34
Germany
A storage space pool can have up to 240 devices (4 jbods with 60 disks each), Server 2012 up to 160 disks.
TechNet Wiki

I think you want to know the number of disks you can put in a storage space (=logical volume). I can't find any offical number for that, looking at hardware and software raid implementations there is often a limit of 32 hdds/ssds/nvme devices for one logical volume. So I would say use 32 devices for the storage space and keep 2 hot spares per storage space.
 

Cutha

Member
Sep 24, 2016
75
7
8
50
Canada
I think you want to know the number of disks you can put in a storage space (=logical volume). I can't find any offical number for that, looking at hardware and software raid implementations there is often a limit of 32 hdds/ssds/nvme devices for one logical volume. So I would say use 32 devices for the storage space and keep 2 hot spares per storage space.
As far as I know, if you had 100 HDDs in a Storage Pool, when you create a virtual disk there is no way to control which disks it spans. With dual parity I think the max number of columns is 17. I am not sure if then it only uses up to 17 disks.

What I am trying to ask is: If I had ~40*3TB HDD's and I am ok with the performance of dual parity, what is the most efficient way to configure Storage Spaces while maintaining a reasonable level of safety? With SS Direct MS recommends leaving unused space per server = 1*capacity HDD for quick auto rebuilds. I was thinking that on a large (single server) pool I could afford to leave 3*HDD unused.
 

cesmith9999

Well-Known Member
Mar 26, 2013
1,417
468
83
Storage Spaces spans all disks in the pool. When you create the virtual disks you need to specify the columncount using PowerShell. The default column count is 8 when you have more than 8 disks in the pool.

Also remember that using dual parity means that you lose 1 disk to global parity. I recommend that you leave 2 or 3 disks worth of space availabl in the pool for sparing. I also recommend that you leave a couple of drives out of the pool to allow for pool disk replacements.

Chris
 
  • Like
Reactions: T_Minus

Cutha

Member
Sep 24, 2016
75
7
8
50
Canada
Thanks for the reply Chris.

So, is a pool with 40 disks unreasonable?

40*3TB - 9TB(for pool replacements) - 9TB(dual parity) = 102TB?

The deal with URE, growing HDD sizes, hardware RAID6 and the likely hood of another drive dropping out during a rebuild has caused me to be a little paranoid. In an effort to hedge my bets the best I can, I settled on SS's w/DP and ReFS.
 

cesmith9999

Well-Known Member
Mar 26, 2013
1,417
468
83
I have servers in my care using Storage Spaces with over 100 disks. so 40 is not unreasonable pool size.

Chris
 

Cutha

Member
Sep 24, 2016
75
7
8
50
Canada
Thanks.

Also, I have 5x500GB SATA SSD's that I want to use with this pool. I was going to create tiered spaces with a small 3 way mirrored for each virtual disk but maybe it's better if I just set them as Journal drives? I think as Journal drives they will server as a write buffer to all the virtual disks in the storage space.(?)
 

Evan

Well-Known Member
Jan 6, 2016
3,346
598
113
If those are not enterprise SSD (sata) and actually consumer drives then the results will not be good, but if it’s really low write environment you may get away with it being useful.
 

Cutha

Member
Sep 24, 2016
75
7
8
50
Canada
I thought I would give MRV (Multi-Resilient Volume) a shot but I get an error when I use a higher column count for the mirror tier. I have 33 x 3TB HDD's and no SSD's in the pool. If I change NumberOfColumns on the mirror tier to anything > 4 I get an error: The storage pool does not have sufficient eligible resources for the creation of the specified virtual disk.

$hddTier = New-StorageTier -StoragePoolFriendlyName PS3_Pool2 -MediaType HDD -FriendlyName "ST_HDD" -NumberOfColumns 17 -ResiliencySettingName Parity -PhysicalDiskRedundancy 2

$hddTierMirror = New-StorageTier -StoragePoolFriendlyName PS3_Pool2 -MediaType HDD -FriendlyName "ST_HDDMirror" -ResiliencySettingName Mirror -NumberOfDataCopies 3 -NumberOfColumns 4

New-Volume -StoragePoolFriendlyName PS3_Pool2 -FriendlyName "PS3_MRV_Media" -FileSystem ReFS -StorageTiers $hddTierMirror,$hddTier -StorageTierSizes 2000GB,30TB

DriveLetter FileSystemLabel FileSystem DriveType HealthStatus OperationalStatus SizeRemaining Size
----------- --------------- ---------- --------- ------------ ----------------- ------------- ----
PS3_MRV_Media ReFS Fixed Healthy OK 31.75 TB 31.97 TB


I thought with 3 way mirror I would be able to have 12 columns.
 

Cutha

Member
Sep 24, 2016
75
7
8
50
Canada
Hi,

So a few years later...

Storage Spaces has been great if a bit painful at times from stupidity on my part.

I have about a zillion 3TB HGST drives over a few different JBOD's and too many servers and I wanted to consolidate and also drop my power bill so I bought a 45x10TB JBOD. I have had zero data loss with using SS and I have only had to replace a couple HDD's and that wasn't so bad especially once MS intruduced the optimize command.

I am planning on using dual parity and leaving 3 disks worth of empty space so that the newer feature of Storage Spaces automagically rebuilding into the spare space is seamless and quicker.

Automatically rebuild storage spaces from storage pool free space
Storage Spaces now includes the ability to automatically rebuild storage spaces from storage pool free space instead of using hot spares.

What value does this change add?

Rebuild times are accelerated because multiple disks in the pool can accept the data that was stored on the failed disk instead of waiting for a single hot spare to write all of the data. Additionally, hot spare drives are no longer needed, and storage pool free space can provide additional capacity and performance to the pool.

What works differently?

When a physical disk fails, instead of writing a copy of the data that was on the failed disk to a single hot spare, the data is copied to multiple physical disks in the pool so that the previous level of resiliency is achieved. Administrators no longer need to allocate physical disks as hot spares in the storage pool.

I'm also going to use a few Intel P3605's as an SSD tier so the write speed doesn't drive me nuts.

I would love some feedback on my plan as long as it's not "Use Linux" :)
 

Lost-Benji

Member
Jan 21, 2013
424
23
18
The arse end of the planet
Only just seen this.
Was going to say, too late, already done and doing this.

- Each array should be on same physical chassis, DO NOT SPAN OVER TO OTHER CHASSIS! You can keep some global Hot-Spares in another chassis, just mark them so easy to ID when you need to swap.
- Double-Parity for anything more than 6-8 drives.
- Use good HBA's and cables.
- Be happy with the shitty write speeds but as mine is a Media Server, I survive.
- RAM, boat loads of it if you want to dump movies to SS parity arrays. When you send large files to the array, it will load into the RAM like a Cache and keep writing after it hasclosed the transfer window.
- Windows is pure shit at detecting bad drives, it will keep telling you that a drive is Healthy despite it throwing disk errors all through the event logs. Keep an eye on the logs from time-to-time.
- Learn the commands for Identifying correct drives and their serial numbers, Windows re-orders drives and their locations all the time making hard to trace bad sucker.
 

Cutha

Member
Sep 24, 2016
75
7
8
50
Canada
- Each array should be on same physical chassis, DO NOT SPAN OVER TO OTHER CHASSIS! You can keep some global Hot-Spares in another chassis, just mark them so easy to ID when you need to swap.
Do you use Hot-Spares? I stopped using hot spares once the feature to "Automatically rebuild storage spaces from storage pool free space" was available. Now I make sure my virtual disk leaves at least a few drives worth of unused space. However, I don't think I have seen it automatically rebuild on it's own but I may not have actually had a drive fail since using this.

- Be happy with the shitty write speeds but as mine is a Media Server, I survive.
Do you use a tiering or cache drive? I was thinking of using a few 1.6TB P3605's as the SSD tier. I think the WBC is still limited to 100GB's so I think it would be a waste for me to use them as a WBC.

- Windows is pure shit at detecting bad drives, it will keep telling you that a drive is Healthy despite it throwing disk errors all through the event logs. Keep an eye on the logs from time-to-time.
That hasn't been my experience so far, the few that failed showed up cleanly in the Server Manager->File and Storage Service window. I couldn't get the flash disk LED function to work so to find the offline drive I just did a file copy and then looked for the HDD that wasn't blinking. Then I made a table with HDD serial # and JBOD slot # so I didn't have to mess about next time.

I also use Hard Disk Sentinel to help me keep an eye on all the HDD's health.
 

Lost-Benji

Member
Jan 21, 2013
424
23
18
The arse end of the planet
Do you use Hot-Spares? I stopped using hot spares once the feature to "Automatically rebuild storage spaces from storage pool free space" was available. Now I make sure my virtual disk leaves at least a few drives worth of unused space. However, I don't think I have seen it automatically rebuild on it's own but I may not have actually had a drive fail since using this.
I use warm drives. A coupld that stay in the main server and I add when I find a failing drive. Then move to chassis after repaired.

Do you use a tiering or cache drive? I was thinking of using a few 1.6TB P3605's as the SSD tier. I think the WBC is still limited to 100GB's so I think it would be a waste for me to use them as a WBC.
Nope, I have 3x 16-drive DAS chassis, all full of 3TB drives with movies and TV, no need to waste SSD's or the time to get tiering working. If I need that sort of I/O, I used HW RAID cards.

That hasn't been my experience so far, the few that failed showed up cleanly in the Server Manager->File and Storage Service window. I couldn't get the flash disk LED function to work so to find the offline drive I just did a file copy and then looked for the HDD that wasn't blinking. Then I made a table with HDD serial # and JBOD slot # so I didn't have to mess about next time.
Windows sucks, trust me. By the time SS tells you are drive is bad, damage is done. Lost multiple arrays from it. Watch your Event Logs closely or if media playback is slugish in my case, I know I have a spinner on way out. As for drive bay lights, comes down to controller (HBA) and expanders passing commands correctly.

I also use Hard Disk Sentinel to help me keep an eye on all the HDD's health.
Had it,gave up as it is useless at the numbers of drives I run and it doesn't talk to all drives either.
 

Cutha

Member
Sep 24, 2016
75
7
8
50
Canada
Nope, I have 3x 16-drive DAS chassis, all full of 3TB drives with movies and TV, no need to waste SSD's or the time to get tiering working. If I need that sort of I/O, I used HW RAID cards.
At different times I tried using WBC and tiering with varying results. With both the speed increase is noticeable but when I was testing it I didn't settle on which was best for my purposes and I thought with the updates to ReFS and server 2019 there would likely be new best practices.

I also have 4 supermicro servers with RDMA (40GB IB) that I have been playing with using Storage Spaces Direct. It took me a couple years but I now have some Intel NVMe drives to go along with my SSD's & HDD's and new to me ConnectX-3 NIC's. I also bought 10GB NIC's for the workstations and servers and a couple Brocade ICX 6610's to hopefully get RDMA working all around. I'm sure it will be another learning experience.

Had it,gave up as it is useless at the numbers of drives I run and it doesn't talk to all drives either.
I have a couple of the 45 bay supermicro JBOD's which I love and I have a few 4U 24 bay supermicro servers as well. I just did a check and between 2 of my storage server's I have 106x 3TB (mostly HGST) HDD's. I have yet to lose any data. I did have SS take the array offline when I had a HDD fail but it was no big deal to remove the drive from the array, replace it and then run an optimize to sort things out. With the disks I have Hard Disk Sentinel has worked perfectly and gave me plenty of heads up when I had a WD disk that started having bad sectors.

I also have a 4U 60 bay HGST which I hate for a few reasons, the biggest reason being the HDD temperatures towards the rear of the case are much too hot for my taste. It doesn't help that due to my own stupidity I bent the rails and the server had an unfortunate run in with my basement floor.


I haven't checked this out yet but I will be with my new 45x10TB build. Perhaps they will inspire some confidence in you for Windows Storage Spaces.

New Features in Storage Spaces:

System Insights disk anomaly detection
System Insights is a predictive analytics feature that locally analyzes Windows Server system data and provides insight into the functioning of the server. It comes with a number of built-in capabilities, but we've added the ability to install additional capabilities via Windows Admin Center, starting with disk anomaly detection.

Disk anomaly detection is a new capability that highlights when disks are behaving differently than usual. While different isn't necessarily a bad thing, seeing these anomalous moments can be helpful when troubleshooting issues on your systems.

This capability is also available for servers running Windows Server 2019.


Drive latency outlier detection

Easily identify drives with abnormal latency with proactive monitoring and built-in outlier detection, inspired by Microsoft Azure's long-standing and successful approach. Whether it's average latency or something more subtle like 99th percentile latency that stands out, slow drives are automatically labeled in PowerShell and Windows Admin Center with ‘Abnormal Latency' status.
 

Cutha

Member
Sep 24, 2016
75
7
8
50
Canada
maybe offtopic suggestion....why not to use snapraid?? eventually with drivepool of stablebit??
Thanks, I took a quick peek at your suggestions. Interesting. What's the performance like?

For whatever reason I find myself biased towards either using a native Windows solution or using something like FeeNAS.
 

ari2asem

Active Member
Dec 26, 2018
745
128
43
The Netherlands, Groningen
Thanks, I took a quick peek at your suggestions. Interesting. What's the performance like?

For whatever reason I find myself biased towards either using a native Windows solution or using something like FeeNAS.
performance of snapraid is the performance of the disk you are reading from it or writing to it. meaning dependable on the speed of single drive.
for example, when i sync with snapraid, i hit around 130-140 MB/s because that is the max speed of my hdd.

snapraid you can see as a JBOD, with some sort of manual snapshot (you have to run sync-command) and manual parity checksum (you have to run scrub-command). you can automate this process with some scripts around the internet, but still it is something you have to do it by yourself. snapraid is totally not comparable with real HW RAID.
 

Cutha

Member
Sep 24, 2016
75
7
8
50
Canada
I could really use some help. I try hard not to bother other people when there is a possibility that if I keep working on an issue I'll sort it out myself which is why I was a lurker here for so long before posting.

I have hit a wall with this storage space and the performance is much lower than I expected. (Edit: At first I thought it was a caching problem on the SSD but I'm reasonably sure the SSD tier is just not being used at all now.)

I put in 3x NVMe 1.6TB Intel P3605's into the file server and I was kind of freaked out because when I looked at the device properties for the disks under Write-caching policy they come up with the "Enable write caching on the device" as unselected and it's impossible to enable it because "Windows could not change the write-caching setting for the device."

I went down a rabbit hole trying to get this to work. I checked the post I had about S2D and verified the P3605 is in the Windows Server Catalog as "Software-Defined Data Center (SDDC) Premium", but it's only for Server 2016 and I'm using Server 2019. Perhaps this is the issue?

When the NVMe drives are not in a pool they perform as expected when benchmarked with CrystalDiskMark. But the caching on the device was driving me nuts so I started messing about in an attempt to force it on and no matter what I tried I couldn't get to be enabled.

Code:
PS C:\Windows\system32> Get-PhysicalDisk |? FriendlyName -Like "*" | Get-StorageAdvancedProperty

FriendlyName             SerialNumber                 IsPowerProtected IsDeviceCacheE
                                                                               nabled
------------             ------------                 ---------------- --------------
NVMe INTEL SSDPEDME01    CVMD520601311P6KGN_00000001.            False          False
NVMe INTEL SSDPEDME01    CVMD513600N01P6KGN_00000001.            False          False
NVMe INTEL SSDPEDME01    CVMD511500171P6KGN_00000001.             True          False
The one that came up with IsPowerProtected as true was because of a registry setting I tried.

I found some references that said the P3600 drives do NOT have a cache. And I also found some Windows programming reference to NVMe devices and then I chased that until I used the Intel Solid-State Drive Toolbox to look at the device information and Word 525 says "Volatile Write Cache" with a big ol "0" for a Hex Value. With that I gave up and figured that there was no write cache and Storage Spaces should be fine. It is not, or I screwed something up. The write speeds are about 5x slower when used as the fast tier in a Storage Space volume (assuming it's actually writing to the SSD's at all).

Using Performance Monitor I used some ReFS counters to monitor the tiers and then copied some big files from a SSD on the same server and this is the results. Clearly I did something wrong:

1592720276723.png

NTFS, using just the Storage Spaces Tier performance tokens, behaves the same as when created with ReFS.

When running the ATTO Disk Benchmark using the same Performance Monitor tokens the SSDTier never does anything either.

These are the PS commands I used to create the pool and make the volume. There are 45x 10TB SAS HGST HDD's (dual parity) and 3x 1.6TB P3605's (2 way mirror). Perhaps the mirror needs to be three way for this to work? (Edit: Negative, tested and it makes no difference.)

Code:
$Disks = Get-PhysicalDisk | Where-Object -Property FriendlyName -like "HGST*"
$Disks += Get-PhysicalDisk | Where-Object -Property FriendlyName -like "NVMe*"

New-StoragePool -PhysicalDisks $Disks -StorageSubSystemFriendlyName "Windows Storage*" -FriendlyName "PS4_SP_GiantAssPool" -LogicalSectorSizeDefault 4096

$hddTier = New-StorageTier -StoragePoolFriendlyName "PS4_SP_GiantAssPool" -MediaType HDD -FriendlyName "PS4_SP_GiantAssPool_HDDTier" -ResiliencySettingName Parity -PhysicalDiskRedundancy 2 -NumberOfColumns 17

$ssdTier = New-StorageTier -StoragePoolFriendlyName "PS4_SP_GiantAssPool" -MediaType SSD -FriendlyName "PS4_SP_GiantAssPool_SSDTier" -ResiliencySettingName Mirror -PhysicalDiskRedundancy 1

New-Volume -StoragePoolFriendlyName "PS4_SP_GiantAssPool" -FriendlyName "VHD_GiantAssPool_Tiered" -FileSystem ReFS -StorageTiers $ssdTier,$hddTier -StorageTierSizes 1.4TB, 298TB

Set-StoragePool –FriendlyName PS4_SP_GiantAssPool –RepairPolicy Parallel

Set-StoragePool –FriendlyName PS4_SP_GiantAssPool -RetireMissingPhysicalDisks Always

Set-StoragePool -FriendlyName PS4_SP_GiantAssPool -IsPowerProtected $true
I was also a bit surprised about the resulting volume size as I thought it was 2 or 3 disk's used for parity. The 10TB drives are painful only really 9.1TB real world and 9.1TBx(45drives-3paritydrives) = 382TB but the biggest volume I can create is 300TB which leaves 21TB Free on the volume. What am I not understanding that would account for the missing ~60TB? Is there a also a % overhead cost per drive or something?

Thanks for taking the time to read this. I know it's long but hopefully it provides enough information to make it easy to help me out.
 
Last edited: