Storage Spaces (Server2016) with lots of disks

Cutha

Member
Sep 24, 2016
58
6
8
46
Canada
Hi,

I have about 40-50 3TB drives and my plan was to have them connected in a giant pool with much of it being used as 20-40TB virtual disks created as dual parity. I was planning on leaving about 10TB RAW free on the storage space so that on a disk failure storage spaces would have enough room to auto repair.

I can't seem to find a recommendation for the max number of drives to have in a pool to maintain a healthy safety margin. I know, or at least think I know, that this does not really have an analogous comparison with a HW RAID controller. Does anyone have a recommendation for this?

Also, I have 5x500GB SATA SSD's that I want to use with this pool. I was going to create tiered spaces with a small 3 way mirrored for each virtual disk but maybe it's better if I just set them as Journal drives? I think as Journal drives they will server as a write buffer to all the virtual disks in the storage space.(?)

This entire pool is primarily for media storage. At one point I was thinking I was just going to make a single massive virtual disk but I figured it would be a pain to work with. But maybe not...

Thanks for taking the time to read this. I hope it's reasonably coherent; you would think I would learn about creating posts at 4:30AM.
 

i386

Well-Known Member
Mar 18, 2016
1,849
471
83
31
A storage space pool can have up to 240 devices (4 jbods with 60 disks each), Server 2012 up to 160 disks.
TechNet Wiki

I think you want to know the number of disks you can put in a storage space (=logical volume). I can't find any offical number for that, looking at hardware and software raid implementations there is often a limit of 32 hdds/ssds/nvme devices for one logical volume. So I would say use 32 devices for the storage space and keep 2 hot spares per storage space.
 

Cutha

Member
Sep 24, 2016
58
6
8
46
Canada
I think you want to know the number of disks you can put in a storage space (=logical volume). I can't find any offical number for that, looking at hardware and software raid implementations there is often a limit of 32 hdds/ssds/nvme devices for one logical volume. So I would say use 32 devices for the storage space and keep 2 hot spares per storage space.
As far as I know, if you had 100 HDDs in a Storage Pool, when you create a virtual disk there is no way to control which disks it spans. With dual parity I think the max number of columns is 17. I am not sure if then it only uses up to 17 disks.

What I am trying to ask is: If I had ~40*3TB HDD's and I am ok with the performance of dual parity, what is the most efficient way to configure Storage Spaces while maintaining a reasonable level of safety? With SS Direct MS recommends leaving unused space per server = 1*capacity HDD for quick auto rebuilds. I was thinking that on a large (single server) pool I could afford to leave 3*HDD unused.
 

cesmith9999

Well-Known Member
Mar 26, 2013
1,146
347
83
Storage Spaces spans all disks in the pool. When you create the virtual disks you need to specify the columncount using PowerShell. The default column count is 8 when you have more than 8 disks in the pool.

Also remember that using dual parity means that you lose 1 disk to global parity. I recommend that you leave 2 or 3 disks worth of space availabl in the pool for sparing. I also recommend that you leave a couple of drives out of the pool to allow for pool disk replacements.

Chris
 
  • Like
Reactions: T_Minus

Cutha

Member
Sep 24, 2016
58
6
8
46
Canada
Thanks for the reply Chris.

So, is a pool with 40 disks unreasonable?

40*3TB - 9TB(for pool replacements) - 9TB(dual parity) = 102TB?

The deal with URE, growing HDD sizes, hardware RAID6 and the likely hood of another drive dropping out during a rebuild has caused me to be a little paranoid. In an effort to hedge my bets the best I can, I settled on SS's w/DP and ReFS.
 

cesmith9999

Well-Known Member
Mar 26, 2013
1,146
347
83
I have servers in my care using Storage Spaces with over 100 disks. so 40 is not unreasonable pool size.

Chris
 

Cutha

Member
Sep 24, 2016
58
6
8
46
Canada
Thanks.

Also, I have 5x500GB SATA SSD's that I want to use with this pool. I was going to create tiered spaces with a small 3 way mirrored for each virtual disk but maybe it's better if I just set them as Journal drives? I think as Journal drives they will server as a write buffer to all the virtual disks in the storage space.(?)
 

Evan

Well-Known Member
Jan 6, 2016
3,012
494
83
If those are not enterprise SSD (sata) and actually consumer drives then the results will not be good, but if it’s really low write environment you may get away with it being useful.
 

Cutha

Member
Sep 24, 2016
58
6
8
46
Canada
I thought I would give MRV (Multi-Resilient Volume) a shot but I get an error when I use a higher column count for the mirror tier. I have 33 x 3TB HDD's and no SSD's in the pool. If I change NumberOfColumns on the mirror tier to anything > 4 I get an error: The storage pool does not have sufficient eligible resources for the creation of the specified virtual disk.

$hddTier = New-StorageTier -StoragePoolFriendlyName PS3_Pool2 -MediaType HDD -FriendlyName "ST_HDD" -NumberOfColumns 17 -ResiliencySettingName Parity -PhysicalDiskRedundancy 2

$hddTierMirror = New-StorageTier -StoragePoolFriendlyName PS3_Pool2 -MediaType HDD -FriendlyName "ST_HDDMirror" -ResiliencySettingName Mirror -NumberOfDataCopies 3 -NumberOfColumns 4

New-Volume -StoragePoolFriendlyName PS3_Pool2 -FriendlyName "PS3_MRV_Media" -FileSystem ReFS -StorageTiers $hddTierMirror,$hddTier -StorageTierSizes 2000GB,30TB

DriveLetter FileSystemLabel FileSystem DriveType HealthStatus OperationalStatus SizeRemaining Size
----------- --------------- ---------- --------- ------------ ----------------- ------------- ----
PS3_MRV_Media ReFS Fixed Healthy OK 31.75 TB 31.97 TB


I thought with 3 way mirror I would be able to have 12 columns.
 

Cutha

Member
Sep 24, 2016
58
6
8
46
Canada
Hi,

So a few years later...

Storage Spaces has been great if a bit painful at times from stupidity on my part.

I have about a zillion 3TB HGST drives over a few different JBOD's and too many servers and I wanted to consolidate and also drop my power bill so I bought a 45x10TB JBOD. I have had zero data loss with using SS and I have only had to replace a couple HDD's and that wasn't so bad especially once MS intruduced the optimize command.

I am planning on using dual parity and leaving 3 disks worth of empty space so that the newer feature of Storage Spaces automagically rebuilding into the spare space is seamless and quicker.

Automatically rebuild storage spaces from storage pool free space
Storage Spaces now includes the ability to automatically rebuild storage spaces from storage pool free space instead of using hot spares.

What value does this change add?

Rebuild times are accelerated because multiple disks in the pool can accept the data that was stored on the failed disk instead of waiting for a single hot spare to write all of the data. Additionally, hot spare drives are no longer needed, and storage pool free space can provide additional capacity and performance to the pool.

What works differently?

When a physical disk fails, instead of writing a copy of the data that was on the failed disk to a single hot spare, the data is copied to multiple physical disks in the pool so that the previous level of resiliency is achieved. Administrators no longer need to allocate physical disks as hot spares in the storage pool.

I'm also going to use a few Intel P3605's as an SSD tier so the write speed doesn't drive me nuts.

I would love some feedback on my plan as long as it's not "Use Linux" :)
 

Lost-Benji

Member
Jan 21, 2013
424
22
18
The arse end of the planet
Only just seen this.
Was going to say, too late, already done and doing this.

- Each array should be on same physical chassis, DO NOT SPAN OVER TO OTHER CHASSIS! You can keep some global Hot-Spares in another chassis, just mark them so easy to ID when you need to swap.
- Double-Parity for anything more than 6-8 drives.
- Use good HBA's and cables.
- Be happy with the shitty write speeds but as mine is a Media Server, I survive.
- RAM, boat loads of it if you want to dump movies to SS parity arrays. When you send large files to the array, it will load into the RAM like a Cache and keep writing after it hasclosed the transfer window.
- Windows is pure shit at detecting bad drives, it will keep telling you that a drive is Healthy despite it throwing disk errors all through the event logs. Keep an eye on the logs from time-to-time.
- Learn the commands for Identifying correct drives and their serial numbers, Windows re-orders drives and their locations all the time making hard to trace bad sucker.
 

Cutha

Member
Sep 24, 2016
58
6
8
46
Canada
- Each array should be on same physical chassis, DO NOT SPAN OVER TO OTHER CHASSIS! You can keep some global Hot-Spares in another chassis, just mark them so easy to ID when you need to swap.
Do you use Hot-Spares? I stopped using hot spares once the feature to "Automatically rebuild storage spaces from storage pool free space" was available. Now I make sure my virtual disk leaves at least a few drives worth of unused space. However, I don't think I have seen it automatically rebuild on it's own but I may not have actually had a drive fail since using this.

- Be happy with the shitty write speeds but as mine is a Media Server, I survive.
Do you use a tiering or cache drive? I was thinking of using a few 1.6TB P3605's as the SSD tier. I think the WBC is still limited to 100GB's so I think it would be a waste for me to use them as a WBC.

- Windows is pure shit at detecting bad drives, it will keep telling you that a drive is Healthy despite it throwing disk errors all through the event logs. Keep an eye on the logs from time-to-time.
That hasn't been my experience so far, the few that failed showed up cleanly in the Server Manager->File and Storage Service window. I couldn't get the flash disk LED function to work so to find the offline drive I just did a file copy and then looked for the HDD that wasn't blinking. Then I made a table with HDD serial # and JBOD slot # so I didn't have to mess about next time.

I also use Hard Disk Sentinel to help me keep an eye on all the HDD's health.
 

Lost-Benji

Member
Jan 21, 2013
424
22
18
The arse end of the planet
Do you use Hot-Spares? I stopped using hot spares once the feature to "Automatically rebuild storage spaces from storage pool free space" was available. Now I make sure my virtual disk leaves at least a few drives worth of unused space. However, I don't think I have seen it automatically rebuild on it's own but I may not have actually had a drive fail since using this.
I use warm drives. A coupld that stay in the main server and I add when I find a failing drive. Then move to chassis after repaired.

Do you use a tiering or cache drive? I was thinking of using a few 1.6TB P3605's as the SSD tier. I think the WBC is still limited to 100GB's so I think it would be a waste for me to use them as a WBC.
Nope, I have 3x 16-drive DAS chassis, all full of 3TB drives with movies and TV, no need to waste SSD's or the time to get tiering working. If I need that sort of I/O, I used HW RAID cards.

That hasn't been my experience so far, the few that failed showed up cleanly in the Server Manager->File and Storage Service window. I couldn't get the flash disk LED function to work so to find the offline drive I just did a file copy and then looked for the HDD that wasn't blinking. Then I made a table with HDD serial # and JBOD slot # so I didn't have to mess about next time.
Windows sucks, trust me. By the time SS tells you are drive is bad, damage is done. Lost multiple arrays from it. Watch your Event Logs closely or if media playback is slugish in my case, I know I have a spinner on way out. As for drive bay lights, comes down to controller (HBA) and expanders passing commands correctly.

I also use Hard Disk Sentinel to help me keep an eye on all the HDD's health.
Had it,gave up as it is useless at the numbers of drives I run and it doesn't talk to all drives either.