I have a Windows 2016 server with 2x 2TB Intel P3700 SSDs + 36x 8TB Spinning Rust. They were configured as a tiered RAID6 using Storage Spaces (not Storage Spaces Direct) using code similar (but not exactly the same) to this:
The virtual disk is formatted with ReFS.
This configuration works fine accepting incoming writes until the SSD tier fills up. After that performance tanks. I've spent some time looking at PerfMon counters trying to find something that stands out as wrong/unusual. I found it, but I don't understand it, and therefore need your help.
This is a picture of Physical Disk Idle Times (%) for the disks used in the Storage Pool. The two on the right-hand side I believe are the SSDs. Everything else is Spinning Rust. When the SSD tier fills up, it starts destaging/detiering data from SSD --> HDD. Since this is a RAID6/Double Parity volume that consumes all the available space (250TB), I would have expected all the Spinning Rust to show similar Idle Times. Instead, it looks like two of the drives are getting pounded while the other drives are mostly idle.
This suggests that I have misconfigured something and now I'm stuck with a performance-tanking bottleneck. What did I do wrong?
Bash:
$ErrorActionPreference = "Stop";
$StoragePoolFriendlyName = 'Middle Chassis 36 Bays';
$StoragePool = Get-StoragePool -FriendlyName $StoragePoolFriendlyName;
$PhysicalDisks = $StoragePool | Get-PhysicalDisk;
# NOTE: Two of the $PhysicalDisks were marked as hot spares.
New-VirtualDisk -StoragePoolFriendlyName $StoragePoolFriendlyName `
-FriendlyName "RAID6" `
-UseMaximumSize `
-ProvisioningType Fixed `
-ResiliencySettingName "Parity" `
-PhysicalDiskRedundancy 2;
# Create Storage Tiers
$SpinningRust = New-StorageTier -StoragePoolFriendlyName $StoragePoolFriendlyName `
-FriendlyName SpinningRust `
-MediaType HDD `
-ResiliencySettingName Parity `
-PhysicalDiskRedundancy 2;
$NVMe = New-StorageTier -StoragePoolFriendlyName $StoragePoolFriendlyName `
-FriendlyName NVMe `
-MediaType SSD `
-ResiliencySettingName Mirror `
-PhysicalDiskRedundancy 1 `
-NumberOfDataCopies 2;
# Create Virtual Disk (2-way NVMe mirror + Dual-Parity HDD)
# Put the Virtual Disk creation code inside a loop so it finds the max size allowable.
$SizeTB = 190;
Do {
$FailureEncountered = $False;
Try {
Write-Host "Trying $SizeTB";
$VirtualDisk = New-Volume -StoragePoolFriendlyName $StoragePoolFriendlyName `
-FriendlyName TieredRAID6 `
-StorageTierFriendlyNames $NVMe.FriendlyName,$SpinningRust.FriendlyName `
-StorageTierSizes 900GB,($SizeTB*1TB) `
-WriteCacheSize 900GB `
-FileSystem ReFS `
-AccessPath "D:" `
-AllocationUnitSize 64KB;
}
Catch {
$Error.Clear();
$SizeTB = $SizeTB -1;
$FailureEncountered = $True;
}
} While ($FailureEncountered)
This configuration works fine accepting incoming writes until the SSD tier fills up. After that performance tanks. I've spent some time looking at PerfMon counters trying to find something that stands out as wrong/unusual. I found it, but I don't understand it, and therefore need your help.
This is a picture of Physical Disk Idle Times (%) for the disks used in the Storage Pool. The two on the right-hand side I believe are the SSDs. Everything else is Spinning Rust. When the SSD tier fills up, it starts destaging/detiering data from SSD --> HDD. Since this is a RAID6/Double Parity volume that consumes all the available space (250TB), I would have expected all the Spinning Rust to show similar Idle Times. Instead, it looks like two of the drives are getting pounded while the other drives are mostly idle.
This suggests that I have misconfigured something and now I'm stuck with a performance-tanking bottleneck. What did I do wrong?
Attachments
-
81.8 KB Views: 0