With S2D I am getting terrible writes no matter what I try. This is not a post about slow writes because of a parity setup as I have my stuff configured as all mirror. It seems like the cache devices are never written to. At first I thought I needed to use SOFS but I am using this in a hyper converged setup and going by Deploy Storage Spaces Direct (Step 4) it is not required. I tried SOFS and General File Server as cluster roles with storage attached in various ways but the results were always pretty poor.
To keep it simple there is one VM allocated with 8 vCPU’s and a drive (VHDX fixed) for benchmarking against. Currently I have no FS roles setup (SOFS or General) and I am just using the C:\ClusterStorage\. I have tried it with SOFS and General but from what I can tell I shouldn’t need it for the setup I am using. At some point I would like to have a share for client workstation VHDX’s with the quickest configuration possible, CA not required but HA would be nice.
RDMA is tested and working and drives perform as expected on their own. I am waiting until I get this sorted out before I deploy this so at the moment I can do whatever configuring and testing is required. I am stumped and disappointed with myself. I have spent 2 years gathering the hardware to make this work and for the life of me I can't get it to perform. Any help would be appreciated.
I have 4 nodes and each node has:
2xNVMe 250GB (SM961, has PLP)(Does NOT have PLP, I was incorrect)
3xSSD 480GB (MZ7WD480, has PLP)
4xHDD 4TB (consumer grade SATA)
ConnectX-2 dual port
2xE5-2670v1
96GB RAM
Windows Server 2016
I was a little confused as to which performance counters I should be looking at to see if the cache was being used but from everything I could see, it was not being used during any of my tests. I have read everything I could find online that I think might help me and I don't know what to try next.
Benchmarks on the client with the client VHDX on different volumes. The info on the volumes is below and they are in the same order.
Individual drives I tested some time ago:
1xNVMe
1xSATA SSD
I wanted to use the HDD's in a 3 way mirror so I created the CapacityMirror with:
I set the CSV cache with:
To keep it simple there is one VM allocated with 8 vCPU’s and a drive (VHDX fixed) for benchmarking against. Currently I have no FS roles setup (SOFS or General) and I am just using the C:\ClusterStorage\. I have tried it with SOFS and General but from what I can tell I shouldn’t need it for the setup I am using. At some point I would like to have a share for client workstation VHDX’s with the quickest configuration possible, CA not required but HA would be nice.
RDMA is tested and working and drives perform as expected on their own. I am waiting until I get this sorted out before I deploy this so at the moment I can do whatever configuring and testing is required. I am stumped and disappointed with myself. I have spent 2 years gathering the hardware to make this work and for the life of me I can't get it to perform. Any help would be appreciated.
I have 4 nodes and each node has:
2xNVMe 250GB (SM961, has PLP)(Does NOT have PLP, I was incorrect)
3xSSD 480GB (MZ7WD480, has PLP)
4xHDD 4TB (consumer grade SATA)
ConnectX-2 dual port
2xE5-2670v1
96GB RAM
Windows Server 2016
I was a little confused as to which performance counters I should be looking at to see if the cache was being used but from everything I could see, it was not being used during any of my tests. I have read everything I could find online that I think might help me and I don't know what to try next.
Benchmarks on the client with the client VHDX on different volumes. The info on the volumes is below and they are in the same order.
Code:
New-Volume -FriendlyName "CSV-1" -FileSystem CSVFS_ReFS -StoragePoolFriendlyName S2D* -Size 100GB
New-Volume -FriendlyName "Test0SSDOnly" -FileSystem CSVFS_ReFS -StoragePoolFriendlyName S2D* -Size 10GB -ResiliencySettingName Mirror
New-Volume -FriendlyName "Test2MRV" -FileSystem CSVFS_ReFS -StoragePoolFriendlyName S2D* -StorageTierFriendlyNames Performance, Capacity -StorageTierSizes 50GB, 50GB
New-Volume -FriendlyName "Test3Tiered" -FileSystem CSVFS_ReFS -StoragePoolFriendlyName S2D* -StorageTierFriendlyNames Performance, CapacityMirror -StorageTierSizes 50GB, 50GB
1xNVMe
1xSATA SSD
Code:
Get-ClusterS2D
CacheMetadataReserveBytes : 34359738368
CacheModeHDD : ReadWrite
CacheModeSSD : WriteOnly
CachePageSizeKBytes : 16
CacheState : Enabled
State : Enabled
Code:
Get-ClusterStorageSpacesDirect
CacheMetadataReserveBytes : 34359738368
CacheModeHDD : ReadWrite
CacheModeSSD : WriteOnly
CachePageSizeKBytes : 16
CacheState : Enabled
State : Enabled
Code:
Get-StorageTier | FT FriendlyName, ResiliencySettingName, MediaType, PhysicalDiskRedundancy -autosize
FriendlyName ResiliencySettingName MediaType PhysicalDiskRedundancy
------------ --------------------- --------- ----------------------
CapacityMirror Mirror HDD 2
Capacity Parity HDD 2
Performance Mirror SSD 2
Code:
$hddTierMirror = New-StorageTier -StoragePoolFriendlyName "S2D on theCluster" -MediaType HDD -FriendlyName "CapacityMirror" -ResiliencySettingName Mirror -NumberOfDataCopies 3
Code:
(Get-Cluster $ClusterName).BlockCacheSize = $CSVCacheSize
Code:
Get-StorageSubSystem *cluster* | Get-StorageHealthReport
CPUUsageAverage : 2.52 %
CapacityPhysicalPooledAvailable : 45.26 TB
CapacityPhysicalPooledTotal : 65.32 TB
CapacityPhysicalTotal : 65.32 TB
CapacityPhysicalUnpooled : 0 B
CapacityVolumesAvailable : 5.83 TB
CapacityVolumesTotal : 5.91 TB
IOLatencyAverage : 0 ns
IOLatencyRead : 0 ns
IOLatencyWrite : 0 ns
IOPSRead : 0 /S
IOPSTotal : 0 /S
IOPSWrite : 0 /S
IOThroughputRead : 0 B/S
IOThroughputTotal : 0 B/S
IOThroughputWrite : 0 B/S
MemoryAvailable : 377.19 GB
MemoryTotal : 416 GB
ExtendedStatus :
ReturnValue : 0
PSComputerName :
Code:
Get-PhysicalDisk
FriendlyName SerialNumber CanPool OperationalStatus HealthStatus Usage Size
------------ ------------ ------- ----------------- ------------ ----- ----
ATA INTEL SSDSA2BW16 CVPR1290041L160DGN False OK Healthy Auto-Select 149.05 GB
ATA SAMSUNG MZ7WD480 S1G1NYAF926440 False OK Healthy Auto-Select 447 GB
ATA SAMSUNG MZ7WD480 S16MNYAD911565 False OK Healthy Auto-Select 447 GB
ATA SAMSUNG MZ7WD480 S16MNYAF605357 False OK Healthy Auto-Select 447 GB
ATA SAMSUNG MZ7WD480 S16MNYAD911436 False OK Healthy Auto-Select 447 GB
SAMSUNG MZVPW256HEGL-00000 0025_38CC_6100_E84F. False OK Healthy Journal 238.25 GB
ATA ST4000DM005-2DP1 WDH1L3DG False OK Healthy Auto-Select 3.64 TB
SAMSUNG MZVPW256HEGL-00000 0025_38CC_6100_E88B. False OK Healthy Journal 238.25 GB
ATA ST4000DM005-2DP1 ZDH0L26D False OK Healthy Auto-Select 3.64 TB
SAMSUNG MZVPW256HEGL-00000 0025_38CB_6100_B550. False OK Healthy Journal 238.25 GB
ATA SAMSUNG MZ7WD480 S16MNEAD740753 False OK Healthy Auto-Select 447 GB
ATA ST4000DM005-2DP1 ZDH0X9QM False OK Healthy Auto-Select 3.64 TB
SAMSUNG MZVPW256HEGL-00000 0025_38CC_6100_E846. False OK Healthy Journal 238.25 GB
ATA ST4000VN000-1H41 Z301MZSZ False OK Healthy Auto-Select 3.64 TB
SAMSUNG MZVPW256HEGL-00000 0025_38CB_6100_BFA2. False OK Healthy Journal 238.25 GB
ATA SAMSUNG MZ7WD480 S16MNEAD500108 False OK Healthy Auto-Select 447 GB
ATA ST4000DM005-2DP1 ZDH0BSBV False OK Healthy Auto-Select 3.64 TB
ATA ST4000VN000-1H41 Z306C6GF False OK Healthy Auto-Select 3.64 TB
SAMSUNG MZVPW256HEGL-00000 0025_38CC_6100_E8EF. False OK Healthy Journal 238.25 GB
ATA ST4000VN000-2AH1 WDH0KMY3 False OK Healthy Auto-Select 3.64 TB
SAMSUNG MZVPW256HEGL-00000 0025_38CC_6100_E84A. False OK Healthy Journal 238.25 GB
ATA ST4000VN000-1H41 Z3060E2J False OK Healthy Auto-Select 3.64 TB
ATA ST4000DM005-2DP1 WDH1L39R False OK Healthy Auto-Select 3.64 TB
ATA ST4000VN000-2AH1 WDH0KN75 False OK Healthy Auto-Select 3.64 TB
ATA SAMSUNG MZ7WD480 S16MNYAF604909 False OK Healthy Auto-Select 447 GB
ATA ST4000VN000-2AH1 WDH0KNEM False OK Healthy Auto-Select 3.64 TB
ATA ST4000VN000-1H41 Z301NEKY False OK Healthy Auto-Select 3.64 TB
ATA ST4000VN000-1H41 Z304XGDB False OK Healthy Auto-Select 3.64 TB
ATA ST4000DM005-2DP1 ZDH0X90N False OK Healthy Auto-Select 3.64 TB
SAMSUNG MZVPW256HEGL-00000 0025_38CC_6100_E8C7. False OK Healthy Journal 238.25 GB
ATA SAMSUNG MZ7WD480 S1G1NYAF921789 False OK Healthy Auto-Select 447 GB
ATA SAMSUNG MZ7WD480 S1G1NYAF925500 False OK Healthy Auto-Select 447 GB
ATA SAMSUNG MZ7WD480 S16MNYAF114484 False OK Healthy Auto-Select 447 GB
ATA ST4000DM005-2DP1 ZDH0X9SJ False OK Healthy Auto-Select 3.64 TB
ATA SAMSUNG MZ7WD480 S16MNYAF334083 False OK Healthy Auto-Select 447 GB
ATA SAMSUNG MZ7WD480 S1G1NYAF921332 False OK Healthy Auto-Select 447 GB
ATA ST4000VN000-2AH1 WDH0KNE5 False OK Healthy Auto-Select 3.64 TB
Last edited: