I deployed a 3 node S2D cluster to run VM's in the failover cluster. I made the cluster with the following hardware configuration. Each node has the same configuration.
Dell PowerEdge R740xd
1x Xeon Gold 6152 22 Cores
386GB DDR-4 RAM
8x 1.2TB Seagate ST1200mm007 1.2TB SAS HDD
2x PM1725 3.2TB NVME PCI-SSD
1x Mellanox ConnectX-3 40Gbit network adapters
1x Dell BOSS PCI card 2x 256GB SATA SSD RAID 1
The servers are connected to a Dell S4048-ON switch with 40Gbit Direct Attach cables. Both NIC slots connect to the switch. In Windows Server I configurered the uplinks as failover with one uplink as standby.
I enabled S2D pool by entering the CMDlet: "Enable-ClusterStorageSpacesDirect -PoolFriendlyName S2DPool -CacheState Enabled"
Then I created a virtual disk three-way-mirror and assigned 80% of the storage of the pool to the virtual disk.
The NVME drives are assigned the cache role and the HDD's are assigned in the Capacity Tier. I did a benchmark from a VM running in the S2D cluster. On Linux I tested the performance with fio with the following command: "fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=random_read_write.fio --bs=4k --iodepth=64 --size=4G --readwrite=randrw"
The result from the VM inside S2D is:
READ: bw=6889KiB/s (7054kB/s), 6889KiB/s-6889KiB/s (7054kB/s-7054kB/s), io=2049MiB (2149MB), run=304620-304620msec
WRITE: bw=6880KiB/s (7045kB/s), 6880KiB/s-6880KiB/s (7045kB/s-7045kB/s), io=2047MiB (2146MB), run=304620-304620msec
I did the same test on a standalone server with a local storage - PM1725 NVME SSD
Same OS and same command for the benchmark the result:
READ: io=2049.2MB, aggrb=238366KB/s, minb=238366KB/s, maxb=238366KB/s, mint=8803msec, maxt=8803msec
WRITE: io=2046.9MB, aggrb=238096KB/s, minb=238096KB/s, maxb=238096KB/s, mint=8803msec, maxt=8803msec
I destroyed the S2D and disabled the S2D-cluster to confirm the speeds of the NVME drives. I tested all 6 NVME drives one by one and the speeds I get from the direct storage is around the screenshot below.
Next day try, I created a new S2D cluster with no cache and the NVME drives assigned in the Performance Tier and SAS drives in the capacity tier. Again speeds with same slow results. There must be a misconfiguration. I read another topic here about PLP, I did some research and the PM1725 should support the PLP.
Result of Get-StoragePool "S2D on HVCLUSTER" | FL
Get-ClusterStorageSpacesDirect
Get-NetAdapterRdma
Get-StorageJob does not return anything.
I ran the following command, this is returning some errors:
Get-PhysicalDisk | Get-StorageAdvancedProperty
I can confirm the write/read cache is being used from the Windows Admin Centre. However how could it be so slow?
Dell PowerEdge R740xd
1x Xeon Gold 6152 22 Cores
386GB DDR-4 RAM
8x 1.2TB Seagate ST1200mm007 1.2TB SAS HDD
2x PM1725 3.2TB NVME PCI-SSD
1x Mellanox ConnectX-3 40Gbit network adapters
1x Dell BOSS PCI card 2x 256GB SATA SSD RAID 1
The servers are connected to a Dell S4048-ON switch with 40Gbit Direct Attach cables. Both NIC slots connect to the switch. In Windows Server I configurered the uplinks as failover with one uplink as standby.
I enabled S2D pool by entering the CMDlet: "Enable-ClusterStorageSpacesDirect -PoolFriendlyName S2DPool -CacheState Enabled"
Then I created a virtual disk three-way-mirror and assigned 80% of the storage of the pool to the virtual disk.
The NVME drives are assigned the cache role and the HDD's are assigned in the Capacity Tier. I did a benchmark from a VM running in the S2D cluster. On Linux I tested the performance with fio with the following command: "fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=random_read_write.fio --bs=4k --iodepth=64 --size=4G --readwrite=randrw"
The result from the VM inside S2D is:
READ: bw=6889KiB/s (7054kB/s), 6889KiB/s-6889KiB/s (7054kB/s-7054kB/s), io=2049MiB (2149MB), run=304620-304620msec
WRITE: bw=6880KiB/s (7045kB/s), 6880KiB/s-6880KiB/s (7045kB/s-7045kB/s), io=2047MiB (2146MB), run=304620-304620msec
I did the same test on a standalone server with a local storage - PM1725 NVME SSD
Same OS and same command for the benchmark the result:
READ: io=2049.2MB, aggrb=238366KB/s, minb=238366KB/s, maxb=238366KB/s, mint=8803msec, maxt=8803msec
WRITE: io=2046.9MB, aggrb=238096KB/s, minb=238096KB/s, maxb=238096KB/s, mint=8803msec, maxt=8803msec
I destroyed the S2D and disabled the S2D-cluster to confirm the speeds of the NVME drives. I tested all 6 NVME drives one by one and the speeds I get from the direct storage is around the screenshot below.
Next day try, I created a new S2D cluster with no cache and the NVME drives assigned in the Performance Tier and SAS drives in the capacity tier. Again speeds with same slow results. There must be a misconfiguration. I read another topic here about PLP, I did some research and the PM1725 should support the PLP.
Result of Get-StoragePool "S2D on HVCLUSTER" | FL
Code:
PassThroughClass :
PassThroughIds :
PassThroughNamespace :
PassThroughServer :
AllocatedSize : 28101165711360
ClearOnDeallocate : False
EnclosureAwareDefault : False
FaultDomainAwarenessDefault : StorageScaleUnit
FriendlyName : S2D on HVCLUSTER
HealthStatus : Healthy
IsClustered : True
IsPowerProtected : True
IsPrimordial : False
IsReadOnly : False
LogicalSectorSize : 4096
MediaTypeDefault : Unspecified
Name :
OperationalStatus : OK
OtherOperationalStatusDescription :
OtherUsageDescription : Reserved for S2D
PhysicalSectorSize : 4096
ProvisioningTypeDefault : Fixed
ReadOnlyReason : None
RepairPolicy : Parallel
ResiliencySettingNameDefault : Mirror
RetireMissingPhysicalDisks : Never
Size : 28982995058688
SupportedProvisioningTypes : Fixed
SupportsDeduplication : True
ThinProvisioningAlertThresholds : {70}
Usage : Other
Version : Windows Server 2019
WriteCacheSizeDefault : Auto
WriteCacheSizeMax : 18446744073709551614
WriteCacheSizeMin : 0
PSComputerName :
Code:
CacheMetadataReserveBytes : 34359738368
CacheModeHDD : ReadWrite
CacheModeSSD : WriteOnly
CachePageSizeKBytes : 16
CacheState : Enabled
Name : HVCLUSTER
ScmUse : Cache
State : Enabled
Code:
Name InterfaceDescription Enabled PFC ETS
---- -------------------- ------- --- ---
SLOT 3 Mellanox ConnectX-3 Pro Ethernet Adapter True False False
SLOT 3 2 Mellanox ConnectX-3 Pro Ethernet Adap... True False False
Get-StorageJob does not return anything.
I ran the following command, this is returning some errors:
Get-PhysicalDisk | Get-StorageAdvancedProperty
Code:
FriendlyName SerialNumber IsPowerProtected IsDeviceCacheEnabled
------------ ------------ ---------------- --------------------
MSFT Virtual HD 4B621B92-2B4A-4B0F-90A8-B0EA6AF33E92 False True
DELLBOSS VD 5d8224e28f810010 False True
NETAPP X425_SIRMN1T2A10 S3L0Z76F0000M5174T7Z False
NETAPP X425_SIRMN1T2A10 S3L0Z9XZ0000M5184N3B False
WARNING: Retrieving IsDeviceCacheEnabled failed with ErrorCode 1.
MS1PC5ED3ORA3.2T 0025_3893_6100_08B2. False
WARNING: Retrieving IsDeviceCacheEnabled failed with ErrorCode 1.
NETAPP X425_SIRMN1T2A10 S3L0XB4B0000M5168C2Q False
WARNING: Retrieving IsDeviceCacheEnabled failed with ErrorCode 1.
NETAPP X425_SIRMN1T2A10 S3L0YPT70000M516EAY7 False
WARNING: Retrieving IsDeviceCacheEnabled failed with ErrorCode 1.
NETAPP X425_SIRMN1T2A10 S3L0XDRB0000M511CXDN False
WARNING: Retrieving IsDeviceCacheEnabled failed with ErrorCode 1.
NETAPP X425_SIRMN1T2A10 S3L0X47S0000M515EUDS False
WARNING: Retrieving IsDeviceCacheEnabled failed with ErrorCode 1.
NETAPP X425_SIRMN1T2A10 S3L0XC330000M516AWXK False
WARNING: Retrieving IsDeviceCacheEnabled failed with ErrorCode 1.
NETAPP X425_SIRMN1T2A10 S3L0XEG50000M516008L False
WARNING: Retrieving IsDeviceCacheEnabled failed with ErrorCode 1.
NETAPP X425_SIRMN1T2A10 S3L0XBYB0000M516AWW7 False
WARNING: Retrieving IsDeviceCacheEnabled failed with ErrorCode 1.
NETAPP X425_SIRMN1T2A10 S3L0XE400000M515UM96 False
WARNING: Retrieving IsDeviceCacheEnabled failed with ErrorCode 1.
NETAPP X425_SIRMN1T2A10 S3L0ZAPW0000M51800C9 False
WARNING: Retrieving IsDeviceCacheEnabled failed with ErrorCode 1.
MS1PC5ED3ORA3.2T 0025_3893_6100_08BF. False
WARNING: Retrieving IsDeviceCacheEnabled failed with ErrorCode 1.
NETAPP X425_SIRMN1T2A10 S3L0YNW20000M516AXGE False
WARNING: Retrieving IsDeviceCacheEnabled failed with ErrorCode 1.
NETAPP X425_SIRMN1T2A10 S3L0YQ3T0000M515XP6G False
WARNING: Retrieving IsDeviceCacheEnabled failed with ErrorCode 1.
MS1PC5ED3ORA3.2T 0025_3894_6100_06D0. False
WARNING: Retrieving IsDeviceCacheEnabled failed with ErrorCode 1.
NETAPP X425_SIRMN1T2A10 S3L0YPTA0000M516EB0G False
WARNING: Retrieving IsDeviceCacheEnabled failed with ErrorCode 1.
NETAPP X425_SIRMN1T2A10 S3L0YDQP0000M5175H4D False
WARNING: Retrieving IsDeviceCacheEnabled failed with ErrorCode 1.
NETAPP X425_SIRMN1T2A10 S3L0Z9XC0000M5184MZ9 False
WARNING: Retrieving IsDeviceCacheEnabled failed with ErrorCode 1.
NETAPP X425_SIRMN1T2A10 S3L0ZASY0000M5159KWG False
WARNING: Retrieving IsDeviceCacheEnabled failed with ErrorCode 1.
NETAPP X425_SIRMN1T2A10 S3L0XAYP0000M515XPHQ False
WARNING: Retrieving IsDeviceCacheEnabled failed with ErrorCode 1.
NETAPP X425_SIRMN1T2A10 S3L0XEEF0000M516EAZG False
WARNING: Retrieving IsDeviceCacheEnabled failed with ErrorCode 1.
MS1PC5ED3ORA3.2T 0025_3893_6100_08C1. False
WARNING: Retrieving IsDeviceCacheEnabled failed with ErrorCode 1.
NETAPP X425_SIRMN1T2A10 S3L0XEH50000M51600B7 False
WARNING: Retrieving IsDeviceCacheEnabled failed with ErrorCode 1.
MS1PC5ED3ORA3.2T 0025_3894_6100_06CC. False
WARNING: Retrieving IsDeviceCacheEnabled failed with ErrorCode 1.
NETAPP X425_SIRMN1T2A10 S3L0XB6T0000M5168CD0 False
WARNING: Retrieving IsDeviceCacheEnabled failed with ErrorCode 1.
NETAPP X425_SIRMN1T2A10 S3L0XELZ0000M516EAG7 False
WARNING: Retrieving IsDeviceCacheEnabled failed with ErrorCode 1.
NETAPP X425_SIRMN1T2A10 S3L0XELX0000M516EAKK False
WARNING: Retrieving IsDeviceCacheEnabled failed with ErrorCode 1.
MS1PC5ED3ORA3.2T 0025_3894_6100_06DF. False
WARNING: Retrieving IsDeviceCacheEnabled failed with ErrorCode 1.
NETAPP X425_SIRMN1T2A10 S3L0ZAT00000M514SAG2 False
Last edited: