S2D poor cache performance

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

Timmerdanny

New Member
Oct 21, 2019
8
0
1
I deployed a 3 node S2D cluster to run VM's in the failover cluster. I made the cluster with the following hardware configuration. Each node has the same configuration.

Dell PowerEdge R740xd
1x Xeon Gold 6152 22 Cores
386GB DDR-4 RAM
8x 1.2TB Seagate ST1200mm007 1.2TB SAS HDD
2x PM1725 3.2TB NVME PCI-SSD
1x Mellanox ConnectX-3 40Gbit network adapters
1x Dell BOSS PCI card 2x 256GB SATA SSD RAID 1

The servers are connected to a Dell S4048-ON switch with 40Gbit Direct Attach cables. Both NIC slots connect to the switch. In Windows Server I configurered the uplinks as failover with one uplink as standby.

I enabled S2D pool by entering the CMDlet: "Enable-ClusterStorageSpacesDirect -PoolFriendlyName S2DPool -CacheState Enabled"

Then I created a virtual disk three-way-mirror and assigned 80% of the storage of the pool to the virtual disk.

The NVME drives are assigned the cache role and the HDD's are assigned in the Capacity Tier. I did a benchmark from a VM running in the S2D cluster. On Linux I tested the performance with fio with the following command: "fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=random_read_write.fio --bs=4k --iodepth=64 --size=4G --readwrite=randrw"

The result from the VM inside S2D is:
READ: bw=6889KiB/s (7054kB/s), 6889KiB/s-6889KiB/s (7054kB/s-7054kB/s), io=2049MiB (2149MB), run=304620-304620msec

WRITE: bw=6880KiB/s (7045kB/s), 6880KiB/s-6880KiB/s (7045kB/s-7045kB/s), io=2047MiB (2146MB), run=304620-304620msec

I did the same test on a standalone server with a local storage - PM1725 NVME SSD
Same OS and same command for the benchmark the result:
READ: io=2049.2MB, aggrb=238366KB/s, minb=238366KB/s, maxb=238366KB/s, mint=8803msec, maxt=8803msec
WRITE: io=2046.9MB, aggrb=238096KB/s, minb=238096KB/s, maxb=238096KB/s, mint=8803msec, maxt=8803msec

I destroyed the S2D and disabled the S2D-cluster to confirm the speeds of the NVME drives. I tested all 6 NVME drives one by one and the speeds I get from the direct storage is around the screenshot below.



Next day try, I created a new S2D cluster with no cache and the NVME drives assigned in the Performance Tier and SAS drives in the capacity tier. Again speeds with same slow results. There must be a misconfiguration. I read another topic here about PLP, I did some research and the PM1725 should support the PLP.

Result of Get-StoragePool "S2D on HVCLUSTER" | FL

Code:
PassThroughClass                  :
PassThroughIds                    :
PassThroughNamespace              :
PassThroughServer                 :
AllocatedSize                     : 28101165711360
ClearOnDeallocate                 : False
EnclosureAwareDefault             : False
FaultDomainAwarenessDefault       : StorageScaleUnit
FriendlyName                      : S2D on HVCLUSTER
HealthStatus                      : Healthy
IsClustered                       : True
IsPowerProtected                  : True
IsPrimordial                      : False
IsReadOnly                        : False
LogicalSectorSize                 : 4096
MediaTypeDefault                  : Unspecified
Name                              :
OperationalStatus                 : OK
OtherOperationalStatusDescription :
OtherUsageDescription             : Reserved for S2D
PhysicalSectorSize                : 4096
ProvisioningTypeDefault           : Fixed
ReadOnlyReason                    : None
RepairPolicy                      : Parallel
ResiliencySettingNameDefault      : Mirror
RetireMissingPhysicalDisks        : Never
Size                              : 28982995058688
SupportedProvisioningTypes        : Fixed
SupportsDeduplication             : True
ThinProvisioningAlertThresholds   : {70}
Usage                             : Other
Version                           : Windows Server 2019
WriteCacheSizeDefault             : Auto
WriteCacheSizeMax                 : 18446744073709551614
WriteCacheSizeMin                 : 0
PSComputerName                    :
Get-ClusterStorageSpacesDirect

Code:
CacheMetadataReserveBytes : 34359738368
CacheModeHDD              : ReadWrite
CacheModeSSD              : WriteOnly
CachePageSizeKBytes       : 16
CacheState                : Enabled
Name                      : HVCLUSTER
ScmUse                    : Cache
State                     : Enabled
Get-NetAdapterRdma

Code:
Name                      InterfaceDescription                     Enabled     PFC        ETS
----                      --------------------                     -------     ---        ---
SLOT 3                    Mellanox ConnectX-3 Pro Ethernet Adapter True        False      False
SLOT 3 2                  Mellanox ConnectX-3 Pro Ethernet Adap... True        False      False

Get-StorageJob does not return anything.

I ran the following command, this is returning some errors:
Get-PhysicalDisk | Get-StorageAdvancedProperty
Code:
FriendlyName            SerialNumber                         IsPowerProtected IsDeviceCacheEnabled
------------            ------------                         ---------------- --------------------
MSFT Virtual HD         4B621B92-2B4A-4B0F-90A8-B0EA6AF33E92            False                 True
DELLBOSS VD             5d8224e28f810010                                False                 True
NETAPP X425_SIRMN1T2A10 S3L0Z76F0000M5174T7Z                            False
NETAPP X425_SIRMN1T2A10 S3L0Z9XZ0000M5184N3B                            False
WARNING: Retrieving IsDeviceCacheEnabled failed with ErrorCode 1.
MS1PC5ED3ORA3.2T        0025_3893_6100_08B2.                            False
WARNING: Retrieving IsDeviceCacheEnabled failed with ErrorCode 1.
NETAPP X425_SIRMN1T2A10 S3L0XB4B0000M5168C2Q                            False
WARNING: Retrieving IsDeviceCacheEnabled failed with ErrorCode 1.
NETAPP X425_SIRMN1T2A10 S3L0YPT70000M516EAY7                            False
WARNING: Retrieving IsDeviceCacheEnabled failed with ErrorCode 1.
NETAPP X425_SIRMN1T2A10 S3L0XDRB0000M511CXDN                            False
WARNING: Retrieving IsDeviceCacheEnabled failed with ErrorCode 1.
NETAPP X425_SIRMN1T2A10 S3L0X47S0000M515EUDS                            False
WARNING: Retrieving IsDeviceCacheEnabled failed with ErrorCode 1.
NETAPP X425_SIRMN1T2A10 S3L0XC330000M516AWXK                            False
WARNING: Retrieving IsDeviceCacheEnabled failed with ErrorCode 1.
NETAPP X425_SIRMN1T2A10 S3L0XEG50000M516008L                            False
WARNING: Retrieving IsDeviceCacheEnabled failed with ErrorCode 1.
NETAPP X425_SIRMN1T2A10 S3L0XBYB0000M516AWW7                            False
WARNING: Retrieving IsDeviceCacheEnabled failed with ErrorCode 1.
NETAPP X425_SIRMN1T2A10 S3L0XE400000M515UM96                            False
WARNING: Retrieving IsDeviceCacheEnabled failed with ErrorCode 1.
NETAPP X425_SIRMN1T2A10 S3L0ZAPW0000M51800C9                            False
WARNING: Retrieving IsDeviceCacheEnabled failed with ErrorCode 1.
MS1PC5ED3ORA3.2T        0025_3893_6100_08BF.                            False
WARNING: Retrieving IsDeviceCacheEnabled failed with ErrorCode 1.
NETAPP X425_SIRMN1T2A10 S3L0YNW20000M516AXGE                            False
WARNING: Retrieving IsDeviceCacheEnabled failed with ErrorCode 1.
NETAPP X425_SIRMN1T2A10 S3L0YQ3T0000M515XP6G                            False
WARNING: Retrieving IsDeviceCacheEnabled failed with ErrorCode 1.
MS1PC5ED3ORA3.2T        0025_3894_6100_06D0.                            False
WARNING: Retrieving IsDeviceCacheEnabled failed with ErrorCode 1.
NETAPP X425_SIRMN1T2A10 S3L0YPTA0000M516EB0G                            False
WARNING: Retrieving IsDeviceCacheEnabled failed with ErrorCode 1.
NETAPP X425_SIRMN1T2A10 S3L0YDQP0000M5175H4D                            False
WARNING: Retrieving IsDeviceCacheEnabled failed with ErrorCode 1.
NETAPP X425_SIRMN1T2A10 S3L0Z9XC0000M5184MZ9                            False
WARNING: Retrieving IsDeviceCacheEnabled failed with ErrorCode 1.
NETAPP X425_SIRMN1T2A10 S3L0ZASY0000M5159KWG                            False
WARNING: Retrieving IsDeviceCacheEnabled failed with ErrorCode 1.
NETAPP X425_SIRMN1T2A10 S3L0XAYP0000M515XPHQ                            False
WARNING: Retrieving IsDeviceCacheEnabled failed with ErrorCode 1.
NETAPP X425_SIRMN1T2A10 S3L0XEEF0000M516EAZG                            False
WARNING: Retrieving IsDeviceCacheEnabled failed with ErrorCode 1.
MS1PC5ED3ORA3.2T        0025_3893_6100_08C1.                            False
WARNING: Retrieving IsDeviceCacheEnabled failed with ErrorCode 1.
NETAPP X425_SIRMN1T2A10 S3L0XEH50000M51600B7                            False
WARNING: Retrieving IsDeviceCacheEnabled failed with ErrorCode 1.
MS1PC5ED3ORA3.2T        0025_3894_6100_06CC.                            False
WARNING: Retrieving IsDeviceCacheEnabled failed with ErrorCode 1.
NETAPP X425_SIRMN1T2A10 S3L0XB6T0000M5168CD0                            False
WARNING: Retrieving IsDeviceCacheEnabled failed with ErrorCode 1.
NETAPP X425_SIRMN1T2A10 S3L0XELZ0000M516EAG7                            False
WARNING: Retrieving IsDeviceCacheEnabled failed with ErrorCode 1.
NETAPP X425_SIRMN1T2A10 S3L0XELX0000M516EAKK                            False
WARNING: Retrieving IsDeviceCacheEnabled failed with ErrorCode 1.
MS1PC5ED3ORA3.2T        0025_3894_6100_06DF.                            False
WARNING: Retrieving IsDeviceCacheEnabled failed with ErrorCode 1.
NETAPP X425_SIRMN1T2A10 S3L0ZAT00000M514SAG2                            False
I can confirm the write/read cache is being used from the Windows Admin Centre. However how could it be so slow?

 
Last edited:

Timmerdanny

New Member
Oct 21, 2019
8
0
1
I ran some more benchmarks with crystal disk mark.

Inside the S2D storage pool VM running Windows Server 2019:


After the CrystalDiskMark I ran another fio benchmark, the results are still slow there.
 

TerryPhillips

New Member
May 7, 2019
23
6
3
Set-StoragePool -FriendlyName 'S2D on HVCLUSTER' -IsPowerProtected $true

"Indicates whether the physical disks that belong to the storage pool have backup power systems, such as battery backup. If you specify a value of $True for this parameter, the storage pool does not perform flush operations, and the pool removes write-through attributes from commands."

And yes, a UPS is recommended...
 

Timmerdanny

New Member
Oct 21, 2019
8
0
1
Set-StoragePool -FriendlyName 'S2D on HVCLUSTER' -IsPowerProtected $true

"Indicates whether the physical disks that belong to the storage pool have backup power systems, such as battery backup. If you specify a value of $True for this parameter, the storage pool does not perform flush operations, and the pool removes write-through attributes from commands."

And yes, a UPS is recommended...
I tried that command but the parameter was already set as PLP. The Samsung P1725 supports PLP so it should automaticly detect the right settings. I benchmarked all my NVMe SSD's in the cluster, below you see a screenshot of one of the Cluster nodes performaning a disk benchmark on a local NVMe drive.

The complains for the slowness came from a developer who is trying to upload a csv file to one of the VM that is running CentOS with MariaDB 10.4. The CSV upload action is done from the same machine (localhost), on the cluster with cache enabled we got inserts around 230/s. I created a local datastore of one of the NVMe drives then copied over the same VM on that datastore, then started CSV upload again now inserts speeds are around 800/s but still not comparable with our VMware platform. I'm 100% the cluster / memory / network is not overloaded at the moment of testing. I tried to firmware update the P1725 but that doesn't bring high speeds either.
I even tried to edit the IO scheduler to use noop, by setting the grub flags:
elevator=noop.

We have a single Dell R730 server with the following configuration:
2x 22 core CPU
386GB DDR-4 Memory
1x Samsung PM1725a 3.2TB
1x Mallanox ConnectX-3 Pro

So what I did I converted the VM on our Hyper-V host to VMware format and then started the VM on the VMware host. I did the same tests with the inserts and this time I get inserts around 3400/sec. It's strange because I can't see the clue why the inserts so much faster then on the Hyper-V hosts.

Below the disk benchmark of a local NVMe SSD inside the S2D cluster


Below the disk benchmark of a Windows machine running on our VMware host

 

BloodKnight7

New Member
Nov 15, 2022
23
2
3
Did you ever find a solution to this? We are experiencing an almost identical issue.
Im curious... did any of you applied or the reccomended best practices for the physical switch, created a SET switch for the mgmt/vm traffic and left the storage on the physical mellanox cards having RoCe v2 enabled as well as jumbo frames?