Windows Server ACore 2004 and very inconsistent disk performance using Storage Spaces

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

RamGuy

Member
Apr 7, 2015
35
2
8
34
Hi,

I've started to have a lot of issues with my Windows Server ACore 2004 installation. It's running my Plex Media Server and it also handles my torrents. I noticed that the server have started to feel laggy when accessing it using Remote Desktop and transferring files from my Windows 10 2004 desktop is slower than it used to be and both my desktop and my server is featuring a Intel X550-T1 10Gbit NIC with a 10Gbit interconnects on the switches in-between.

The OS installation and all software is running on a Samsung 970 Evo Plus 1TB NVMe drive. All media files, backups etc are running from the following RAID1+0 Storage Space array:

2x Samsung 850 Pro 1TB for write-back cache (100GB cache).
8x Seagate Ironwolf 10TB for data.

Running off a LSI MegaRAID 9211-8i with IT firmware.


This has been working great until lately. All torrents are downloading to the Samsung 970 Evo Plus 1TB and transfer to the Storage Spaces virtual disk after completion. I'm not overviewing the torrent server 24/7 so I have no clue how long this normally takes but today QBittorrent is stuck on "Moving" for hours when trying to transfer 70GB from the NVMe to the Virtual-Disk. This was clearly not the case before.

I use Macrium Reflect for daily automated backups of my desktop and the performance varies from day to day but this is a backup from before moving from Server 2019 to Server ACore 2004:

Server 2019:

Macrium Reflect Log
I/O Performance:Read 10.8 Gb/s - Write 6.5 Gb/s

Server ACore 2004:

Macrium Reflect Log
I/O Performance:Read 5.3 Gb/s - Write 1.0 Gb/s

Yesterday:

Macrium Reflect Log
I/O Performance:Read 4.5 Gb/s - Write 16.4 Mb/s



Judging by the logs performance has taken a hit when moving from Server 2019 to Server ACore 2004. I'm not quite sure what might be causing this, the server is running pretty much the same software and services on it. The hardware is the exact same and the drivers are pretty much the same, some just with a newer version and date as I also grab up-to-date drivers when doing a re-install.

But the current performance is abysmal. From 1.0Gb/s down to 16.4 Mb/s ??


One thing that is different this time around when compared to Server 2019 is how the Intel ProSetACU.exe is not working without Desktop Experience. So with a headless Windows Server I can't run the Intel Proset Tools in order to adjust Receive Buffers, Transfer Buffers etc..

So they are at the default values, I would normally set Receive Buffers and Transfer Buffers to 16384, and Maximum Number of RSS Queues to 16. I'm sure all of this might be done using powershell but I'm not familiar with these commands so all values are at their defaults.


Plex Transcoding is resulting in endless buffering and I don't think it's due to the transcoding itself. I have been able to transcode 4K HDR HEVC files both using hardware (GTX 1050 Ti with modded drives removing the 1x NVENC limitation). Normally it would be capable of transcoding 2-3 4K HDR HEVC streams at the same time. My AMD Ryzen 3900X CPU has been capable transcoding 3-4 4K HDR HEVC streams using software depending on bandwidth per file. I see no reason why the server suddenly can't even transcode 1080P AVC/H264.

So I suspect this is also due to the virutal disk han horrible disk performance.


I'm not entirely sure the best way to really troubleshoot Storage Spaces. I'm not all that familiar with Powershell commands. I have fiddle with a few commands but they all claim everything is healthy. What exactly Microsoft means by healthy I'm not sure, but I find a lot of the numbers provided to look really bad but it might just be me misinterpreting them.


Here are some of my findings:


Get-PhysicalDisk

Number FriendlyName SerialNumber MediaType CanPool OperationalStatus HealthStatus Usage Size
------ ------------ ------------ --------- ------- ----------------- ------------ ----- ----
1 ATA ST10000VN0004-1Z ZA29Z282 HDD False OK Healthy Auto-Select 9.1 TB
2 ATA ST10000VN0004-1Z ZA21DVQ1 HDD False OK Healthy Auto-Select 9.1 TB
6 ATA ST10000VN0004-2G ZCH0DW87 HDD False OK Healthy Auto-Select 9.1 TB
10 NVMe Samsung SSD 970 0025_3855_91B4_3694. SSD False OK Healthy Auto-Select 931.51 GB
4 ATA ST10000VN0004-1Z ZA29ZG3R HDD False OK Healthy Auto-Select 9.1 TB
5 ATA ST10000VN0004-1Z ZA21DKE3 HDD False OK Healthy Auto-Select 9.1 TB
9 Samsung SSD 850 PRO 1TB S3D2NX0J300597Z SSD False OK Healthy Auto-Select 953.87 GB
7 ATA ST10000VN0004-2G ZJV05TDV HDD False OK Healthy Auto-Select 9.1 TB
3 ATA ST10000VN0004-1Z ZA29ZFZC HDD False OK Healthy Auto-Select 9.1 TB
0 ATA ST10000VN0004-1Z ZA29ZT9K HDD False OK Healthy Auto-Select 9.1 TB
8 Samsung SSD 850 PRO 1TB S252NX0H710335R SSD False OK Healthy Auto-Select 953.87 GB


Every disk is reported as healthy, all disks are showing up.


Get-VirtualDisk

FriendlyName ResiliencySettingName FaultDomainRedundancy OperationalStatus HealthStatus Size FootprintOnPool StorageEfficiency
------------ --------------------- --------------------- ----------------- ------------ ---- --------------- -----------------
Seagate-VirtualDisk OK Healthy 37.2 TB 74.59 TB 49,87 %

The virtual disk itself is being reported as healthy.

Get-Volume

DriveLetter FriendlyName FileSystemType DriveType HealthStatus OperationalStatus SizeRemaining Size
----------- ------------ -------------- --------- ------------ ----------------- ------------- ----
NTFS Fixed Healthy OK 84.39 MB 505 MB
FAT32 Fixed Healthy OK 70.55 MB 96 MB
C NTFS Fixed Healthy OK 612.4 GB 930.9 GB
D Seagate-Volume NTFS Fixed Healthy OK 13.86 TB 37.2 TB

The volumes are being reported as healthy.



I can add Get-StorageRealiabilityCounter in order to get some additional information:

Get-PhysicalDisk -FriendlyName 'ATA ST10000VN0004-1Z' | Get-StorageReliabilityCounter | Select *


ObjectId : {1}\\GLAMDRING\root/Microsoft/Windows/Storage/Providers_v2\SPACES_StorageReliabilityCounter.ObjectId="{d035cb0d-a904-11e
a-9031-806e6f6e6963}:RC:{5d03062e-d988-609b-7aa3-60c7ec1239a4}"
PassThroughClass :
PassThroughIds :
PassThroughNamespace :
PassThroughServer :
UniqueId : {d035cb0d-a904-11ea-9031-806e6f6e6963}:RC:{5d03062e-d988-609b-7aa3-60c7ec1239a4}
DeviceId : 5
FlushLatencyMax : 7710
LoadUnloadCycleCount : 21997
LoadUnloadCycleCountMax :
ManufactureDate :
PowerOnHours : 24521
ReadErrorsCorrected : 8
ReadErrorsTotal : 24
ReadErrorsUncorrected : 16
ReadLatencyMax : 28610
StartStopCycleCount : 84
StartStopCycleCountMax :
Temperature : 51
TemperatureMax : 0
Wear : 0
WriteErrorsCorrected :
WriteErrorsTotal :
WriteErrorsUncorrected :
WriteLatencyMax : 3130
PSComputerName :
CimClass : Root/Microsoft/Windows/Storage:MSFT_StorageReliabilityCounter
CimInstanceProperties : {ObjectId, PassThroughClass, PassThroughIds, PassThroughNamespace...}
CimSystemProperties : Microsoft.Management.Infrastructure.CimSystemProperties

ObjectId : {1}\\GLAMDRING\root/Microsoft/Windows/Storage/Providers_v2\SPACES_StorageReliabilityCounter.ObjectId="{d035cb0d-a904-11e
a-9031-806e6f6e6963}:RC:{087ffd2c-5f40-6ea1-b850-87ea07daa09f}"
PassThroughClass :
PassThroughIds :
PassThroughNamespace :
PassThroughServer :
UniqueId : {d035cb0d-a904-11ea-9031-806e6f6e6963}:RC:{087ffd2c-5f40-6ea1-b850-87ea07daa09f}
DeviceId : 2
FlushLatencyMax : 7715
LoadUnloadCycleCount : 22477
LoadUnloadCycleCountMax :
ManufactureDate :
PowerOnHours : 24521
ReadErrorsCorrected : 0
ReadErrorsTotal : 0
ReadErrorsUncorrected : 0
ReadLatencyMax : 8895
StartStopCycleCount : 83
StartStopCycleCountMax :
Temperature : 47
TemperatureMax : 0
Wear : 0
WriteErrorsCorrected :
WriteErrorsTotal :
WriteErrorsUncorrected :
WriteLatencyMax : 2641
PSComputerName :
CimClass : Root/Microsoft/Windows/Storage:MSFT_StorageReliabilityCounter
CimInstanceProperties : {ObjectId, PassThroughClass, PassThroughIds, PassThroughNamespace...}
CimSystemProperties : Microsoft.Management.Infrastructure.CimSystemProperties

ObjectId : {1}\\GLAMDRING\root/Microsoft/Windows/Storage/Providers_v2\SPACES_StorageReliabilityCounter.ObjectId="{d035cb0d-a904-11e
a-9031-806e6f6e6963}:RC:{052d9a5e-a4d3-5ab5-ca67-f35f51298912}"
PassThroughClass :
PassThroughIds :
PassThroughNamespace :
PassThroughServer :
UniqueId : {d035cb0d-a904-11ea-9031-806e6f6e6963}:RC:{052d9a5e-a4d3-5ab5-ca67-f35f51298912}
DeviceId : 1
FlushLatencyMax : 7715
LoadUnloadCycleCount : 15655
LoadUnloadCycleCountMax :
ManufactureDate :
PowerOnHours : 10965
ReadErrorsCorrected : 8
ReadErrorsTotal : 8
ReadErrorsUncorrected : 0
ReadLatencyMax : 8583
StartStopCycleCount : 43
StartStopCycleCountMax :
Temperature : 51
TemperatureMax : 0
Wear : 0
WriteErrorsCorrected :
WriteErrorsTotal :
WriteErrorsUncorrected :
WriteLatencyMax : 3242
PSComputerName :
CimClass : Root/Microsoft/Windows/Storage:MSFT_StorageReliabilityCounter
CimInstanceProperties : {ObjectId, PassThroughClass, PassThroughIds, PassThroughNamespace...}
CimSystemProperties : Microsoft.Management.Infrastructure.CimSystemProperties

ObjectId : {1}\\GLAMDRING\root/Microsoft/Windows/Storage/Providers_v2\SPACES_StorageReliabilityCounter.ObjectId="{d035cb0d-a904-11e
a-9031-806e6f6e6963}:RC:{d5f60916-ff3e-a1a7-c796-23a745ffefbd}"
PassThroughClass :
PassThroughIds :
PassThroughNamespace :
PassThroughServer :
UniqueId : {d035cb0d-a904-11ea-9031-806e6f6e6963}:RC:{d5f60916-ff3e-a1a7-c796-23a745ffefbd}
DeviceId : 0
FlushLatencyMax : 7701
LoadUnloadCycleCount : 15273
LoadUnloadCycleCountMax :
ManufactureDate :
PowerOnHours : 10965
ReadErrorsCorrected : 0
ReadErrorsTotal : 0
ReadErrorsUncorrected : 0
ReadLatencyMax : 28610
StartStopCycleCount : 43
StartStopCycleCountMax :
Temperature : 49
TemperatureMax : 0
Wear : 0
WriteErrorsCorrected :
WriteErrorsTotal :
WriteErrorsUncorrected :
WriteLatencyMax : 2871
PSComputerName :
CimClass : Root/Microsoft/Windows/Storage:MSFT_StorageReliabilityCounter
CimInstanceProperties : {ObjectId, PassThroughClass, PassThroughIds, PassThroughNamespace...}
CimSystemProperties : Microsoft.Management.Infrastructure.CimSystemProperties

ObjectId : {1}\\GLAMDRING\root/Microsoft/Windows/Storage/Providers_v2\SPACES_StorageReliabilityCounter.ObjectId="{d035cb0d-a904-11e
a-9031-806e6f6e6963}:RC:{5c11e10e-4842-25a5-0dfb-9c7a9d90dab1}"
PassThroughClass :
PassThroughIds :
PassThroughNamespace :
PassThroughServer :
UniqueId : {d035cb0d-a904-11ea-9031-806e6f6e6963}:RC:{5c11e10e-4842-25a5-0dfb-9c7a9d90dab1}
DeviceId : 4
FlushLatencyMax : 7705
LoadUnloadCycleCount : 15267
LoadUnloadCycleCountMax :
ManufactureDate :
PowerOnHours : 10965
ReadErrorsCorrected : 0
ReadErrorsTotal : 0
ReadErrorsUncorrected : 0
ReadLatencyMax : 28607
StartStopCycleCount : 43
StartStopCycleCountMax :
Temperature : 49
TemperatureMax : 0
Wear : 0
WriteErrorsCorrected :
WriteErrorsTotal :
WriteErrorsUncorrected :
WriteLatencyMax : 2644
PSComputerName :
CimClass : Root/Microsoft/Windows/Storage:MSFT_StorageReliabilityCounter
CimInstanceProperties : {ObjectId, PassThroughClass, PassThroughIds, PassThroughNamespace...}
CimSystemProperties : Microsoft.Management.Infrastructure.CimSystemProperties

ObjectId : {1}\\GLAMDRING\root/Microsoft/Windows/Storage/Providers_v2\SPACES_StorageReliabilityCounter.ObjectId="{d035cb0d-a904-11e
a-9031-806e6f6e6963}:RC:{a3193f28-fc2e-9199-fa2c-0cff1d341e93}"
PassThroughClass :
PassThroughIds :
PassThroughNamespace :
PassThroughServer :
UniqueId : {d035cb0d-a904-11ea-9031-806e6f6e6963}:RC:{a3193f28-fc2e-9199-fa2c-0cff1d341e93}
DeviceId : 3
FlushLatencyMax : 7705
LoadUnloadCycleCount : 15276
LoadUnloadCycleCountMax :
ManufactureDate :
PowerOnHours : 10965
ReadErrorsCorrected : 0
ReadErrorsTotal : 0
ReadErrorsUncorrected : 0
ReadLatencyMax : 8602
StartStopCycleCount : 43
StartStopCycleCountMax :
Temperature : 51
TemperatureMax : 0
Wear : 0
WriteErrorsCorrected :
WriteErrorsTotal :
WriteErrorsUncorrected :
WriteLatencyMax : 2439
PSComputerName :
CimClass : Root/Microsoft/Windows/Storage:MSFT_StorageReliabilityCounter
CimInstanceProperties : {ObjectId, PassThroughClass, PassThroughIds, PassThroughNamespace...}
CimSystemProperties : Microsoft.Management.Infrastructure.CimSystemProperties


Get-PhysicalDisk -FriendlyName 'NVMe Samsung SSD 970' | Get-StorageReliabilityCounter | Select *


ObjectId : {1}\\GLAMDRING\root/Microsoft/Windows/Storage/Providers_v2\SPACES_StorageReliabilityCounter.ObjectId="{d035cb0d-a904-11e
a-9031-806e6f6e6963}:RC:{343d9c67-7b4a-66e3-d0f4-bc75ac72fac5}"
PassThroughClass :
PassThroughIds :
PassThroughNamespace :
PassThroughServer :
UniqueId : {d035cb0d-a904-11ea-9031-806e6f6e6963}:RC:{343d9c67-7b4a-66e3-d0f4-bc75ac72fac5}
DeviceId : 10
FlushLatencyMax : 70
LoadUnloadCycleCount :
LoadUnloadCycleCountMax :
ManufactureDate :
PowerOnHours :
ReadErrorsCorrected :
ReadErrorsTotal :
ReadErrorsUncorrected :
ReadLatencyMax : 128
StartStopCycleCount :
StartStopCycleCountMax :
Temperature : 48
TemperatureMax : 85
Wear : 0
WriteErrorsCorrected :
WriteErrorsTotal :
WriteErrorsUncorrected :
WriteLatencyMax : 323
PSComputerName :
CimClass : Root/Microsoft/Windows/Storage:MSFT_StorageReliabilityCounter
CimInstanceProperties : {ObjectId, PassThroughClass, PassThroughIds, PassThroughNamespace...}
CimSystemProperties : Microsoft.Management.Infrastructure.CimSystemProperties



Get-PhysicalDisk -FriendlyName 'Samsung SSD 850 PRO 1TB' | Get-StorageReliabilityCounter | Select *


ObjectId : {1}\\GLAMDRING\root/Microsoft/Windows/Storage/Providers_v2\SPACES_StorageReliabilityCounter.ObjectId="{d035cb0d-a904-11e
a-9031-806e6f6e6963}:RC:{60bea741-7528-da56-93b8-83ed6019fa95}"
PassThroughClass :
PassThroughIds :
PassThroughNamespace :
PassThroughServer :
UniqueId : {d035cb0d-a904-11ea-9031-806e6f6e6963}:RC:{60bea741-7528-da56-93b8-83ed6019fa95}
DeviceId : 9
FlushLatencyMax : 0
LoadUnloadCycleCount :
LoadUnloadCycleCountMax :
ManufactureDate :
PowerOnHours : 10962
ReadErrorsCorrected : 0
ReadErrorsTotal : 0
ReadErrorsUncorrected :
ReadLatencyMax : 592
StartStopCycleCount :
StartStopCycleCountMax :
Temperature : 0
TemperatureMax : 0
Wear : 0
WriteErrorsCorrected :
WriteErrorsTotal :
WriteErrorsUncorrected :
WriteLatencyMax : 606
PSComputerName :
CimClass : Root/Microsoft/Windows/Storage:MSFT_StorageReliabilityCounter
CimInstanceProperties : {ObjectId, PassThroughClass, PassThroughIds, PassThroughNamespace...}
CimSystemProperties : Microsoft.Management.Infrastructure.CimSystemProperties

ObjectId : {1}\\GLAMDRING\root/Microsoft/Windows/Storage/Providers_v2\SPACES_StorageReliabilityCounter.ObjectId="{d035cb0d-a904-11e
a-9031-806e6f6e6963}:RC:{e0d68880-1bf5-bdd8-1be9-88043bca92c8}"
PassThroughClass :
PassThroughIds :
PassThroughNamespace :
PassThroughServer :
UniqueId : {d035cb0d-a904-11ea-9031-806e6f6e6963}:RC:{e0d68880-1bf5-bdd8-1be9-88043bca92c8}
DeviceId : 8
FlushLatencyMax : 0
LoadUnloadCycleCount :
LoadUnloadCycleCountMax :
ManufactureDate :
PowerOnHours : 11093
ReadErrorsCorrected : 0
ReadErrorsTotal : 0
ReadErrorsUncorrected :
ReadLatencyMax : 326
StartStopCycleCount :
StartStopCycleCountMax :
Temperature : 0
TemperatureMax : 0
Wear : 0
WriteErrorsCorrected :
WriteErrorsTotal :
WriteErrorsUncorrected :
WriteLatencyMax : 326
PSComputerName :
CimClass : Root/Microsoft/Windows/Storage:MSFT_StorageReliabilityCounter
CimInstanceProperties : {ObjectId, PassThroughClass, PassThroughIds, PassThroughNamespace...}
CimSystemProperties : Microsoft.Management.Infrastructure.CimSystemProperties




Here we can see that some of the disks have some ReadErrorsCorrected, ReadErrorsTotal and even ReadErrorsUncorrected. But again, I'm not sure if I'm supposed to read too much into this? They are marked as "healthy" and I guess this status takes all of this date into consideration? And bad secorts are to be expected on any hard drive isn't it?

The WriteLatencyMax, ReadLatencyMax and FlushLatencyMax looks downright horrific. 7000-29000? Is suppose this is measured in miliseconds? That seems like some horrible numbers, but then again they are "Max" so might be that there are some something happening at times that paints this in a much bader picture than it really is. I don't really know. The numbers vary from disk to disk and is overall really bad on every disk. I find it highly unlikely that all eight drives are having mechanical issues all at once so I guess there is something else at play here causing some insane max latency numbers?




I tried to do some speed tests using diskspd.exe directly on the Server itself.

Here are the results from the virtual disk:

diskspd.exe -c25G -d300 -s -t24 -w40 -o32 -b64k -Sh -L D:\diskspdtmp.dat

Input parameters:

timespan: 1
-------------
duration: 300s
warm up time: 5s
cool down time: 0s
measuring latency
random seed: 0
path: 'D:\diskspdtmp.dat'
think time: 0ms
burst size: 0
software cache disabled
hardware write cache disabled, writethrough on
performing mix test (read/write ratio: 60/40)
block size: 65536
using sequential I/O (stride: 65536)
number of outstanding I/O operations: 32
thread stride size: 0
threads per file: 24
using I/O Completion Ports
IO priority: normal

System information:

computer name: glamdring
start time: 2020/07/12 16:24:03 UTC

Results for timespan 1:
*******************************************************************************

actual test time: 300.01s
thread count: 24
proc count: 24

CPU | Usage | User | Kernel | Idle
-------------------------------------------
0| 5.31%| 2.18%| 3.14%| 94.69%
1| 3.36%| 0.84%| 2.53%| 96.64%
2| 5.84%| 3.14%| 2.70%| 94.16%
3| 1.92%| 1.12%| 0.80%| 98.08%
4| 0.55%| 0.11%| 0.44%| 99.45%
5| 0.56%| 0.10%| 0.46%| 99.44%
6| 0.44%| 0.09%| 0.35%| 99.56%
7| 0.67%| 0.07%| 0.60%| 99.33%
8| 13.94%| 0.73%| 13.20%| 86.06%
9| 1.98%| 0.10%| 1.89%| 98.02%
10| 1.78%| 0.16%| 1.62%| 98.22%
11| 4.62%| 0.08%| 4.54%| 95.38%
12| 0.85%| 0.22%| 0.63%| 99.15%
13| 0.56%| 0.09%| 0.47%| 99.44%
14| 1.43%| 0.76%| 0.67%| 98.57%
15| 0.57%| 0.09%| 0.48%| 99.43%
16| 6.41%| 1.90%| 4.51%| 93.59%
17| 5.97%| 2.33%| 3.64%| 94.03%
18| 0.53%| 0.12%| 0.41%| 99.47%
19| 1.04%| 0.13%| 0.91%| 98.96%
20| 0.46%| 0.12%| 0.34%| 99.54%
21| 0.52%| 0.11%| 0.40%| 99.48%
22| 0.42%| 0.06%| 0.35%| 99.58%
23| 0.49%| 0.13%| 0.36%| 99.51%
-------------------------------------------
avg.| 2.51%| 0.62%| 1.89%| 97.49%

Total IO
thread | bytes | I/Os | MiB/s | I/O per s | AvgLat | LatStdDev | file
-----------------------------------------------------------------------------------------------------
0 | 5244256256 | 80021 | 16.67 | 266.73 | 120.007 | 299.347 | D:\diskspdtmp.dat (25GiB)
1 | 5241044992 | 79972 | 16.66 | 266.57 | 120.076 | 292.386 | D:\diskspdtmp.dat (25GiB)
2 | 8764194816 | 133731 | 27.86 | 445.76 | 70.600 | 296.394 | D:\diskspdtmp.dat (25GiB)
3 | 5054529536 | 77126 | 16.07 | 257.08 | 123.609 | 653.372 | D:\diskspdtmp.dat (25GiB)
4 | 4896194560 | 74710 | 15.56 | 249.03 | 121.726 | 643.628 | D:\diskspdtmp.dat (25GiB)
5 | 5339414528 | 81473 | 16.97 | 271.57 | 117.856 | 422.838 | D:\diskspdtmp.dat (25GiB)
6 | 5302845440 | 80915 | 16.86 | 269.71 | 118.667 | 405.223 | D:\diskspdtmp.dat (25GiB)
7 | 5823397888 | 88858 | 18.51 | 296.19 | 105.227 | 282.365 | D:\diskspdtmp.dat (25GiB)
8 | 4617011200 | 70450 | 14.68 | 234.83 | 121.369 | 798.641 | D:\diskspdtmp.dat (25GiB)
9 | 4471259136 | 68226 | 14.21 | 227.42 | 119.991 | 784.455 | D:\diskspdtmp.dat (25GiB)
10 | 5276434432 | 80512 | 16.77 | 268.37 | 119.281 | 315.140 | D:\diskspdtmp.dat (25GiB)
11 | 5666570240 | 86465 | 18.01 | 288.21 | 111.032 | 299.602 | D:\diskspdtmp.dat (25GiB)
12 | 4945149952 | 75457 | 15.72 | 251.52 | 116.316 | 343.042 | D:\diskspdtmp.dat (25GiB)
13 | 5227872256 | 79771 | 16.62 | 265.90 | 120.376 | 380.980 | D:\diskspdtmp.dat (25GiB)
14 | 5256904704 | 80214 | 16.71 | 267.37 | 119.725 | 312.427 | D:\diskspdtmp.dat (25GiB)
15 | 5326045184 | 81269 | 16.93 | 270.89 | 118.166 | 341.341 | D:\diskspdtmp.dat (25GiB)
16 | 5296685056 | 80821 | 16.84 | 269.40 | 118.822 | 298.261 | D:\diskspdtmp.dat (25GiB)
17 | 5393743872 | 82302 | 17.15 | 274.33 | 116.679 | 392.543 | D:\diskspdtmp.dat (25GiB)
18 | 4715446272 | 71952 | 14.99 | 239.83 | 127.535 | 878.468 | D:\diskspdtmp.dat (25GiB)
19 | 4801822720 | 73270 | 15.26 | 244.23 | 118.798 | 583.786 | D:\diskspdtmp.dat (25GiB)
20 | 5064687616 | 77281 | 16.10 | 257.60 | 120.994 | 613.111 | D:\diskspdtmp.dat (25GiB)
21 | 5289738240 | 80715 | 16.82 | 269.04 | 118.972 | 314.936 | D:\diskspdtmp.dat (25GiB)
22 | 4765122560 | 72710 | 15.15 | 242.36 | 120.090 | 643.618 | D:\diskspdtmp.dat (25GiB)
23 | 5302976512 | 80917 | 16.86 | 269.72 | 118.668 | 300.593 | D:\diskspdtmp.dat (25GiB)
-----------------------------------------------------------------------------------------------------
total: 127083347968 | 1939138 | 403.98 | 6463.66 | 115.376 | 476.441

Read IO
thread | bytes | I/Os | MiB/s | I/O per s | AvgLat | LatStdDev | file
-----------------------------------------------------------------------------------------------------
0 | 3142516736 | 47951 | 9.99 | 159.83 | 95.654 | 333.674 | D:\diskspdtmp.dat (25GiB)
1 | 3146973184 | 48019 | 10.00 | 160.06 | 94.715 | 308.568 | D:\diskspdtmp.dat (25GiB)
2 | 5251072000 | 80125 | 16.69 | 267.08 | 20.054 | 344.718 | D:\diskspdtmp.dat (25GiB)
3 | 3040477184 | 46394 | 9.67 | 154.64 | 104.047 | 825.082 | D:\diskspdtmp.dat (25GiB)
4 | 2939551744 | 44854 | 9.34 | 149.51 | 101.595 | 805.048 | D:\diskspdtmp.dat (25GiB)
5 | 3190685696 | 48686 | 10.14 | 162.28 | 92.058 | 506.305 | D:\diskspdtmp.dat (25GiB)
6 | 3177250816 | 48481 | 10.10 | 161.60 | 93.971 | 487.858 | D:\diskspdtmp.dat (25GiB)
7 | 3485794304 | 53189 | 11.08 | 177.29 | 72.259 | 312.309 | D:\diskspdtmp.dat (25GiB)
8 | 2764767232 | 42187 | 8.79 | 140.62 | 104.928 | 1011.418 | D:\diskspdtmp.dat (25GiB)
9 | 2676490240 | 40840 | 8.51 | 136.13 | 101.556 | 988.906 | D:\diskspdtmp.dat (25GiB)
10 | 3157393408 | 48178 | 10.04 | 160.59 | 94.411 | 351.247 | D:\diskspdtmp.dat (25GiB)
11 | 3388014592 | 51697 | 10.77 | 172.32 | 81.778 | 345.379 | D:\diskspdtmp.dat (25GiB)
12 | 2987589632 | 45587 | 9.50 | 151.95 | 92.880 | 393.361 | D:\diskspdtmp.dat (25GiB)
13 | 3149725696 | 48061 | 10.01 | 160.20 | 95.645 | 443.734 | D:\diskspdtmp.dat (25GiB)
14 | 3159359488 | 48208 | 10.04 | 160.69 | 96.702 | 357.235 | D:\diskspdtmp.dat (25GiB)
15 | 3190947840 | 48690 | 10.14 | 162.30 | 93.921 | 399.606 | D:\diskspdtmp.dat (25GiB)
16 | 3198025728 | 48798 | 10.17 | 162.66 | 94.760 | 331.415 | D:\diskspdtmp.dat (25GiB)
17 | 3246587904 | 49539 | 10.32 | 165.13 | 91.150 | 466.524 | D:\diskspdtmp.dat (25GiB)
18 | 2830499840 | 43190 | 9.00 | 143.96 | 111.118 | 1114.279 | D:\diskspdtmp.dat (25GiB)
19 | 2865168384 | 43719 | 9.11 | 145.73 | 97.728 | 726.947 | D:\diskspdtmp.dat (25GiB)
20 | 3043295232 | 46437 | 9.67 | 154.79 | 100.035 | 768.862 | D:\diskspdtmp.dat (25GiB)
21 | 3168075776 | 48341 | 10.07 | 161.13 | 94.603 | 352.639 | D:\diskspdtmp.dat (25GiB)
22 | 2859073536 | 43626 | 9.09 | 145.42 | 100.182 | 803.989 | D:\diskspdtmp.dat (25GiB)
23 | 3189964800 | 48675 | 10.14 | 162.25 | 94.855 | 334.575 | D:\diskspdtmp.dat (25GiB)
-----------------------------------------------------------------------------------------------------
total: 76249300992 | 1163472 | 242.38 | 3878.16 | 90.110 | 582.969

Write IO
thread | bytes | I/Os | MiB/s | I/O per s | AvgLat | LatStdDev | file
-----------------------------------------------------------------------------------------------------
0 | 2101739520 | 32070 | 6.68 | 106.90 | 156.418 | 234.321 | D:\diskspdtmp.dat (25GiB)
1 | 2094071808 | 31953 | 6.66 | 106.51 | 158.188 | 261.642 | D:\diskspdtmp.dat (25GiB)
2 | 3513122816 | 53606 | 11.17 | 178.68 | 146.151 | 178.930 | D:\diskspdtmp.dat (25GiB)
3 | 2014052352 | 30732 | 6.40 | 102.44 | 153.141 | 205.436 | D:\diskspdtmp.dat (25GiB)
4 | 1956642816 | 29856 | 6.22 | 99.52 | 151.970 | 247.825 | D:\diskspdtmp.dat (25GiB)
5 | 2148728832 | 32787 | 6.83 | 109.29 | 156.164 | 247.341 | D:\diskspdtmp.dat (25GiB)
6 | 2125594624 | 32434 | 6.76 | 108.11 | 155.582 | 227.200 | D:\diskspdtmp.dat (25GiB)
7 | 2337603584 | 35669 | 7.43 | 118.89 | 154.387 | 221.674 | D:\diskspdtmp.dat (25GiB)
8 | 1852243968 | 28263 | 5.89 | 94.21 | 145.909 | 248.882 | D:\diskspdtmp.dat (25GiB)
9 | 1794768896 | 27386 | 5.71 | 91.28 | 147.481 | 270.968 | D:\diskspdtmp.dat (25GiB)
10 | 2119041024 | 32334 | 6.74 | 107.78 | 156.337 | 247.321 | D:\diskspdtmp.dat (25GiB)
11 | 2278555648 | 34768 | 7.24 | 115.89 | 154.530 | 206.626 | D:\diskspdtmp.dat (25GiB)
12 | 1957560320 | 29870 | 6.22 | 99.56 | 152.082 | 242.916 | D:\diskspdtmp.dat (25GiB)
13 | 2078146560 | 31710 | 6.61 | 105.70 | 157.858 | 253.718 | D:\diskspdtmp.dat (25GiB)
14 | 2097545216 | 32006 | 6.67 | 106.68 | 154.403 | 224.531 | D:\diskspdtmp.dat (25GiB)
15 | 2135097344 | 32579 | 6.79 | 108.59 | 154.401 | 223.164 | D:\diskspdtmp.dat (25GiB)
16 | 2098659328 | 32023 | 6.67 | 106.74 | 155.490 | 234.351 | D:\diskspdtmp.dat (25GiB)
17 | 2147155968 | 32763 | 6.83 | 109.21 | 155.279 | 235.619 | D:\diskspdtmp.dat (25GiB)
18 | 1884946432 | 28762 | 5.99 | 95.87 | 152.186 | 255.067 | D:\diskspdtmp.dat (25GiB)
19 | 1936654336 | 29551 | 6.16 | 98.50 | 149.970 | 248.125 | D:\diskspdtmp.dat (25GiB)
20 | 2021392384 | 30844 | 6.43 | 102.81 | 152.550 | 224.027 | D:\diskspdtmp.dat (25GiB)
21 | 2121662464 | 32374 | 6.74 | 107.91 | 155.359 | 243.703 | D:\diskspdtmp.dat (25GiB)
22 | 1906049024 | 29084 | 6.06 | 96.94 | 149.952 | 254.020 | D:\diskspdtmp.dat (25GiB)
23 | 2113011712 | 32242 | 6.72 | 107.47 | 154.618 | 235.845 | D:\diskspdtmp.dat (25GiB)
-----------------------------------------------------------------------------------------------------
total: 50834046976 | 775666 | 161.59 | 2585.50 | 153.275 | 235.201



total:
%-ile | Read (ms) | Write (ms) | Total (ms)
----------------------------------------------
min | 0.151 | 10.726 | 0.151
25th | 42.469 | 112.768 | 59.523
50th | 63.744 | 127.254 | 106.525
75th | 106.970 | 156.255 | 130.253
90th | 132.938 | 216.008 | 171.116
95th | 152.088 | 253.583 | 214.031
99th | 233.017 | 378.066 | 328.471
3-nines | 6570.602 | 5297.244 | 5386.003
4-nines | 13711.668 | 9306.358 | 12189.064
5-nines | 75053.938 | 9310.284 | 74792.031
6-nines | 88810.844 | 9324.811 | 88810.844
7-nines | 89267.703 | 9324.811 | 89267.703
8-nines | 89267.703 | 9324.811 | 89267.703
9-nines | 89267.703 | 9324.811 | 89267.703
max | 89267.703 | 9324.811 | 89267.703



These results doesn't seem all that great considering we are talking 8x disks in a RAID1+0 array. At the same time it doesn't look to be bad enough to cause everything to feel so sluiggsh either?


Here is the test when running it to the OS disk:

.\diskspd.exe -c25G -d300 -s -t24 -w40 -o32 -b64k -Sh -L C:\diskspdtmp.dat
WARNING: target access pattern will not be sequential, consider -si

Command Line: C:\Users\Administrator\Downloads\diskspd\amd64\diskspd.exe -c25G -d300 -s -t24 -w40 -o32 -b64k -Sh -L C:\diskspdtmp.dat

Input parameters:

timespan: 1
-------------
duration: 300s
warm up time: 5s
cool down time: 0s
measuring latency
random seed: 0
path: 'C:\diskspdtmp.dat'
think time: 0ms
burst size: 0
software cache disabled
hardware write cache disabled, writethrough on
performing mix test (read/write ratio: 60/40)
block size: 65536
using sequential I/O (stride: 65536)
number of outstanding I/O operations: 32
thread stride size: 0
threads per file: 24
using I/O Completion Ports
IO priority: normal

System information:

computer name: glamdring
start time: 2020/07/12 16:32:54 UTC

Results for timespan 1:
*******************************************************************************

actual test time: 300.01s
thread count: 24
proc count: 24

CPU | Usage | User | Kernel | Idle
-------------------------------------------
0| 4.42%| 1.29%| 3.13%| 95.58%
1| 2.98%| 1.22%| 1.76%| 97.02%
2| 2.52%| 0.67%| 1.85%| 97.48%
3| 3.38%| 0.70%| 2.67%| 96.63%
4| 2.09%| 0.27%| 1.82%| 97.91%
5| 1.72%| 0.18%| 1.54%| 98.28%
6| 1.66%| 0.19%| 1.46%| 98.34%
7| 2.17%| 0.20%| 1.97%| 97.83%
8| 2.64%| 0.68%| 1.95%| 97.36%
9| 1.54%| 0.19%| 1.34%| 98.46%
10| 1.56%| 0.18%| 1.39%| 98.44%
11| 1.68%| 0.20%| 1.48%| 98.32%
12| 1.68%| 0.20%| 1.48%| 98.32%
13| 1.53%| 0.18%| 1.35%| 98.47%
14| 1.49%| 0.18%| 1.31%| 98.51%
15| 1.64%| 0.18%| 1.46%| 98.36%
16| 3.38%| 0.74%| 2.64%| 96.62%
17| 3.45%| 1.37%| 2.08%| 96.55%
18| 1.91%| 0.13%| 1.79%| 98.09%
19| 1.95%| 0.20%| 1.75%| 98.05%
20| 1.67%| 0.18%| 1.48%| 98.33%
21| 1.64%| 0.22%| 1.41%| 98.36%
22| 1.65%| 0.20%| 1.44%| 98.35%
23| 1.83%| 0.23%| 1.60%| 98.17%
-------------------------------------------
avg.| 2.17%| 0.42%| 1.76%| 97.83%

Total IO
thread | bytes | I/Os | MiB/s | I/O per s | AvgLat | LatStdDev | file
-----------------------------------------------------------------------------------------------------
0 | 18944294912 | 289067 | 60.22 | 963.53 | 33.210 | 9.492 | C:\diskspdtmp.dat (25GiB)
1 | 19133497344 | 291954 | 60.82 | 973.16 | 32.881 | 7.549 | C:\diskspdtmp.dat (25GiB)
2 | 19261947904 | 293914 | 61.23 | 979.69 | 32.663 | 6.864 | C:\diskspdtmp.dat (25GiB)
3 | 19046137856 | 290621 | 60.54 | 968.71 | 33.032 | 9.449 | C:\diskspdtmp.dat (25GiB)
4 | 19333775360 | 295010 | 61.46 | 983.34 | 32.540 | 6.489 | C:\diskspdtmp.dat (25GiB)
5 | 19334889472 | 295027 | 61.46 | 983.40 | 32.539 | 6.498 | C:\diskspdtmp.dat (25GiB)
6 | 19331809280 | 294980 | 61.45 | 983.24 | 32.544 | 6.481 | C:\diskspdtmp.dat (25GiB)
7 | 19324272640 | 294865 | 61.43 | 982.86 | 32.557 | 6.497 | C:\diskspdtmp.dat (25GiB)
8 | 19315556352 | 294732 | 61.40 | 982.42 | 32.571 | 6.515 | C:\diskspdtmp.dat (25GiB)
9 | 19334496256 | 295021 | 61.46 | 983.38 | 32.539 | 6.484 | C:\diskspdtmp.dat (25GiB)
10 | 19331743744 | 294979 | 61.45 | 983.24 | 32.544 | 6.474 | C:\diskspdtmp.dat (25GiB)
11 | 19334365184 | 295019 | 61.46 | 983.37 | 32.539 | 6.482 | C:\diskspdtmp.dat (25GiB)
12 | 19328663552 | 294932 | 61.44 | 983.08 | 32.549 | 6.481 | C:\diskspdtmp.dat (25GiB)
13 | 19332071424 | 294984 | 61.45 | 983.26 | 32.543 | 6.478 | C:\diskspdtmp.dat (25GiB)
14 | 19333054464 | 294999 | 61.46 | 983.31 | 32.542 | 6.478 | C:\diskspdtmp.dat (25GiB)
15 | 19332988928 | 294998 | 61.46 | 983.30 | 32.542 | 6.474 | C:\diskspdtmp.dat (25GiB)
16 | 18971492352 | 289482 | 60.31 | 964.92 | 33.162 | 6.916 | C:\diskspdtmp.dat (25GiB)
17 | 19200933888 | 292983 | 61.04 | 976.59 | 32.766 | 6.585 | C:\diskspdtmp.dat (25GiB)
18 | 19329581056 | 294946 | 61.45 | 983.13 | 32.548 | 6.491 | C:\diskspdtmp.dat (25GiB)
19 | 19334234112 | 295017 | 61.46 | 983.37 | 32.540 | 6.482 | C:\diskspdtmp.dat (25GiB)
20 | 19328925696 | 294936 | 61.44 | 983.10 | 32.549 | 6.501 | C:\diskspdtmp.dat (25GiB)
21 | 19334823936 | 295026 | 61.46 | 983.40 | 32.539 | 6.489 | C:\diskspdtmp.dat (25GiB)
22 | 19334103040 | 295015 | 61.46 | 983.36 | 32.540 | 6.471 | C:\diskspdtmp.dat (25GiB)
23 | 19334234112 | 295017 | 61.46 | 983.37 | 32.540 | 6.471 | C:\diskspdtmp.dat (25GiB)
-----------------------------------------------------------------------------------------------------
total: 462521892864 | 7057524 | 1470.28 | 23524.52 | 32.645 | 6.865

Read IO
thread | bytes | I/Os | MiB/s | I/O per s | AvgLat | LatStdDev | file
-----------------------------------------------------------------------------------------------------
0 | 11381440512 | 173667 | 36.18 | 578.88 | 33.799 | 9.499 | C:\diskspdtmp.dat (25GiB)
1 | 11495931904 | 175414 | 36.54 | 584.70 | 33.462 | 7.435 | C:\diskspdtmp.dat (25GiB)
2 | 11537547264 | 176049 | 36.68 | 586.82 | 33.246 | 6.886 | C:\diskspdtmp.dat (25GiB)
3 | 11423252480 | 174305 | 36.31 | 581.00 | 33.641 | 9.462 | C:\diskspdtmp.dat (25GiB)
4 | 11571757056 | 176571 | 36.78 | 588.56 | 33.072 | 6.509 | C:\diskspdtmp.dat (25GiB)
5 | 11600330752 | 177007 | 36.88 | 590.01 | 33.120 | 6.560 | C:\diskspdtmp.dat (25GiB)
6 | 11605311488 | 177083 | 36.89 | 590.26 | 33.014 | 6.507 | C:\diskspdtmp.dat (25GiB)
7 | 11580342272 | 176702 | 36.81 | 588.99 | 33.126 | 6.549 | C:\diskspdtmp.dat (25GiB)
8 | 11578769408 | 176678 | 36.81 | 588.91 | 33.101 | 6.540 | C:\diskspdtmp.dat (25GiB)
9 | 11595087872 | 176927 | 36.86 | 589.74 | 33.061 | 6.551 | C:\diskspdtmp.dat (25GiB)
10 | 11601772544 | 177029 | 36.88 | 590.08 | 32.962 | 6.551 | C:\diskspdtmp.dat (25GiB)
11 | 11580407808 | 176703 | 36.81 | 589.00 | 32.982 | 6.485 | C:\diskspdtmp.dat (25GiB)
12 | 11617107968 | 177263 | 36.93 | 590.86 | 33.047 | 6.510 | C:\diskspdtmp.dat (25GiB)
13 | 11615338496 | 177236 | 36.92 | 590.77 | 33.059 | 6.493 | C:\diskspdtmp.dat (25GiB)
14 | 11587682304 | 176814 | 36.84 | 589.37 | 33.038 | 6.517 | C:\diskspdtmp.dat (25GiB)
15 | 11580342272 | 176702 | 36.81 | 588.99 | 33.030 | 6.558 | C:\diskspdtmp.dat (25GiB)
16 | 11408769024 | 174084 | 36.27 | 580.27 | 33.758 | 6.973 | C:\diskspdtmp.dat (25GiB)
17 | 11546460160 | 176185 | 36.70 | 587.27 | 33.385 | 6.675 | C:\diskspdtmp.dat (25GiB)
18 | 11577524224 | 176659 | 36.80 | 588.85 | 33.104 | 6.532 | C:\diskspdtmp.dat (25GiB)
19 | 11574181888 | 176608 | 36.79 | 588.68 | 33.055 | 6.537 | C:\diskspdtmp.dat (25GiB)
20 | 11611537408 | 177178 | 36.91 | 590.58 | 33.122 | 6.576 | C:\diskspdtmp.dat (25GiB)
21 | 11586306048 | 176793 | 36.83 | 589.30 | 32.988 | 6.557 | C:\diskspdtmp.dat (25GiB)
22 | 11592728576 | 176891 | 36.85 | 589.62 | 33.036 | 6.515 | C:\diskspdtmp.dat (25GiB)
23 | 11627659264 | 177424 | 36.96 | 591.40 | 32.975 | 6.516 | C:\diskspdtmp.dat (25GiB)
-----------------------------------------------------------------------------------------------------
total: 277477588992 | 4233972 | 882.06 | 14112.90 | 33.173 | 6.902

Write IO
thread | bytes | I/Os | MiB/s | I/O per s | AvgLat | LatStdDev | file
-----------------------------------------------------------------------------------------------------
0 | 7562854400 | 115400 | 24.04 | 384.66 | 32.323 | 9.412 | C:\diskspdtmp.dat (25GiB)
1 | 7637565440 | 116540 | 24.28 | 388.46 | 32.007 | 7.633 | C:\diskspdtmp.dat (25GiB)
2 | 7724400640 | 117865 | 24.55 | 392.87 | 31.791 | 6.737 | C:\diskspdtmp.dat (25GiB)
3 | 7622885376 | 116316 | 24.23 | 387.71 | 32.120 | 9.355 | C:\diskspdtmp.dat (25GiB)
4 | 7762018304 | 118439 | 24.67 | 394.79 | 31.748 | 6.378 | C:\diskspdtmp.dat (25GiB)
5 | 7734558720 | 118020 | 24.59 | 393.39 | 31.667 | 6.304 | C:\diskspdtmp.dat (25GiB)
6 | 7726497792 | 117897 | 24.56 | 392.98 | 31.838 | 6.376 | C:\diskspdtmp.dat (25GiB)
7 | 7743930368 | 118163 | 24.62 | 393.87 | 31.705 | 6.324 | C:\diskspdtmp.dat (25GiB)
8 | 7736786944 | 118054 | 24.59 | 393.50 | 31.778 | 6.396 | C:\diskspdtmp.dat (25GiB)
9 | 7739408384 | 118094 | 24.60 | 393.64 | 31.758 | 6.302 | C:\diskspdtmp.dat (25GiB)
10 | 7729971200 | 117950 | 24.57 | 393.16 | 31.916 | 6.305 | C:\diskspdtmp.dat (25GiB)
11 | 7753957376 | 118316 | 24.65 | 394.38 | 31.879 | 6.421 | C:\diskspdtmp.dat (25GiB)
12 | 7711555584 | 117669 | 24.51 | 392.22 | 31.799 | 6.364 | C:\diskspdtmp.dat (25GiB)
13 | 7716732928 | 117748 | 24.53 | 392.48 | 31.767 | 6.376 | C:\diskspdtmp.dat (25GiB)
14 | 7745372160 | 118185 | 24.62 | 393.94 | 31.799 | 6.347 | C:\diskspdtmp.dat (25GiB)
15 | 7752646656 | 118296 | 24.64 | 394.31 | 31.812 | 6.276 | C:\diskspdtmp.dat (25GiB)
16 | 7562723328 | 115398 | 24.04 | 384.65 | 32.263 | 6.729 | C:\diskspdtmp.dat (25GiB)
17 | 7654473728 | 116798 | 24.33 | 389.32 | 31.832 | 6.333 | C:\diskspdtmp.dat (25GiB)
18 | 7752056832 | 118287 | 24.64 | 394.28 | 31.716 | 6.340 | C:\diskspdtmp.dat (25GiB)
19 | 7760052224 | 118409 | 24.67 | 394.69 | 31.771 | 6.321 | C:\diskspdtmp.dat (25GiB)
20 | 7717388288 | 117758 | 24.53 | 392.52 | 31.686 | 6.290 | C:\diskspdtmp.dat (25GiB)
21 | 7748517888 | 118233 | 24.63 | 394.10 | 31.867 | 6.327 | C:\diskspdtmp.dat (25GiB)
22 | 7741374464 | 118124 | 24.61 | 393.74 | 31.797 | 6.332 | C:\diskspdtmp.dat (25GiB)
23 | 7706574848 | 117593 | 24.50 | 391.97 | 31.883 | 6.345 | C:\diskspdtmp.dat (25GiB)
-----------------------------------------------------------------------------------------------------
total: 185044303872 | 2823552 | 588.23 | 9411.61 | 31.854 | 6.732



total:
%-ile | Read (ms) | Write (ms) | Total (ms)
----------------------------------------------
min | 15.294 | 14.127 | 14.127
25th | 29.785 | 28.592 | 29.270
50th | 31.839 | 30.532 | 31.321
75th | 34.431 | 33.005 | 33.909
90th | 37.681 | 36.128 | 37.150
95th | 41.743 | 39.970 | 41.164
99th | 66.078 | 64.348 | 65.413
3-nines | 87.690 | 84.878 | 86.730
4-nines | 170.048 | 169.880 | 169.908
5-nines | 263.539 | 267.918 | 266.118
6-nines | 337.594 | 341.475 | 340.676
7-nines | 345.963 | 342.474 | 345.963
8-nines | 345.963 | 342.474 | 345.963
9-nines | 345.963 | 342.474 | 345.963
max | 345.963 | 342.474 | 345.963



I also tested using diskspd.exe from my Server to my desktop. I figured this would be a nice test to see how the throughput using the Intel X550-1T was working out:

diskspd.exe -c25G -d300 -s -t24 -w40 -o32 -b64k -Sh -L Z:\diskspdtmp.dat

Input parameters:

timespan: 1
-------------
duration: 300s
warm up time: 5s
cool down time: 0s
measuring latency
random seed: 0
path: 'Z:\diskspdtmp.dat'
think time: 0ms
burst size: 0
software cache disabled
hardware write cache disabled, writethrough on
performing mix test (read/write ratio: 60/40)
block size: 65536
using sequential I/O (stride: 65536)
number of outstanding I/O operations: 32
thread stride size: 0
threads per file: 24
using I/O Completion Ports
IO priority: normal

System information:

computer name: glamdring
start time: 2020/07/12 17:21:11 UTC

Results for timespan 1:
*******************************************************************************

actual test time: 300.01s
thread count: 24
proc count: 24

CPU | Usage | User | Kernel | Idle
-------------------------------------------
0| 5.98%| 1.00%| 4.98%| 94.02%
1| 3.31%| 0.94%| 2.36%| 96.69%
2| 2.13%| 0.66%| 1.47%| 97.87%
3| 1.91%| 0.59%| 1.31%| 98.09%
4| 98.68%| 0.14%| 98.54%| 1.32%
5| 1.20%| 0.15%| 1.06%| 98.80%
6| 1.06%| 0.09%| 0.96%| 98.94%
7| 1.49%| 0.15%| 1.34%| 98.51%
8| 3.04%| 0.53%| 2.51%| 96.96%
9| 1.39%| 0.13%| 1.26%| 98.61%
10| 1.03%| 0.18%| 0.85%| 98.97%
11| 1.11%| 0.14%| 0.97%| 98.89%
12| 1.72%| 0.11%| 1.61%| 98.28%
13| 1.22%| 0.18%| 1.04%| 98.78%
14| 1.23%| 0.15%| 1.08%| 98.77%
15| 1.11%| 0.17%| 0.94%| 98.89%
16| 3.26%| 1.17%| 2.09%| 96.74%
17| 2.81%| 0.55%| 2.26%| 97.19%
18| 1.10%| 0.12%| 0.98%| 98.90%
19| 1.34%| 0.18%| 1.16%| 98.66%
20| 0.90%| 0.11%| 0.79%| 99.10%
21| 1.01%| 0.15%| 0.86%| 98.99%
22| 1.10%| 0.12%| 0.98%| 98.90%
23| 0.94%| 0.08%| 0.86%| 99.06%
-------------------------------------------
avg.| 5.84%| 0.32%| 5.51%| 94.16%

Total IO
thread | bytes | I/Os | MiB/s | I/O per s | AvgLat | LatStdDev | file
-----------------------------------------------------------------------------------------------------
0 | 12442730496 | 189861 | 39.55 | 632.84 | 50.563 | 29.258 | Z:\diskspdtmp.dat (25GiB)
1 | 12353929216 | 188506 | 39.27 | 628.33 | 50.927 | 28.197 | Z:\diskspdtmp.dat (25GiB)
2 | 12426608640 | 189615 | 39.50 | 632.02 | 50.622 | 28.231 | Z:\diskspdtmp.dat (25GiB)
3 | 12428574720 | 189645 | 39.51 | 632.12 | 50.620 | 28.527 | Z:\diskspdtmp.dat (25GiB)
4 | 11848318976 | 180791 | 37.66 | 602.61 | 53.099 | 39.240 | Z:\diskspdtmp.dat (25GiB)
5 | 12468551680 | 190255 | 39.63 | 634.16 | 50.458 | 27.878 | Z:\diskspdtmp.dat (25GiB)
6 | 12455444480 | 190055 | 39.59 | 633.49 | 50.511 | 28.208 | Z:\diskspdtmp.dat (25GiB)
7 | 12456034304 | 190064 | 39.60 | 633.52 | 50.510 | 28.596 | Z:\diskspdtmp.dat (25GiB)
8 | 12457541632 | 190087 | 39.60 | 633.60 | 50.504 | 28.093 | Z:\diskspdtmp.dat (25GiB)
9 | 12492079104 | 190614 | 39.71 | 635.35 | 50.363 | 27.783 | Z:\diskspdtmp.dat (25GiB)
10 | 12481200128 | 190448 | 39.68 | 634.80 | 50.409 | 27.295 | Z:\diskspdtmp.dat (25GiB)
11 | 12445548544 | 189904 | 39.56 | 632.99 | 50.552 | 28.272 | Z:\diskspdtmp.dat (25GiB)
12 | 12435718144 | 189754 | 39.53 | 632.49 | 50.593 | 28.720 | Z:\diskspdtmp.dat (25GiB)
13 | 12421431296 | 189536 | 39.49 | 631.76 | 50.652 | 29.097 | Z:\diskspdtmp.dat (25GiB)
14 | 12486574080 | 190530 | 39.69 | 635.07 | 50.386 | 28.046 | Z:\diskspdtmp.dat (25GiB)
15 | 12465471488 | 190208 | 39.63 | 634.00 | 50.472 | 28.217 | Z:\diskspdtmp.dat (25GiB)
16 | 12465078272 | 190202 | 39.62 | 633.98 | 50.473 | 28.446 | Z:\diskspdtmp.dat (25GiB)
17 | 12470845440 | 190290 | 39.64 | 634.27 | 50.450 | 27.828 | Z:\diskspdtmp.dat (25GiB)
18 | 12463439872 | 190177 | 39.62 | 633.90 | 50.478 | 28.221 | Z:\diskspdtmp.dat (25GiB)
19 | 12456886272 | 190077 | 39.60 | 633.56 | 50.505 | 28.841 | Z:\diskspdtmp.dat (25GiB)
20 | 12480217088 | 190433 | 39.67 | 634.75 | 50.413 | 27.914 | Z:\diskspdtmp.dat (25GiB)
21 | 12475826176 | 190366 | 39.66 | 634.53 | 50.428 | 27.980 | Z:\diskspdtmp.dat (25GiB)
22 | 12468355072 | 190252 | 39.63 | 634.15 | 50.460 | 28.144 | Z:\diskspdtmp.dat (25GiB)
23 | 12454526976 | 190041 | 39.59 | 633.44 | 50.516 | 28.129 | Z:\diskspdtmp.dat (25GiB)
-----------------------------------------------------------------------------------------------------
total: 298300932096 | 4551711 | 948.23 | 15171.75 | 50.618 | 28.781

Read IO
thread | bytes | I/Os | MiB/s | I/O per s | AvgLat | LatStdDev | file
-----------------------------------------------------------------------------------------------------
0 | 7471104000 | 114000 | 23.75 | 379.98 | 51.115 | 29.923 | Z:\diskspdtmp.dat (25GiB)
1 | 7428112384 | 113344 | 23.61 | 377.80 | 51.326 | 28.065 | Z:\diskspdtmp.dat (25GiB)
2 | 7439908864 | 113524 | 23.65 | 378.40 | 51.100 | 28.168 | Z:\diskspdtmp.dat (25GiB)
3 | 7455178752 | 113757 | 23.70 | 379.17 | 51.078 | 28.674 | Z:\diskspdtmp.dat (25GiB)
4 | 7088111616 | 108156 | 22.53 | 360.51 | 53.572 | 39.368 | Z:\diskspdtmp.dat (25GiB)
5 | 7483883520 | 114195 | 23.79 | 380.63 | 50.939 | 27.921 | Z:\diskspdtmp.dat (25GiB)
6 | 7479492608 | 114128 | 23.78 | 380.41 | 51.014 | 28.686 | Z:\diskspdtmp.dat (25GiB)
7 | 7461797888 | 113858 | 23.72 | 379.51 | 51.006 | 28.942 | Z:\diskspdtmp.dat (25GiB)
8 | 7465205760 | 113910 | 23.73 | 379.68 | 50.836 | 28.332 | Z:\diskspdtmp.dat (25GiB)
9 | 7493844992 | 114347 | 23.82 | 381.14 | 50.732 | 27.157 | Z:\diskspdtmp.dat (25GiB)
10 | 7479558144 | 114129 | 23.78 | 380.41 | 50.814 | 27.439 | Z:\diskspdtmp.dat (25GiB)
11 | 7444299776 | 113591 | 23.66 | 378.62 | 51.028 | 28.321 | Z:\diskspdtmp.dat (25GiB)
12 | 7484211200 | 114200 | 23.79 | 380.65 | 50.973 | 28.383 | Z:\diskspdtmp.dat (25GiB)
13 | 7464550400 | 113900 | 23.73 | 379.65 | 51.072 | 29.730 | Z:\diskspdtmp.dat (25GiB)
14 | 7483359232 | 114187 | 23.79 | 380.61 | 50.741 | 27.595 | Z:\diskspdtmp.dat (25GiB)
15 | 7473332224 | 114034 | 23.76 | 380.10 | 50.918 | 28.334 | Z:\diskspdtmp.dat (25GiB)
16 | 7509573632 | 114587 | 23.87 | 381.94 | 50.987 | 29.079 | Z:\diskspdtmp.dat (25GiB)
17 | 7494172672 | 114352 | 23.82 | 381.16 | 50.870 | 28.010 | Z:\diskspdtmp.dat (25GiB)
18 | 7472480256 | 114021 | 23.75 | 380.05 | 50.972 | 28.682 | Z:\diskspdtmp.dat (25GiB)
19 | 7449542656 | 113671 | 23.68 | 378.89 | 50.941 | 28.876 | Z:\diskspdtmp.dat (25GiB)
20 | 7496990720 | 114395 | 23.83 | 381.30 | 50.852 | 27.913 | Z:\diskspdtmp.dat (25GiB)
21 | 7475494912 | 114067 | 23.76 | 380.21 | 51.028 | 29.012 | Z:\diskspdtmp.dat (25GiB)
22 | 7474839552 | 114057 | 23.76 | 380.17 | 50.928 | 28.274 | Z:\diskspdtmp.dat (25GiB)
23 | 7503085568 | 114488 | 23.85 | 381.61 | 50.935 | 28.203 | Z:\diskspdtmp.dat (25GiB)
-----------------------------------------------------------------------------------------------------
total: 178972131328 | 2730898 | 568.91 | 9102.62 | 51.068 | 28.946

Write IO
thread | bytes | I/Os | MiB/s | I/O per s | AvgLat | LatStdDev | file
-----------------------------------------------------------------------------------------------------
0 | 4971626496 | 75861 | 15.80 | 252.86 | 49.735 | 28.209 | Z:\diskspdtmp.dat (25GiB)
1 | 4925816832 | 75162 | 15.66 | 250.53 | 50.325 | 28.386 | Z:\diskspdtmp.dat (25GiB)
2 | 4986699776 | 76091 | 15.85 | 253.63 | 49.909 | 28.309 | Z:\diskspdtmp.dat (25GiB)
3 | 4973395968 | 75888 | 15.81 | 252.95 | 49.934 | 28.293 | Z:\diskspdtmp.dat (25GiB)
4 | 4760207360 | 72635 | 15.13 | 242.11 | 52.395 | 39.037 | Z:\diskspdtmp.dat (25GiB)
5 | 4984668160 | 76060 | 15.85 | 253.52 | 49.737 | 27.799 | Z:\diskspdtmp.dat (25GiB)
6 | 4975951872 | 75927 | 15.82 | 253.08 | 49.755 | 27.457 | Z:\diskspdtmp.dat (25GiB)
7 | 4994236416 | 76206 | 15.88 | 254.01 | 49.769 | 28.055 | Z:\diskspdtmp.dat (25GiB)
8 | 4992335872 | 76177 | 15.87 | 253.91 | 50.007 | 27.724 | Z:\diskspdtmp.dat (25GiB)
9 | 4998234112 | 76267 | 15.89 | 254.21 | 49.810 | 28.687 | Z:\diskspdtmp.dat (25GiB)
10 | 5001641984 | 76319 | 15.90 | 254.39 | 49.803 | 27.068 | Z:\diskspdtmp.dat (25GiB)
11 | 5001248768 | 76313 | 15.90 | 254.37 | 49.843 | 28.183 | Z:\diskspdtmp.dat (25GiB)
12 | 4951506944 | 75554 | 15.74 | 251.84 | 50.018 | 29.211 | Z:\diskspdtmp.dat (25GiB)
13 | 4956880896 | 75636 | 15.76 | 252.11 | 50.019 | 28.105 | Z:\diskspdtmp.dat (25GiB)
14 | 5003214848 | 76343 | 15.90 | 254.47 | 49.854 | 28.698 | Z:\diskspdtmp.dat (25GiB)
15 | 4992139264 | 76174 | 15.87 | 253.90 | 49.806 | 28.026 | Z:\diskspdtmp.dat (25GiB)
16 | 4955504640 | 75615 | 15.75 | 252.04 | 49.695 | 27.440 | Z:\diskspdtmp.dat (25GiB)
17 | 4976672768 | 75938 | 15.82 | 253.12 | 49.818 | 27.541 | Z:\diskspdtmp.dat (25GiB)
18 | 4990959616 | 76156 | 15.87 | 253.84 | 49.739 | 27.501 | Z:\diskspdtmp.dat (25GiB)
19 | 5007343616 | 76406 | 15.92 | 254.68 | 49.856 | 28.776 | Z:\diskspdtmp.dat (25GiB)
20 | 4983226368 | 76038 | 15.84 | 253.45 | 49.752 | 27.903 | Z:\diskspdtmp.dat (25GiB)
21 | 5000331264 | 76299 | 15.89 | 254.32 | 49.532 | 26.336 | Z:\diskspdtmp.dat (25GiB)
22 | 4993515520 | 76195 | 15.87 | 253.97 | 49.760 | 27.933 | Z:\diskspdtmp.dat (25GiB)
23 | 4951441408 | 75553 | 15.74 | 251.83 | 49.882 | 28.006 | Z:\diskspdtmp.dat (25GiB)
-----------------------------------------------------------------------------------------------------
total: 119328800768 | 1820813 | 379.32 | 6069.13 | 49.943 | 28.518



total:
%-ile | Read (ms) | Write (ms) | Total (ms)
----------------------------------------------
min | 4.687 | 4.205 | 4.205
25th | 39.105 | 38.067 | 38.685
50th | 47.278 | 46.199 | 46.851
75th | 57.710 | 56.586 | 57.270
90th | 69.659 | 68.510 | 69.210
95th | 78.581 | 77.388 | 78.110
99th | 103.492 | 102.213 | 102.985
3-nines | 431.535 | 426.310 | 429.629
4-nines | 765.484 | 763.631 | 764.830
5-nines | 808.527 | 806.211 | 807.573
6-nines | 818.972 | 815.348 | 818.366
7-nines | 819.260 | 819.090 | 819.260
8-nines | 819.260 | 819.090 | 819.260
9-nines | 819.260 | 819.090 | 819.260
max | 819.260 | 819.090 | 819.260




I'm not entirely sure where to move from here. Do you find anything suspecious with any of the data provided? What would be your recommended next step in order to further troubleshoot and try to solve this?


Best regards,
RamGuy
 

j_h_o

Active Member
Apr 21, 2015
644
179
43
California, US
I had a ton of problems with 2019 SMB performance, also on Intel NICs. (And I also observed poor media playback performance)

What happens if you apply the RSS tuning parameters from this thread?

Run this Power-Shell commands with "run as Administrator" an all 2019 Server HVs and VMs.

"NGIS NO CUBIC & MORE SCRIPT"

Powershell
#RUN ONLY ON NATIVE2019, HV2019, VM2019
#Change the TCP-Stack-Settings back to state of Server 2016
#Get-NetTCPSetting | ft -AutoSize
Set-NetTCPSetting -SettingName "InternetCustom" -CongestionProvider CTCP
Set-NetTCPSetting -SettingName "InternetCustom" -DelayedAckTimeoutMs 50
Set-NetTCPSetting -SettingName "InternetCustom" -ForceWS Disabled

Set-NetTCPSetting -SettingName "DatacenterCustom" -CongestionProvider DCTCP
Set-NetTCPSetting -SettingName "DatacenterCustom" -CwndRestart True
Set-NetTCPSetting -SettingName "DatacenterCustom" -ForceWS Disabled

Set-NetTCPSetting -SettingName "Compat" -ForceWS Disabled

Set-NetTCPSetting -SettingName "Datacenter" -CongestionProvider DCTCP
Set-NetTCPSetting -SettingName "Datacenter" -CwndRestart True
Set-NetTCPSetting -SettingName "Datacenter" -ForceWS Disabled

Set-NetTCPSetting -SettingName "Internet" -CongestionProvider CTCP
Set-NetTCPSetting -SettingName "Internet" -DelayedAckTimeoutMs 50
Set-NetTCPSetting -SettingName "Internet" -ForceWS Disabled

#RUN ONLY ON NATIVE2019, HV2019, VM2019
#Disable RSS & RSC on the TCP-Stack
#netsh int tcp show global
netsh int tcp set global RSS=Disabled
netsh int tcp set global RSC=Disabled

#RUN ONLY ON HV2019
#Disable Software RSC on all vSwitches
Get-VMSwitch | Set-VMSwitch -EnableSoftwareRsc $false

Set-VMNetworkAdapter <vm-name> -VrssEnabled $FALSE
#This one disable VRSS on all VMs.
Get-VM | Set-VMNetworkAdapter -VrssEnabled $FALSE
 

RamGuy

Member
Apr 7, 2015
35
2
8
34
Thanks for your reply, I will try this later. I'm doing a full "Repair-Volume -FileSystemLabel "*" -OfflineScanAndFix" which will take some time as it has to scan eight 10TB hard drives.

I think one of the drives might be struggling. According to CrystalDiskInfo it has over 39000 reallocated sectors, and when looking in the event logs there are tons of: " The device, \Device\Harddisk7\DR12, has a bad block. ".

I find it baffling how all the various outputs displays everything as "Healthy". I'm no HDD expert, but so many reallocated sectors can't be a good thing? CrystalDiskInfo marks the disk as bad, and it markes three of the others has "caution" as they have 8 reallocated sectors.

There is a big leap from 8 to 39000 so I'm not sure if I need to worry about those other three. But I supposed that the one that is throwing bad sectors left and right has to go.

These drives are only 1,5 years old. I find it odd that 4 out of 8 drives are already showing signs of issues? They replaced WD RE-4 2TB drives which was used for 6 years+ without any hitch whatsoever.


As I'm running a RAID1+0 array. I'm supposed to be able to remove more than one drive without loosing data, but how would I know which one can safely be removed without loosing data? How do I go by and figure out which of these eight drives is serving as parity and which ones are doing the stripping? Or should I simply go by and replace one, repair, replace another one, repair..? Will take forever, but I suppose it's safer.