S2D slow write (no parity & NVMe cache)

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

Jeff Robertson

Active Member
Oct 18, 2016
429
115
43
Chico, CA
Your primary source should be the Windows Server Catalog: Windows Server Catalog

For disk drives supporting S2D you need the chriteria Software-Defined Data Center (SDDC) Standard or Software-Defined Data Center (SDDC) Premium. For your purposes Standard is enough:
Windows Server Catalog

The difference between Standard and Premium:
http://download.microsoft.com/downl...ws_Server_2016_Software-defined_Solutions.pdf

Me personally, I like to check geizhals.de, too. They have an english site as well:
Solid State Drives (SSD) with Special features: Power-Loss Protection Skinflint Price Comparison UK

Some of the newer products may not be certified by Microsoft yet, but still provide it and be visible that way.

Hope that helps you.
Wow that is some great information, VERY helpful! I'm a bit irked that this type of info is not more readily available, I even consulted with Eldon when 2016 was just coming out and I don't think even MS really knew what would work properly, I was just told to use enterprise drives. One of my clusters has 8x PM863 drives and sure enough they show IsPowerProtected = False... No wonder I've never been terribly happy with write speed, it works well enough for a couple dozen VMs but I can definitely overwhelm the system.

I have been going through the links you gave me and am having trouble finding any drives that are reasonably priced on eBay that are listed on MS's site. Any suggestions? I would love to get a hold of some M.2 NVMe drives but SATA would work as well. It looks like (correct me if I'm wrong) the PM863 drives don't qualify but the PM863a drives might?

Thanks!
 

Evan

Well-Known Member
Jan 6, 2016
3,346
598
113
Oh, PM863a ... I can check, that’s what I have and planning to use to take another look at S2D.
 

Jeff Robertson

Active Member
Oct 18, 2016
429
115
43
Chico, CA
Oh, PM863a ... I can check, that’s what I have and planning to use to take another look at S2D.
If you get a chance run this power shell command on a system with one of the drives installed to see if it qualifies has having full PLP support:

Get-PhysicalDisk |? FriendlyName -Like "*SAMSUNG*" | Get-StorageAdvancedProperty

I would love to find some inexpensive drives that work properly. Everything I am finding is either over priced or ancient. Don't forget to post your results!
 

Jeff Robertson

Active Member
Oct 18, 2016
429
115
43
Chico, CA
I just ordered an Intel S3510 to replace one of my samsung drives, if it works I'll replace all of them. According to MS it is SDDC premium compatible and it's a good excuse to add larger drives as space is getting tight.
 

Evan

Well-Known Member
Jan 6, 2016
3,346
598
113
@Jeff Robertson did you find an actual list of SSD’s supported, the page from Microsoft with supported config at least for HPE config did not show actual part numbers or models or components.
 

Jeff Robertson

Active Member
Oct 18, 2016
429
115
43
Chico, CA
@Jeff Robertson did you find an actual list of SSD’s supported, the page from Microsoft with supported config at least for HPE config did not show actual part numbers or models or components.
I did, I needed something reasonably priced that was 800GB or larger so I ended up going with an Intel S3510 since there are a ton on ebay. I used psannz link and then just narrowed it down to intel:

Windows Server Catalog

I did just read an article in which they used the samsung pm863a drives so it MAY be that at least some of the "a" versions support the full PLP required for S2D. I just gave up and went with intel because they are easily found using the link above and all of the listed models are both standard and premium compatible. Good luck, if you find any that work post the models!
 

Evan

Well-Known Member
Jan 6, 2016
3,346
598
113
Will test a few next week got 240gb,960gb,1.92tb,3.84tb but not looked what is ‘a’ or not
Some Samsung, some OEM branded.

Got a little bit excited I may be able to use S2D on a small scale but let’s see, can always do storage replica and hyper-v cluster if not.
 

Jeff Robertson

Active Member
Oct 18, 2016
429
115
43
Chico, CA
Ok, I ordered an LSI 9300-8I HBA and an Intel S3510 SATA drive. I checked and both of these are on MS's list as being SDDC certified. When I run get-physicaldisk | get-storageadvancedproperties it shows ispowerprotected as being false... Any thoughts psannz? I've tried the equipment on both a windows 10 workstation and an HP DL380G8 running Server 2016.
 

Cutha

Member
Sep 24, 2016
75
7
8
50
Canada
When I did mine I had to go to device manager and for each disk, under Policy, Enable write caching on the device and turn off Windows write cache buffer. And then it worked like a charm and responded with true. I did this after destroying the cluster and before I started again.

When I look at the drives now both of those boxes are unchecked.
 

Jeff Robertson

Active Member
Oct 18, 2016
429
115
43
Chico, CA
When I did mine I had to go to device manager and for each disk, under Policy, Enable write caching on the device and turn off Windows write cache buffer. And then it worked like a charm and responded with true. I did this after destroying the cluster and before I started again.

When I look at the drives now both of those boxes are unchecked.
Very interesting, I will have to try that. My end goal is to get the highest performance out of the system and I'm concerned that these extra steps need to be taken and that I can't find documentation on it anywhere. If that's all it takes I'm going to order a large batch of drives and start my s2d cluster over.

I was hoping and was under the impression the ispowerprotected flag would automatically be set to true if using the appropriate drives. Ah well.
 

Necrotyr

Active Member
Jun 25, 2017
206
52
28
Denmark
When I did mine I had to go to device manager and for each disk, under Policy, Enable write caching on the device and turn off Windows write cache buffer. And then it worked like a charm and responded with true. I did this after destroying the cluster and before I started again.

When I look at the drives now both of those boxes are unchecked.
AFAIK S2D will uncheck those boxes by itself. I was trying to trick S2D in to thinking that my 850 EVO's were PLP, but S2D saw through that.

Even HGST hussl4010bss600 are not set as powerprotected even though they are...
 

Cutha

Member
Sep 24, 2016
75
7
8
50
Canada
Before I enabled S2D (for the 1000 time) I checked the boxes and this was the result:

Code:
Get-PhysicalDisk |? FriendlyName -Like "*SAMSUNG*" | Get-StorageAdvancedProperty

FriendlyName         SerialNumber   IsPowerProtected IsDeviceCacheEnabled
------------         ------------   ---------------- --------------------
ATA SAMSUNG MZ7WD480 S1G1NYAF926440             True                 True
ATA SAMSUNG MZ7WD480 S16MNYAD911565             True                 True
ATA SAMSUNG MZ7WD480 S16MNYAF605357             True                 True
ATA SAMSUNG MZ7WD480 S16MNYAD911436             True                 True
ATA SAMSUNG MZ7WD480 S16MNEAD740753             True                 True
ATA SAMSUNG MZ7WD480 S16MNEAD500108             True                 True
ATA SAMSUNG MZ7WD480 S16MNYAF604909             True                 True
ATA SAMSUNG MZ7WD480 S1G1NYAF921789             True                 True
ATA SAMSUNG MZ7WD480 S1G1NYAF925500             True                 True
ATA SAMSUNG MZ7WD480 S16MNYAF114484             True                 True
ATA SAMSUNG MZ7WD480 S16MNYAF334083             True                 True
ATA SAMSUNG MZ7WD480 S1G1NYAF921332             True                 True
My writes went from ~50MB/s to ~400MB/s. It's not great but it's livable.

If I can find 8x affordable NVMe (M.2) drives with PLP then I will redo and retest S2D. Anybody want to trade 8x SM961's for 8x NVMe w/PLP???
 

Necrotyr

Active Member
Jun 25, 2017
206
52
28
Denmark
I can tell you that SM953's are not dectected as having PLP and I'm unable to enable the write cache on them.
 

Jeff Robertson

Active Member
Oct 18, 2016
429
115
43
Chico, CA
Well I've been trying to find an M.2 NVMe PLP drive and I came across this:

Welcome to NextWarehouse.com

It's an Intel P4501 1TB. It's about double what I can afford since I need at least 4 of them. But it should work with S2D. I may have to sell a car or just stick with some S3710s.
 

Jeff Robertson

Active Member
Oct 18, 2016
429
115
43
Chico, CA
I tested an Intel S3500 and S3510 on the built in Intel SATA controller and an LSI 9300-8i and both allowed me to enable write caching.
 

Jeff Robertson

Active Member
Oct 18, 2016
429
115
43
Chico, CA
Ok, I have FINALLY managed to get a volume that shows ispowerprotected as being true. In the end I purchased 4 Toshiba HK4 SATA SSDs and put two each in a couple of old desktops I had lying around. I used the latest server 2019 build and created a cluster and enabled S2D. On both machines I enabled caching under policy in device manager. After doing this the drives showed ispowerprotected as being true. The moment I enabled S2D they report ispowerprotected as being false. I was a bit bummed until I ran Get-StoragePool S2D* | fl *. It shows the volume as being power protected!

I believe that means performance should be as good as it can be with this setup (correct me if I'm wrong). A couple things of note, my current S2D setup includes 4x PM863 SSDs in each of the two nodes. With that setup even though the volume isn't power protected performance seems to be similar. It has higher sequential but lower random performance. I would expect adding two additional HK4 drives would bring the new cluster up to the same sequential and even higher random performance. Overall I'm disappointed I didn't see a more noticeable uptick in performance. Maybe my expectations weren't realistic.

I would like to put two NVMe drives in front of the SATA drives to see what that does but I can't find any for a reasonable price. Does anyone know if it's possible to add a pair of NVMe drives at a later date?

Here is a benchmark I ran, left side is the PM863 cluster and right is the HK4 cluster (4 disks per node vs 2):

crystaldisk.JPG
 

Jeff Robertson

Active Member
Oct 18, 2016
429
115
43
Chico, CA
RDMA is tested and working and drives perform as expected on their own. I am waiting until I get this sorted out before I deploy this so at the moment I can do whatever configuring and testing is required. I am stumped and disappointed with myself. I have spent 2 years gathering the hardware to make this work and for the life of me I can't get it to perform. Any help would be appreciated.
Do you have a list of hardware you could post? What is the issue, slow speeds?
 

Cipher

Member
Aug 8, 2014
159
15
18
53
Just catching up on this great thread. I've currently have, or will soon have, the following three types SAS SSD drives from HGST:

1) HUSMM8040ASS205 (Ultrastar SSD800MM)
2) HUSMM8040ASS204 (Ultrastar SSD800MM)
3) HUSSL4040ASS600 (Ultrastar SSD400S)

According to the Windows Server storage engineering team (Don’t do it: consumer-grade solid-state drives (SSD) in Storage Spaces Direct), this is how you identify a drive with a non-volatile buffer cache:

Datasheet, datasheet, datasheet. Look for language like:
  • “Power loss protection” or “PLP”
    • Samsung SM863, and related
    • Toshiba HK4E series, and related
  • “Enhanced power loss data protection”
    • Intel S3510, S3610, S3710, P3700 series, and related
… along with many others across the industry. You should be able to find a device from your favored provider.

For my HGST drives, when I look at their datasheets they contain related text like the following:

1) Advanced power loss data management technology
2) For complete end-to-end data protection and reliability, the Ultrastar SSD800MM family incorporates the T10 Data Integrity Field (DIF) standard, extended error correction code (ECC), Exclusive-OR (XOR) parity to protect against flash die failure, parity-checked internal data paths without an external write cache, and an exclusive power loss data management feature that does not require supercapacitors.
3) Assures data integrity during power failure

However, all my SSD's are showing their IsPowerProtected value as FALSE in Powershell.

I'm running Server 2016, with plans to move to Server 2019 later this year, but I'm hoping I can get S2D up and running on my current hardware which sounds like it supports PLP and should be recognized as supporting this feature.
 

Cipher

Member
Aug 8, 2014
159
15
18
53
Also, I've found the following text from the Storage Spaces Hardware Requirements page:

Systems, components, devices, and drivers must be Windows Server 2016 Certified per the Windows Server Catalog. In addition, we recommend that servers, drives, host bus adapters, and network adapters have the Software-Defined Data Center (SDDC) Standard and/or Software-Defined Data Center (SDDC) Premium additional qualifications (AQs), as pictured below. There are over 1,000 components with the SDDC AQs.

Storage Spaces Direct Hardware Requirements

Therefore, for drives, it looks like we have the following:

1) Required - Windows Server 2016 Certified
2) Recommended - SDDC Standard and/or Premium qualification

I've found my SSD800MM drives in the Windows Catalog (Windows Server Catalog), but they only have Windows Server 2012 certification and not Windows Server 2016 certification. Maybe this is why they are showing a IsPowerProtected value of FALSE.
 

Jeff Robertson

Active Member
Oct 18, 2016
429
115
43
Chico, CA
A couple of observations, someone correct me if I'm wrong. You have to manually "turn on" plp in the windows device manager per drive. You will then get IsPowerProtected = true on a per drive basis. Once you enable S2D the drives will no longer show powerprotected as being true. You can check the volume to see if IsPowerProtected = true on a volume level. I tested this with some Toshiba HK4 drives and on server 2016 the answer was false. On 2019 the volume showed true even though individual drives showed false... I take this to mean the only thing you need to be worried about is the volume level. I am personally waiting for 2019 to rebuild my S2D deployment as it seemed to work properly with that version. Post your build once you are done!