S2D slow write (no parity & NVMe cache)

Discussion in 'Windows Server, Hyper-V Virtualization' started by Cutha, May 22, 2018.

  1. Jeff Robertson

    Jeff Robertson Active Member

    Joined:
    Oct 18, 2016
    Messages:
    245
    Likes Received:
    51
    Wow that is some great information, VERY helpful! I'm a bit irked that this type of info is not more readily available, I even consulted with Eldon when 2016 was just coming out and I don't think even MS really knew what would work properly, I was just told to use enterprise drives. One of my clusters has 8x PM863 drives and sure enough they show IsPowerProtected = False... No wonder I've never been terribly happy with write speed, it works well enough for a couple dozen VMs but I can definitely overwhelm the system.

    I have been going through the links you gave me and am having trouble finding any drives that are reasonably priced on eBay that are listed on MS's site. Any suggestions? I would love to get a hold of some M.2 NVMe drives but SATA would work as well. It looks like (correct me if I'm wrong) the PM863 drives don't qualify but the PM863a drives might?

    Thanks!
     
    #21
  2. Evan

    Evan Well-Known Member

    Joined:
    Jan 6, 2016
    Messages:
    2,304
    Likes Received:
    322
    Oh, PM863a ... I can check, that’s what I have and planning to use to take another look at S2D.
     
    #22
  3. Jeff Robertson

    Jeff Robertson Active Member

    Joined:
    Oct 18, 2016
    Messages:
    245
    Likes Received:
    51
    If you get a chance run this power shell command on a system with one of the drives installed to see if it qualifies has having full PLP support:

    Get-PhysicalDisk |? FriendlyName -Like "*SAMSUNG*" | Get-StorageAdvancedProperty

    I would love to find some inexpensive drives that work properly. Everything I am finding is either over priced or ancient. Don't forget to post your results!
     
    #23
  4. Jeff Robertson

    Jeff Robertson Active Member

    Joined:
    Oct 18, 2016
    Messages:
    245
    Likes Received:
    51
    I just ordered an Intel S3510 to replace one of my samsung drives, if it works I'll replace all of them. According to MS it is SDDC premium compatible and it's a good excuse to add larger drives as space is getting tight.
     
    #24
  5. Evan

    Evan Well-Known Member

    Joined:
    Jan 6, 2016
    Messages:
    2,304
    Likes Received:
    322
    @Jeff Robertson did you find an actual list of SSD’s supported, the page from Microsoft with supported config at least for HPE config did not show actual part numbers or models or components.
     
    #25
  6. Jeff Robertson

    Jeff Robertson Active Member

    Joined:
    Oct 18, 2016
    Messages:
    245
    Likes Received:
    51
    I did, I needed something reasonably priced that was 800GB or larger so I ended up going with an Intel S3510 since there are a ton on ebay. I used psannz link and then just narrowed it down to intel:

    Windows Server Catalog

    I did just read an article in which they used the samsung pm863a drives so it MAY be that at least some of the "a" versions support the full PLP required for S2D. I just gave up and went with intel because they are easily found using the link above and all of the listed models are both standard and premium compatible. Good luck, if you find any that work post the models!
     
    #26
  7. Evan

    Evan Well-Known Member

    Joined:
    Jan 6, 2016
    Messages:
    2,304
    Likes Received:
    322
    Will test a few next week got 240gb,960gb,1.92tb,3.84tb but not looked what is ‘a’ or not
    Some Samsung, some OEM branded.

    Got a little bit excited I may be able to use S2D on a small scale but let’s see, can always do storage replica and hyper-v cluster if not.
     
    #27
  8. Jeff Robertson

    Jeff Robertson Active Member

    Joined:
    Oct 18, 2016
    Messages:
    245
    Likes Received:
    51
    Ok, I ordered an LSI 9300-8I HBA and an Intel S3510 SATA drive. I checked and both of these are on MS's list as being SDDC certified. When I run get-physicaldisk | get-storageadvancedproperties it shows ispowerprotected as being false... Any thoughts psannz? I've tried the equipment on both a windows 10 workstation and an HP DL380G8 running Server 2016.
     
    #28
  9. Cutha

    Cutha Member

    Joined:
    Sep 24, 2016
    Messages:
    52
    Likes Received:
    2
    When I did mine I had to go to device manager and for each disk, under Policy, Enable write caching on the device and turn off Windows write cache buffer. And then it worked like a charm and responded with true. I did this after destroying the cluster and before I started again.

    When I look at the drives now both of those boxes are unchecked.
     
    #29
  10. Jeff Robertson

    Jeff Robertson Active Member

    Joined:
    Oct 18, 2016
    Messages:
    245
    Likes Received:
    51
    Very interesting, I will have to try that. My end goal is to get the highest performance out of the system and I'm concerned that these extra steps need to be taken and that I can't find documentation on it anywhere. If that's all it takes I'm going to order a large batch of drives and start my s2d cluster over.

    I was hoping and was under the impression the ispowerprotected flag would automatically be set to true if using the appropriate drives. Ah well.
     
    #30
  11. Necrotyr

    Necrotyr Member

    Joined:
    Jun 25, 2017
    Messages:
    64
    Likes Received:
    6
    AFAIK S2D will uncheck those boxes by itself. I was trying to trick S2D in to thinking that my 850 EVO's were PLP, but S2D saw through that.

    Even HGST hussl4010bss600 are not set as powerprotected even though they are...
     
    #31
  12. Cutha

    Cutha Member

    Joined:
    Sep 24, 2016
    Messages:
    52
    Likes Received:
    2
    Before I enabled S2D (for the 1000 time) I checked the boxes and this was the result:

    Code:
    Get-PhysicalDisk |? FriendlyName -Like "*SAMSUNG*" | Get-StorageAdvancedProperty
    
    FriendlyName         SerialNumber   IsPowerProtected IsDeviceCacheEnabled
    ------------         ------------   ---------------- --------------------
    ATA SAMSUNG MZ7WD480 S1G1NYAF926440             True                 True
    ATA SAMSUNG MZ7WD480 S16MNYAD911565             True                 True
    ATA SAMSUNG MZ7WD480 S16MNYAF605357             True                 True
    ATA SAMSUNG MZ7WD480 S16MNYAD911436             True                 True
    ATA SAMSUNG MZ7WD480 S16MNEAD740753             True                 True
    ATA SAMSUNG MZ7WD480 S16MNEAD500108             True                 True
    ATA SAMSUNG MZ7WD480 S16MNYAF604909             True                 True
    ATA SAMSUNG MZ7WD480 S1G1NYAF921789             True                 True
    ATA SAMSUNG MZ7WD480 S1G1NYAF925500             True                 True
    ATA SAMSUNG MZ7WD480 S16MNYAF114484             True                 True
    ATA SAMSUNG MZ7WD480 S16MNYAF334083             True                 True
    ATA SAMSUNG MZ7WD480 S1G1NYAF921332             True                 True
    My writes went from ~50MB/s to ~400MB/s. It's not great but it's livable.

    If I can find 8x affordable NVMe (M.2) drives with PLP then I will redo and retest S2D. Anybody want to trade 8x SM961's for 8x NVMe w/PLP???
     
    #32
  13. Necrotyr

    Necrotyr Member

    Joined:
    Jun 25, 2017
    Messages:
    64
    Likes Received:
    6
    I can tell you that SM953's are not dectected as having PLP and I'm unable to enable the write cache on them.
     
    #33
  14. Jeff Robertson

    Jeff Robertson Active Member

    Joined:
    Oct 18, 2016
    Messages:
    245
    Likes Received:
    51
    Well I've been trying to find an M.2 NVMe PLP drive and I came across this:

    Welcome to NextWarehouse.com

    It's an Intel P4501 1TB. It's about double what I can afford since I need at least 4 of them. But it should work with S2D. I may have to sell a car or just stick with some S3710s.
     
    #34
  15. Jeff Robertson

    Jeff Robertson Active Member

    Joined:
    Oct 18, 2016
    Messages:
    245
    Likes Received:
    51
    I tested an Intel S3500 and S3510 on the built in Intel SATA controller and an LSI 9300-8i and both allowed me to enable write caching.
     
    #35
  16. Jeff Robertson

    Jeff Robertson Active Member

    Joined:
    Oct 18, 2016
    Messages:
    245
    Likes Received:
    51
    Ok, I have FINALLY managed to get a volume that shows ispowerprotected as being true. In the end I purchased 4 Toshiba HK4 SATA SSDs and put two each in a couple of old desktops I had lying around. I used the latest server 2019 build and created a cluster and enabled S2D. On both machines I enabled caching under policy in device manager. After doing this the drives showed ispowerprotected as being true. The moment I enabled S2D they report ispowerprotected as being false. I was a bit bummed until I ran Get-StoragePool S2D* | fl *. It shows the volume as being power protected!

    I believe that means performance should be as good as it can be with this setup (correct me if I'm wrong). A couple things of note, my current S2D setup includes 4x PM863 SSDs in each of the two nodes. With that setup even though the volume isn't power protected performance seems to be similar. It has higher sequential but lower random performance. I would expect adding two additional HK4 drives would bring the new cluster up to the same sequential and even higher random performance. Overall I'm disappointed I didn't see a more noticeable uptick in performance. Maybe my expectations weren't realistic.

    I would like to put two NVMe drives in front of the SATA drives to see what that does but I can't find any for a reasonable price. Does anyone know if it's possible to add a pair of NVMe drives at a later date?

    Here is a benchmark I ran, left side is the PM863 cluster and right is the HK4 cluster (4 disks per node vs 2):

    crystaldisk.JPG
     
    #36
  17. Jeff Robertson

    Jeff Robertson Active Member

    Joined:
    Oct 18, 2016
    Messages:
    245
    Likes Received:
    51
    Do you have a list of hardware you could post? What is the issue, slow speeds?
     
    #37
  18. Cipher

    Cipher Member

    Joined:
    Aug 8, 2014
    Messages:
    136
    Likes Received:
    11
    Just catching up on this great thread. I've currently have, or will soon have, the following three types SAS SSD drives from HGST:

    1) HUSMM8040ASS205 (Ultrastar SSD800MM)
    2) HUSMM8040ASS204 (Ultrastar SSD800MM)
    3) HUSSL4040ASS600 (Ultrastar SSD400S)

    According to the Windows Server storage engineering team (Don’t do it: consumer-grade solid-state drives (SSD) in Storage Spaces Direct), this is how you identify a drive with a non-volatile buffer cache:

    Datasheet, datasheet, datasheet. Look for language like:
    • “Power loss protection” or “PLP”
      • Samsung SM863, and related
      • Toshiba HK4E series, and related
    • “Enhanced power loss data protection”
      • Intel S3510, S3610, S3710, P3700 series, and related
    … along with many others across the industry. You should be able to find a device from your favored provider.

    For my HGST drives, when I look at their datasheets they contain related text like the following:

    1) Advanced power loss data management technology
    2) For complete end-to-end data protection and reliability, the Ultrastar SSD800MM family incorporates the T10 Data Integrity Field (DIF) standard, extended error correction code (ECC), Exclusive-OR (XOR) parity to protect against flash die failure, parity-checked internal data paths without an external write cache, and an exclusive power loss data management feature that does not require supercapacitors.
    3) Assures data integrity during power failure

    However, all my SSD's are showing their IsPowerProtected value as FALSE in Powershell.

    I'm running Server 2016, with plans to move to Server 2019 later this year, but I'm hoping I can get S2D up and running on my current hardware which sounds like it supports PLP and should be recognized as supporting this feature.
     
    #38
  19. Cipher

    Cipher Member

    Joined:
    Aug 8, 2014
    Messages:
    136
    Likes Received:
    11
    Also, I've found the following text from the Storage Spaces Hardware Requirements page:

    Systems, components, devices, and drivers must be Windows Server 2016 Certified per the Windows Server Catalog. In addition, we recommend that servers, drives, host bus adapters, and network adapters have the Software-Defined Data Center (SDDC) Standard and/or Software-Defined Data Center (SDDC) Premium additional qualifications (AQs), as pictured below. There are over 1,000 components with the SDDC AQs.

    Storage Spaces Direct Hardware Requirements

    Therefore, for drives, it looks like we have the following:

    1) Required - Windows Server 2016 Certified
    2) Recommended - SDDC Standard and/or Premium qualification

    I've found my SSD800MM drives in the Windows Catalog (Windows Server Catalog), but they only have Windows Server 2012 certification and not Windows Server 2016 certification. Maybe this is why they are showing a IsPowerProtected value of FALSE.
     
    #39
  20. Jeff Robertson

    Jeff Robertson Active Member

    Joined:
    Oct 18, 2016
    Messages:
    245
    Likes Received:
    51
    A couple of observations, someone correct me if I'm wrong. You have to manually "turn on" plp in the windows device manager per drive. You will then get IsPowerProtected = true on a per drive basis. Once you enable S2D the drives will no longer show powerprotected as being true. You can check the volume to see if IsPowerProtected = true on a volume level. I tested this with some Toshiba HK4 drives and on server 2016 the answer was false. On 2019 the volume showed true even though individual drives showed false... I take this to mean the only thing you need to be worried about is the volume level. I am personally waiting for 2019 to rebuild my S2D deployment as it seemed to work properly with that version. Post your build once you are done!
     
    #40
Similar Threads: slow write
Forum Title Date
Windows Server, Hyper-V Virtualization WS 2016 Slow VPN May 23, 2018
Windows Server, Hyper-V Virtualization Simple RAID0/Stripe using Windows Storage Spaces on Windows Server 2016/1709 very slow? Jan 18, 2018
Windows Server, Hyper-V Virtualization Slow LOCALHOST iperf on Windows (on esxi) Aug 19, 2017
Windows Server, Hyper-V Virtualization 2012R2 in colo - slow transfer speed in one direction only? May 12, 2016
Windows Server, Hyper-V Virtualization Slow mirrored tiered storage spaces Aug 25, 2015

Share This Page