SAS3 HBA Trim Support Without DRAT RZAT Requirement

Discussion in 'RAID Controllers and Host Bus Adapters' started by Ryushin, Nov 17, 2019.

  1. Ryushin

    Ryushin New Member

    Joined:
    Nov 7, 2019
    Messages:
    17
    Likes Received:
    3
    I just upgraded my server to use ZoL .8.x and I filled it with six new Crucial MX500 2TB drives in a raidz2 that are sitting in a Supermicro 36 Drive SAS3 Chassis connected to a LSI 9300-8i (3008 chip) SAS HBA. To my complete surprise, trim does not work. Digging into these threads:

    Can't TRIM Samsung 850 EVO · Issue #8874 · zfsonlinux/zfs
    Topicbox
    Broadcom Inc. | Connecting Everything

    The LSI HBA in IT mode requires DRAT (deterministic read after trim) and RZAT (read zeros after trim) in order for trim to work.

    So the Crucial MX500 SSDs does not support DRAT or RZAT.
    According to hdparm -I:
    hdparm -I /dev/sdn | grep -i trim
    * Data Set Management TRIM supported (limit 8 blocks)

    From what I can see, there is no firmware upgrade for the Crucial MX500s that add support for DRAT or RZAT So the only option left, is a new SAS3 HBA. Can anyone recommend a native Linux kernel supported SAS3 controller that allows trim to happen without needing DRAT and RZAT. I checked Supermicro's site for HBAs and they all seem LSI based.
     
    #1
  2. T_Minus

    T_Minus Moderator

    Joined:
    Feb 15, 2015
    Messages:
    6,882
    Likes Received:
    1,509
    Most of us all run LSI based RAID\HBA cards, but we also run enterprise drives which don't have this TRIM problem. I'm surprised you didni't run into this during your research... everyone saying "enterprise SSD" don't use consumer ;) although it does feel like it's been 2-3 years since this was chanted weekly :D

    WHen I first got going with enterprise gear at home a few years ago I made the same mistake and got 8x NEW in box prosumer intel SSD, very disappointing, then I got enterprise SSD and all smiles :D :D :D
     
    #2
  3. Ryushin

    Ryushin New Member

    Joined:
    Nov 7, 2019
    Messages:
    17
    Likes Received:
    3
    I remember the mantra. The mantra from memory, where for issues like endurance, power off protection, garbage collection, and warranty. The MX500 drives finally met the enterprise specs that I was looking for for a price half of the Samsung 883 DCTs. Considering I needed six 2TB drives to replace 15K platters, the Crucial MX500 met those requirements and still set me back a hefty $1400 for six of them (and I bought another 10 4TB drives to fill the chassis). I did not think there would be issues with trim though and I did not even think of researching that.

    So now that I'm in this boat, I need to solve it, if possible.
     
    #3
  4. Ryushin

    Ryushin New Member

    Joined:
    Nov 7, 2019
    Messages:
    17
    Likes Received:
    3
    Well, the Crucial MX500 drives are on their way back to Amazon. To not support DRAT and RZAT in this day in age is just not good. Going to do research and see if the new WD Red SSDs support proper trim.
     
    #4
  5. T_Minus

    T_Minus Moderator

    Joined:
    Feb 15, 2015
    Messages:
    6,882
    Likes Received:
    1,509
    Manufacturers restrict features for consumer drives so that they cannot \ are not made to run in multi-drive setup \ arrays \ pools for business work loads. No matter what you do with consumer drives you're on that line of "it just may not work".

    WD RED HDD line is limited, I suspect their SSD will be too.

    Always keep in mind they're not going to cannibalize their enterprise market.


    I personally, and have done for dozens and dozens of SSD and NVME is turn to eBay for 2nd hand enterprise gear.
    I think most here would agree.
     
    #5
    itronin likes this.
  6. Ryushin

    Ryushin New Member

    Joined:
    Nov 7, 2019
    Messages:
    17
    Likes Received:
    3
    I don't mind buying some gear from eBay and I just got a temporary CPU from eBay while waiting for Supermicro to release their new Epyc boards. SSDs have only so much endurance, wouldn't buying used SSDs from eBay be risky?

    Are there any SAS3 HBAs that do not have the DRAT and RZAT requirement?
     
    #6
  7. Ryushin

    Ryushin New Member

    Joined:
    Nov 7, 2019
    Messages:
    17
    Likes Received:
    3
    #7
  8. zack$

    zack$ Active Member

    Joined:
    Aug 16, 2018
    Messages:
    233
    Likes Received:
    84
  9. Ryushin

    Ryushin New Member

    Joined:
    Nov 7, 2019
    Messages:
    17
    Likes Received:
    3
    That test is interesting. Hopefully the 2TB 860 Evo won't be like that.

    My end goal is to fully populate a Supermicro 72 drive JBOD for a very cost conscious customer:
    SC417BE1C-R1K23JBOD | 4U | Chassis | Products | Super Micro Computer, Inc.
    with 2TB drives running ZFS with six raidz2 vdevs of 12 drives each and should give us about 100TB of SSD storage. JBOD will be delivered as NAS for SMB and NFS for the purpose of post raw video processing of HD and 4K video.
     
    #9
  10. T_Minus

    T_Minus Moderator

    Joined:
    Feb 15, 2015
    Messages:
    6,882
    Likes Received:
    1,509
    Woah, wait... so this isn't just an 8 drive at home system?

    You're selling 72 consumer drives to a client to use in a huge pool for video processing? :eek:

    This is not going to end well.
     
    #10
  11. Ryushin

    Ryushin New Member

    Joined:
    Nov 7, 2019
    Messages:
    17
    Likes Received:
    3
    The six drives are for my system to do testing. Then we'll buy enough to build one full vdev and do more testing.

    My customer has a ZFS system with 150 spinning platters with 500TB of space in it's pool. The spinning platters cannot keep up with that they are doing. So SSDs are the way to go going forward. A good reason they are still in business is that we do things as cost affective as possible. Tier one post video production houses had no problem dropping 250-500K on a SSD storage solution. Now some of them are no longer in business.

    These may be consumer drives. The warranty and TBW will allow them to be used for their intended purpose. They will provided far more IOPS and speed then the spinning platters. I'm not selling the system to them. They buy the chassis and and drives. It's going to end far better than what they have now.
     
    #11
  12. gea

    gea Well-Known Member

    Joined:
    Dec 31, 2010
    Messages:
    2,287
    Likes Received:
    758
    Videoediting is mostly a sequential workload. While there is no discussion that SSDs have much better iops values, I doubt that you will see an overall faster video filer with desktop Sata SSDs vs 12G SAS mechanical disks in a multi-mirror under load (with a lot of RAM, maybe an Optane L2Arc+read ahead and a special vdev mirror). Trim is bad under load for performance so no replacement for datacenter SSDs. Write performance will go down after a short time of steady write anyway.

    I would prefer for sure a 100TB video filer with up to 24x 10/12TB HE SAS disks ex HGST/WD Ultrastar in a multi mirror setup and a solution without expander.(Regular Supermicro case with 24 x 3,5" disks and expanderless backplane). With expander I would not think of up to 72 sata disks behind.

    About trim
    As ZFS and not the HBA handles trim, i doubt that you get a different result with a different HBA.
     
    #12
    T_Minus likes this.
  13. T_Minus

    T_Minus Moderator

    Joined:
    Feb 15, 2015
    Messages:
    6,882
    Likes Received:
    1,509

    The rated IOPs and TBW is just that, a rating for consumer work load.
    That is not a rating assuming the drives will be in a large pool, be in constant steady-state with severely degraded performance because they are consumer drives not meant for the workload you are looking to accomplish.

    I'd go back to reading and researching before you jump on buying more Drives or HBAs.

    Again, myself, and everyone I've talked to and seen on forums regret going into a large consumer SSD pool setup.
     
    #13
  14. Ryushin

    Ryushin New Member

    Joined:
    Nov 7, 2019
    Messages:
    17
    Likes Received:
    3
    Then maybe it's best to let the deal for the 860 Evo's go for now and see what these new WD SA500 SSDs are like and if they support proper trim behind a LSI card.
     
    #14
  15. Terry Kennedy

    Terry Kennedy Well-Known Member

    Joined:
    Jun 25, 2015
    Messages:
    1,038
    Likes Received:
    487
    This puzzles me. TRIM (and its SAS cousin UNMAP) is the OS telling the drive "I don't need this data any more, go prepare these logical blocks for re-writing with new data at some future time". It would be nice to have the blocks be all 0's, or A5, or DEADBEEF, but unless the OS is reading data it specifically told the drive it didn't care about, I don't see the need for either DRAT or RZAT unless there is a data confidentiality issue (which would be better served by full disk encryption, anyway).

    Can someone enlighten me?
     
    #15
Similar Threads: SAS3 Trim
Forum Title Date
RAID Controllers and Host Bus Adapters 12Gbps SAS3 drives in Dell Perc H710 Mini (SAS2)? Oct 28, 2019
RAID Controllers and Host Bus Adapters SAS3 Backplaine and HBA Upgrade Aug 6, 2019
RAID Controllers and Host Bus Adapters any info on this HP SAS3 expander? Mar 29, 2019
RAID Controllers and Host Bus Adapters LSI card differences? (and a sas3 raid5/6 lsi card) Jan 23, 2019
RAID Controllers and Host Bus Adapters SOLVED: SAS3008 under Freenas detecting SATA drives as 1.5 GBps only Dec 22, 2018

Share This Page