Recommended SSDs today?

[cyberjock]

In 2021 and 2022 I was able to find a few SSDs that met my requirements and were reasonably priced as used, so I bought them. In particular, the Sandisk LT1600MO. I had purchased one or two others, but they aren't in my system, so I can't easily look up what those were. My LT1600MO works great for my Proxmox cluster, and I was hoping to buy more since I want to setup a second cluster. The SSDs I used checked all of the boxes that I was concerned about:

2. 12Gb SAS preferred, but 6Gb SAS or SATA is an option
3. Firmware was obtainable and flashable myself (this didn't necessarily mean I had to be able to download from the manufacturer, this forum has a lot of firmwares available)
4. Supported trim/unmap
5. Good write endurance
6. Minimum of 1TB of space per drive, and ideally 1.5TB or larger.
7. Can buy 10+ of them relatively easily (ideally from the same source).

I really can't seem to find any SSD that matches all of this. I see DWPD regularly at 3 (or less) for SSDs billed as "write heavy" and such, and that is pretty disappointing. I realize that for my specific use-case, 3DWPD is *probably* going to be just fine.

Right now I feel like my "best" option is Samsung PM883, but:

1. It's not 12Gb SAS
2. write endurance is only 1.3 DWPD, and only for 3 years. This seems to be terrible IMO.

I know I should be able to overprovision them to something like 1.6TB and extend their lifespan, I'd really prefer to get a drive that is already designed properly.

What are people using for situations like this? I've been watching for the Sandisk LT1600MO's for a few months, and I never see more than 2 or 3 at a time on eBay, and they are often ridiculously priced.

So what is everyone using these days? I cannot seem to really find any SSD that checks all of those boxes.

Thanks!

 

[ano]

pm883 endurance is not an issue, really but you do get sm883 if you want to pay

sas drives are costly theese days

nvme is cheap

 

[Tech Junky]

@cyberjock

KCD8XRUG15T3 - This is what I'm currently using after blowing through a couple of Micron's

Hard Drives | Wiredzone

wiredzone.com

This will give you an idea of what's out there and ability to compare the options and get model numbers for the hunt.

If you go with more capacity your DWPD can drop down.... If you venture into U.x territory you can get the speed and capacity you gain with M2's but at a 50% lower price typically when you get into the 8TB+ range. Also, Oculink is the least bottlenecked option right now.

 

[ano]

CD8's are superb

 

[i386]

optane!

I see DWPD regularly at 3 (or less) for SSDs billed as "write heavy" and such, and that is pretty disappointing. I realize that for my specific use-case, 3DWPD is *probably* going to be just fine.

I find dwpd a useless metric without the drive size.
10dwpd on a 200gb ssd is ~2tb, 3dwpd on a 1tb ssd is 3tb, 0.1dwpd on a 30tb ssd is 3tb...

 

[ano]

DWPD and TBW is also useless, really because depending on io pattern, size of IO etc, wear is less

 

[Tech Junky]

CD8's are superb

Yeah, i have no complaints after several months now. My initial drive was to consolidate 5 spinners into some sort of SSD setup instead due to the aging rust. My first thought was to build an M2 based setup but, the issue there was M2's cap out at 8TB and for the same price I could get the U drives with 16TB. The M2 approach to keep the costs down meant going with 4*4TB and put them in a raid to match my rust configuration somewhat while downsizing a bit. The whole upgrade process though revealed an underlying issue with my Intel setup that forced my hand to switch to AMD a bit sooner than I was thinking. So, I made the lateral from a 12700K > 7900X and haven't looked back since. I did the basics for the AMD conversion for ~$800 and sold the Intel for ~$700. The spinners cost me ~$800 and the U ~$1000. So, with a little planning and foresight you can achieve better results than you might think.

The only real issue is that most consumer boards don't have the SFF connections on them natively and you'll need to think about adapters. I tried a couple of PCB mount on the slot options with the Micron drives that failed and switched to an M2 / Oculink setup and benched the disk at full speed 6.5GB/s. I dropped the whole raid issue in the process as well figuring the drive will be just fine for years to come and I really don't need the redundancy anyway. I have dual M2 drives in the server anyway now to rsync between them and my laptop has dual M2's as well. The only storage thought I've had recently is maybe bump the secondary laptop drive from 1 > 4TB just to store a bit more locally but, still not really needed.

 

[joerambo]

I tried a couple of PCB mount on the slot options with the Micron drives that failed and switched to an M2 / Oculink setup and benched the disk at full speed 6.5GB/s.

Any specifics what was wrong with slot mount PCBs? I think M.2 has an advantage of full speed PCIE 4 or 5 depending on CPU, but if PCIE slot is connected directly to CPU ( as Threadrippers and workstation Xeons are ) they should be full speed?

Or is something inherently wrong about PCB mounts in slots?

 

[Tech Junky]

PCB

I think in my case it was more of a combination of issues. They worked until the drives lost their partition table. Ultimately though I think there was a gremlin in the Intel board that caused some issues. The micron drives run hot as well. Someone else I was talking with for a couple of them as well and ended up making a direct fan mount to keep them cool I his systems but they were still running a bit hot even with direct cooling. Putting them on a cable though allows for varied placement away from the heat sources. Since they're light on terms of weight it's easy to put them just about anywhere.

 

[DaveLTX]

I think in my case it was more of a combination of issues. They worked until the drives lost their partition table. Ultimately though I think there was a gremlin in the Intel board that caused some issues. The micron drives run hot as well. Someone else I was talking with for a couple of them as well and ended up making a direct fan mount to keep them cool I his systems but they were still running a bit hot even with direct cooling. Putting them on a cable though allows for varied placement away from the heat sources. Since they're light on terms of weight it's easy to put them just about anywhere.

Typical of U.2 drives. My PM1733 ran hot as hell at idle until I put it closer to the front fan
They expect some actual airflow from being in front of the system after all

 

[Tech Junky]

closer to the front fan

I had several fans on the prior build but never really looked at the temps because the micron drives didn't last long enough. The Kioxia though is typically sitting around 40 though. The AMD build only has dual 180s though compared to the 6*120s in the Intel setup. First drive failed within hours and the second one died within a week. Well, they didn't die but couldn't store data as the partition table disappeared and couldn't be applied again no matter the approach. I could still see them on the lspci output and grab smart data from them but couldn't do much more.

 

[DaveLTX]

I had several fans on the prior build but never really looked at the temps because the micron drives didn't last long enough. The Kioxia though is typically sitting around 40 though. The AMD build only has dual 180s though compared to the 6*120s in the Intel setup. First drive failed within hours and the second one died within a week. Well, they didn't die but couldn't store data as the partition table disappeared and couldn't be applied again no matter the approach. I could still see them on the lspci output and grab smart data from them but couldn't do much more.

Even if I put them on a 3.5" caddy that should get airflow, i had to take it out and literally rest it on the front fan as I was seeing 73c idle temps... not great when the SSD is already 80% heatsink!

 

[Tech Junky]

73c idle temps

Which is why I mentioned with the Micron drives the person I was talking to mounted them with fans attached to them because they were running so hot.

sudo smartctl -a /dev/nvme0n1
smartctl 7.3 2022-02-28 r5338 [x86_64-linux-6.8.0-060800rc1-generic] (local build)
Copyright (C) 2002-22, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Number:                       KIOXIA KCD8XRUG15T3
Serial Number:                     
Firmware Version:                   0103
PCI Vendor/Subsystem ID:            0x1e0f
IEEE OUI Identifier:                0x8ce38e
Total NVM Capacity:                 15,360,950,534,144 [15.3 TB]
Unallocated NVM Capacity:           0
Controller ID:                      1
NVMe Version:                       1.4
Number of Namespaces:               64
Namespace 1 Size/Capacity:          15,360,950,534,144 [15.3 TB]
Namespace 1 Utilization:            2,104,690,343,936 [2.10 TB]
Namespace 1 Formatted LBA Size:     512
Namespace 1 IEEE EUI-64:            8ce38e e30048e8c7
Local Time is:                      Sun Feb  4 07:49:58 2024 CST
Firmware Updates (0x16):            3 Slots, no Reset required
Optional Admin Commands (0x025f):   Security Format Frmw_DL NS_Mngmt Self_Test MI_Snd/Rec Get_LBA_Sts
Optional NVM Commands (0x00ff):     Comp Wr_Unc DS_Mngmt Wr_Zero Sav/Sel_Feat Resv Timestmp Verify
Log Page Attributes (0x1e):         Cmd_Eff_Lg Ext_Get_Lg Telmtry_Lg Pers_Ev_Lg
Maximum Data Transfer Size:         1024 Pages
Warning  Comp. Temp. Threshold:     77 Celsius
Critical Comp. Temp. Threshold:     85 Celsius
Namespace 1 Features (0x14):        Dea/Unw_Error NP_Fields

Supported Power States
St Op     Max   Active     Idle   RL RT WL WT  Ent_Lat  Ex_Lat
 0 +    25.00W   25.00W       -    0  0  0  0   500000  500000
 1 +    12.00W   12.00W       -    1  1  1  1   500000  500000

Supported LBA Sizes (NSID 0x1)
Id Fmt  Data  Metadt  Rel_Perf
 0 +     512       0         0
 1 -     512       8         0
 2 -    4096       0         0
 3 -    4096       8         0
 4 -    4096      64         0

=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

SMART/Health Information (NVMe Log 0x02)
Critical Warning:                   0x00
Temperature:                        44 Celsius
Available Spare:                    100%
Available Spare Threshold:          7%
Percentage Used:                    0%
Data Units Read:                    27,427,683 [14.0 TB]
Data Units Written:                 18,900,158 [9.67 TB]
Host Read Commands:                 171,717,678
Host Write Commands:                351,491,023
Controller Busy Time:               358
Power Cycles:                       167
Power On Hours:                     3,299
Unsafe Shutdowns:                   88
Media and Data Integrity Errors:    0
Error Information Log Entries:      0
Warning  Comp. Temperature Time:    0
Critical Comp. Temperature Time:    0

Error Information (NVMe Log 0x01, 16 of 256 entries)
No Errors Logged

My takeaway is not all U drives are made the same. I switched brands after the 2 failures because I didn't want to deal with it a third time. When these things aren't "cheap" waiting on returns / credits is time consuming. I also figured maybe there was a reason they were selling considerably cheaper than other brands. I also considered maybe it was the PCB mounts which was why I switched to an M2+cable instead. Each drive though had a different PCB card it was attached to which made me think maybe the 1st card/drive had an issue when the second one lasted a bit longer. Then digging in a bit more on the second failure I saw some hints that maybe it was the Intel system and saw some things with the Kioxia that pushed me to rebuild sooner than planned. The Intel setup was indicating some issues and required running disk checks manually to clear them which pointed towards some IO issue with that particular MOBO. I further confirmed it when the AMD build was together by putting in my internal OTA TV card and it working like it did prior to the upgrade to the 12700K/MOBO a couple of years prior which resulted in loss of channels being tunable. Originally I thought maybe it was something with the move from 8700K > 12700K causing the OTA card to act funny but, now it seems more like an issue with the MOBO I was using even though other PCIE devices worked fine for the duration.

There's just sometimes some weird stuff that shows up but isn't a clear indicator of an issue being present overall when doing system rebuilds. Even this AMD build had some teething issues to deal with but, that seems to be mostly UEFI update related. AM5 being a new platform took AMD/OEM a bit of time to iron out AGESA/UEFI to be easier to deal with. The MOBO I picked up on Amazon for $160 now can't be found for under $200-250. Since they released a working / stable update the returns flow has dropped and prices have gone up. There were some quirks with the UEFI not doing certain things on the original releases which I suspect are why there were so many returns across the board for different vendors. Nothing critical but still annoying to deal with depending on how you use the system.

 

[DaveLTX]

Which is why I mentioned with the Micron drives the person I was talking to mounted them with fans attached to them because they were running so hot.

sudo smartctl -a /dev/nvme0n1
smartctl 7.3 2022-02-28 r5338 [x86_64-linux-6.8.0-060800rc1-generic] (local build)
Copyright (C) 2002-22, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Number:                       KIOXIA KCD8XRUG15T3
Serial Number:                   
Firmware Version:                   0103
PCI Vendor/Subsystem ID:            0x1e0f
IEEE OUI Identifier:                0x8ce38e
Total NVM Capacity:                 15,360,950,534,144 [15.3 TB]
Unallocated NVM Capacity:           0
Controller ID:                      1
NVMe Version:                       1.4
Number of Namespaces:               64
Namespace 1 Size/Capacity:          15,360,950,534,144 [15.3 TB]
Namespace 1 Utilization:            2,104,690,343,936 [2.10 TB]
Namespace 1 Formatted LBA Size:     512
Namespace 1 IEEE EUI-64:            8ce38e e30048e8c7
Local Time is:                      Sun Feb  4 07:49:58 2024 CST
Firmware Updates (0x16):            3 Slots, no Reset required
Optional Admin Commands (0x025f):   Security Format Frmw_DL NS_Mngmt Self_Test MI_Snd/Rec Get_LBA_Sts
Optional NVM Commands (0x00ff):     Comp Wr_Unc DS_Mngmt Wr_Zero Sav/Sel_Feat Resv Timestmp Verify
Log Page Attributes (0x1e):         Cmd_Eff_Lg Ext_Get_Lg Telmtry_Lg Pers_Ev_Lg
Maximum Data Transfer Size:         1024 Pages
Warning  Comp. Temp. Threshold:     77 Celsius
Critical Comp. Temp. Threshold:     85 Celsius
Namespace 1 Features (0x14):        Dea/Unw_Error NP_Fields

Supported Power States
St Op     Max   Active     Idle   RL RT WL WT  Ent_Lat  Ex_Lat
0 +    25.00W   25.00W       -    0  0  0  0   500000  500000
1 +    12.00W   12.00W       -    1  1  1  1   500000  500000

Supported LBA Sizes (NSID 0x1)
Id Fmt  Data  Metadt  Rel_Perf
0 +     512       0         0
1 -     512       8         0
2 -    4096       0         0
3 -    4096       8         0
4 -    4096      64         0

=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

SMART/Health Information (NVMe Log 0x02)
Critical Warning:                   0x00
Temperature:                        44 Celsius
Available Spare:                    100%
Available Spare Threshold:          7%
Percentage Used:                    0%
Data Units Read:                    27,427,683 [14.0 TB]
Data Units Written:                 18,900,158 [9.67 TB]
Host Read Commands:                 171,717,678
Host Write Commands:                351,491,023
Controller Busy Time:               358
Power Cycles:                       167
Power On Hours:                     3,299
Unsafe Shutdowns:                   88
Media and Data Integrity Errors:    0
Error Information Log Entries:      0
Warning  Comp. Temperature Time:    0
Critical Comp. Temperature Time:    0

Error Information (NVMe Log 0x01, 16 of 256 entries)
No Errors Logged

My takeaway is not all U drives are made the same. I switched brands after the 2 failures because I didn't want to deal with it a third time. When these things aren't "cheap" waiting on returns / credits is time consuming. I also figured maybe there was a reason they were selling considerably cheaper than other brands. I also considered maybe it was the PCB mounts which was why I switched to an M2+cable instead. Each drive though had a different PCB card it was attached to which made me think maybe the 1st card/drive had an issue when the second one lasted a bit longer. Then digging in a bit more on the second failure I saw some hints that maybe it was the Intel system and saw some things with the Kioxia that pushed me to rebuild sooner than planned. The Intel setup was indicating some issues and required running disk checks manually to clear them which pointed towards some IO issue with that particular MOBO. I further confirmed it when the AMD build was together by putting in my internal OTA TV card and it working like it did prior to the upgrade to the 12700K/MOBO a couple of years prior which resulted in loss of channels being tunable. Originally I thought maybe it was something with the move from 8700K > 12700K causing the OTA card to act funny but, now it seems more like an issue with the MOBO I was using even though other PCIE devices worked fine for the duration.

There's just sometimes some weird stuff that shows up but isn't a clear indicator of an issue being present overall when doing system rebuilds. Even this AMD build had some teething issues to deal with but, that seems to be mostly UEFI update related. AM5 being a new platform took AMD/OEM a bit of time to iron out AGESA/UEFI to be easier to deal with. The MOBO I picked up on Amazon for $160 now can't be found for under $200-250. Since they released a working / stable update the returns flow has dropped and prices have gone up. There were some quirks with the UEFI not doing certain things on the original releases which I suspect are why there were so many returns across the board for different vendors. Nothing critical but still annoying to deal with depending on how you use the system.

Yeah, the 600 series chipsets on Intel's side. My experience with them is that they are at best, buggy.
I had a string of PCs I built for my customers have issues because... the bios they came with (This was post non K release and post B6xx!) would cause the CPUs to return faulty data and wasn't fixed until 2 months later.
Prime95 would almost immediately crash. Wasn't safe at all. Also probably why I had to reinstall windows many times!
If the microcode itself was buggy I certainly can't imagine the PCIe signalling integrity

 

[Tech Junky]

600 series chipsets

I can't say one way or another as everything except that one card seemed to run 100%. I ran it 24/7 w/ Raid10 and dual NVME OS drives and not a peep from anything else until I started messing with going all flash for storage. That's when things started to unveil themselves that there might be an actual issue. I didn't bother trying the OTA card in the first iteration which was a Z790 / 4*M2 before finding the U capacity/price being a better option in terms of simplicity vs dealing with M2 pricing and lack of capacity.

I had already been planning the AMD switch though for awhile this just pushed me to do it. I got sick of dealing with Intel's planned obsolescence of every couple of years. There really isn't much innovation beyond 12th at this point and ARL is the next big step but, with AMD AM5 when they shift things it's an easy drop in for a few years. Though I'm disappointed with the chipset uplink being x4 due to limiting the idea of packing it with more stuff. Intel at least put x8 bandwidth to the DMI allowing for less bottlenecks.

Info on the X880/E right now doesn't help me decide whether it will be something to consider bothering with or not. If there was something compelling like upping the bandwidth or including USB4 @ Gen 4 speeds with ASM controllers instead of Gen 3 Intel TB.

Then there's the talk about Qualcomm getting into the game potentially which could be interesting.