How to buy used U.2 drives (and recommendations for specific models)?

[zachj]

I'm not necessarily looking at all NVMe, but I am looking at all SSD. No way I am running 5 to 8 drives on the desk I work at.

I wouldn’t want to sit next to a spinner either, and have not for many years. But @nabsltd is right that almost any crappy ssd will work from a write endurance perspective. While QLC would ordinarily be recommended against, for your use case it would be fine (especially if you put zfs zil on a different disk).

D5-P4326 is right around the same price as consumer m.2 QLC drives on a $/tb basis. While you can certainly find cheaper drives in smaller capacities I’d bet you’ll spend 100% of the savings on cables.

 

[zachj]

I imagine that's for disk encryption. What does that bring rather than the usual setup with non self-encrypting disks?

ZFS supports software encryption. Some hypervisors (ex vsphere) do too. All client operating systems (windows/linux/macos) do too.

So TCG Opal is redundant in that context.

But for me it’s so easy to implement Opal encryption on the drive itself and then I don’t have to worry about whether I encrypt the files that collect there over many years. It’s easy to forget to enable encryption when you create a new pool or forget to enable bitlocker when you build a new windows vm or forget that the zfs pool is unencrypted when you’re trying to backup your laptop before reimaging it. Setting up encryption on the NAS means that doesn’t matter.

I personally enable both software encryption in truenas as well as physical encryption via TCG Opal. There’s no performance penalty compared to standalone software encryption, so why the heck not?

 

[nabsltd]

I'm not necessarily looking at all NVMe, but I am looking at all SSD. No way I am running 5 to 8 drives on the desk I work at.

I don't really notice my 6 drive array compared to the other sounds of the machine (GPU fans, etc.).

That said, going with SATA or SAS SSD is much easier than all NVMe. You can get 3.84TB, 1DWPD enterprise SSDs for $225 used. These will have 95% or more remaining life. Slap them into whatever you want, like this Icy Dock 6-bay.

D5-P4326 is right around the same price as consumer m.2 QLC drives on a $/tb basis.

These really shine if you want to get larger NVMe drives. You can easily get the 15.4TB for $900 used.

If I could get a 12-bay NVMe backplane for a Supermicro 2U chassis, I'd probably build an all-NVMe storage box. As long as I don't need to use any kind of HBA, but instead can connect the drives directly to PCIe ports (Oculink, etc.), it would be a nice setup.

 

[nexox]

I don't really notice my 6 drive array compared to the other sounds of the machine (GPU fans, etc.).

Indeed, I thought I got my workstation pretty quiet, but I can still barely hear the seven 2x14 drives I'm performance testing, aside from the initial spin up at boot when they whack all 14 heads back and forth at once. Two 24TB disks can't be all that loud, and in a mirror would give the same raw storage as 8x4TB SSDs in RaidZ2.

If I could get a 12-bay NVMe backplane for a Supermicro 2U chassis, I'd probably build an all-NVMe storage box.

Not the cheapest ever, but really not bad, considering what you get and how much similar products cost: Supermicro BPN-SAS3-LA26A-N12 12-Slot LFF Backplane Supports SAS3/SATA3/NVMe4 Storage Devices

 

[guiniol]

ZFS supports software encryption. Some hypervisors (ex vsphere) do too. All client operating systems (windows/linux/macos) do too.

So TCG Opal is redundant in that context.

But for me it’s so easy to implement Opal encryption on the drive itself and then I don’t have to worry about whether I encrypt the files that collect there over many years. It’s easy to forget to enable encryption when you create a new pool or forget to enable bitlocker when you build a new windows vm or forget that the zfs pool is unencrypted when you’re trying to backup your laptop before reimaging it. Setting up encryption on the NAS means that doesn’t matter.

I personally enable both software encryption in truenas as well as physical encryption via TCG Opal. There’s no performance penalty compared to standalone software encryption, so why the heck not?

Ack. I just use software encryption, but good to know.

I don't really notice my 6 drive array compared to the other sounds of the machine (GPU fans, etc.).

That said, going with SATA or SAS SSD is much easier than all NVMe. You can get 3.84TB, 1DWPD enterprise SSDs for $225 used. These will have 95% or more remaining life. Slap them into whatever you want, like this Icy Dock 6-bay.

I'm hoping to keep temps low and run fans as slow as possible. Background noise makes it difficult to concentrate for me. I gather that U.2 drives are going to generate more heat, so I'm back to SATA SSDs. I did find a MB with 8 SATA ports and may just get that one.

Indeed, I thought I got my workstation pretty quiet, but I can still barely hear the seven 2x14 drives I'm performance testing, aside from the initial spin up at boot when they whack all 14 heads back and forth at once. Two 24TB disks can't be all that loud, and in a mirror would give the same raw storage as 8x4TB SSDs in RaidZ2.

It's not just about raw storage but how many drives can fail until I lose my data. Raidz2 gives me up to 2 failures. And I plan on starting with 5 drives and expand as needed later.

 

[nabsltd]

Not the cheapest ever, but really not bad, considering what you get and how much similar products cost: Supermicro BPN-SAS3-LA26A-N12 12-Slot LFF Backplane Supports SAS3/SATA3/NVMe4 Storage Devices

That's actually fine...$23 per slot isn't much if you are spending $300 for a U.2 drive. And, it appears to be compatible with my 826 chassis.

So, I'll blame you now that I have to spend money on more U.2 drives.

 

[mattventura]

The gen5 version of that backplane (BPN-NVME5-LA26A-S12) can sometimes be found for a bit less (220-230) but is often out of stock. Only difference AFAICT is that it uses MCIO connectors for the NVMe ports instead of SlimSAS for everything.

 

[nabsltd]

Only difference AFAICT is that it uses MCIO connectors for the NVMe ports instead of SlimSAS for everything.

The cabling for a DIY NVMe server seems to be the biggest pain. Some motherboards have MCIO, some have SlimSAS, and I would likely add a PCIe card for more connections, so I'd have to make sure everything matched, or find the right adapter cables.

 

[guiniol]

One last thing before I let you all go on about U.2 and I go buy some SATA drives

If I get the Asrock B550 Taichi, and it has:

• 4 x SATA3 6.0 Gb/s Connectors, support RAID (RAID 0, RAID 1 and RAID 10), NCQ, AHCI and Hot Plug
• 4 x SATA3 6.0 Gb/s Connectors by ASMedia ASM1061, support NCQ, AHCI and Hot Plug

Is there anything wrong with the fact that the last 4 ports are on a separate controller? (other than maybe messing up the C-states?)
I will still be limited by the total bandwidth between the B550 chipset and the CPU, but we've established that shouldn't be an issue for my use case. Anything else I should be wary about?

 

[Marsh]

4 x SATA3 6.0 Gb/s Connectors by ASMedia ASM1061, support NCQ, AHCI and Hot Plug

Con:
ASMEDIA ASM1061 : Just a bit slower . will need actual performance testing to determine how much slower.
SATA SSD per TB is price same as m2 or u2

Add:
I have a SATA SSD NAS now, not looking for performance. goal is storage (8 x 1.92TB SATA SSD ) ,
brought 1.92TB at $45 each last year.

 

[guiniol]

Con:
ASMEDIA ASM1061 : Just a bit slower . will need actual performance testing to determine how much slower.
SATA SSD per TB is price same as m2 or u2

Already tried that. Getting enough lanes to connect 8 M.2 is tricky. And I don't think I can have a quiet enough setup with U.2 since it requires enough cooling.

Add:
I have a SATA SSD NAS now, not looking for performance. goal is storage (8 x 1.92TB SATA SSD ) ,
brought 1.92TB at $45 each last year.

That's insanely cheap. What drives did you get and where?

 

[nexox]

That's insanely cheap. What drives did you get and where?

That would be the used SanDisk Cloudspeed drives from last summer (by which I mean 2023,) unfortunately prices went back up, I should have grabbed more than the 5 or so I ended up with.

 

[guiniol]

Alright, because I am looking at 3.84Tb drives and they are significantly more than 2x that, so I just wanted to make sure I wasn't missing anything

 

[guiniol]

One last question, though maybe everybody's left already.
My plan is to start with 5 drives (in RaidZ2) and expand up to 8 drives when I need more capacity.
If I expect the 5 drives to last 5 years (random number, trying to figure out the horizon on that), are the prices of drives going down enough that I should be thinking of just buying 5 bigger drives and expand that way rather than going to 8?

 

[name stolen]

You might consider just replacing a smaller number of drives sooner as the easiest, most trouble-free strategy. In 3-5 years, or 7-10 or whatever, if endurance or space are running low, just replace each drive one by one with something bigger. For my home NAS needs, twice as big has worked for a long time for spinners - 4x2T, 4x4T, 4x8T etc.

 

[guiniol]

You might consider just replacing a smaller number of drives sooner as the easiest, most trouble-free strategy. In 3-5 years, or 7-10 or whatever, if endurance or space are running low, just replace each drive one by one with something bigger. For my home NAS needs, twice as big has worked for a long time for spinners - 4x2T, 4x4T, 4x8T etc.

Ack. Initially, I thought drives number 6 to 8 could already be larger, and then I would only have the 5 original ones to buy again once I get to that limit. But the time after that would still be 8 drives at once which is a good chunk of money. Also, going with a lower number of drives makes it easier to find a motherboard.

 

[adman_c]

Ack. Initially, I thought drives number 6 to 8 could already be larger, and then I would only have the 5 original ones to buy again once I get to that limit. But the time after that would still be 8 drives at once which is a good chunk of money. Also, going with a lower number of drives makes it easier to find a motherboard.

Assuming that raidz expansion works properly, and also assuming that you set the zpool option autoexpand=on, your thinking *should* be correct. I have never tested autoexpand for a raidz pool though--only with mirrors (and it worked as expected).

 

[guiniol]

Assuming that raidz expansion works properly, and also assuming that you set the zpool option autoexpand=on, your thinking *should* be correct. I have never tested autoexpand for a raidz pool though--only with mirrors (and it worked as expected).

I would hope it does as it's now in the beta version of TrueNAS IIRC.

 

[adman_c]

I will also reiterate the excellent advice from earlier in the thread that enterprise NVME will need active cooling or they will overheat/throttle under relatively light load.

 

[guiniol]

I will also reiterate the excellent advice from earlier in the thread that enterprise NVME will need active cooling or they will overheat/throttle under relatively light load.

Yes, I got that part
I'm thinking Node 804 now and have been trying to figure out if the airflow would be enough. I'm also thinking I would use a max of 6 drives (which would reduce heat generation, but the main reason is for easier upgrade when I want to switch to larger drives).
Alternatively, I am trying to convince myself that buying more expensive (and "worse") SATA SSDs would be a better deal.