How to buy used U.2 drives (and recommendations for specific models)?

[guiniol]

So, I'm looking at buying used U.2 drives to fill a NAS (current build here). Trying to cram 8 disks on an AM4 platform, I found this card with the PLX8747 chip which would allow connecting 8 U.2 drives via a single x16 PCIE slot.
My plan was to start with 5 * 4TB drives in RaidZ2, and expand as needed up to 8 drives.
I have, however, never bought a used drive and I worried about getting a drive that's already on its last legs.

So, first question: What do you look for when buying a used drive?
I've been looking at eBay and AliExpress and I don't see any mention of how much use a used drive might have seen. Do sellers give a SMART diagnostic?

Second qestion: What brands/models are recommended? I see talk of Kioxia, Samung and Intel. If I buy the card above, the drives would be limited to PCIE 3.0, so I don't necessarily need something super new, but at the same time, if it uses less power/doesn't heat up as much, that's great too.

EDIT: alternatives on how to get 8 * M.2 NVMe drives in my build are also welcome

 

[TRACKER]

I would recommend you first to clarify how exactly are you going to cool these U.2 drives
cause they run hot.

 

[guiniol]

Good point. Still learning a lot about U.2 drives.
I am not 100% decided on the case but like the looks of the Jonsbo N5. This case provides airflow through the disk bays, but I couldn't find how much. From reading threads on this forum, the drives don't necessarily need a lot of airflow, just some. So I could hope it's enough?

I also found some PM983 drives for about 200 each (for the 3.84TB models), and those are supposed to run somewhat cooler.
Though, I don't know if they're any good. Still looking at models randomly atm.

EDIT: Just realised the backplane of the N5 is SATA, so that's not great . I mean, more airflow if I remove it I guess. But I'm open to suggestions on cases too

 

[TRACKER]

From my personal experience, intel drives run cooler than micron ones.
My p4610 1.6T run around 45-46°C idle and around 52-55° under heavy load.
Compared to my micron 9200 Pro 1.92TB which runs idle at 60°C with small fan attached to it.
Under heavy load it reaches around 70°C. Without small fan i even get SMART warnings about high temp.
So definitely would try to avoid micron models (at least 9200 series).
Don't know about samsung, never had U.2 samsung drive

 

[TRACKER]

Jonsbo N5? definitely NO GO you need some airflow, but not 'some' like Jonsbo N5...

 

[nexox]

I have had pretty good results with Intel drives (mostly p4800x) sitting in front of a low speed 140mm fan, any drive with openings in the end (like some Microns) will need air pushed/pulled through those openings (in addition to airflow over the case,) which means you need some serious static pressure.

Cabling 8 U.2 drives will be a nightmare, you really want a backplane for that many.

Some ebay sellers will list a health percentage or include screenshots of SMART data, if not you can contact the seller and ask, or go with a seller that accepts returns. Realistically most enterprise drives don't get used anywhere near their rating, and at 80-85% NAND life remaining they will last forever in home use.

If you're just planning to use these as bulk file storage then TLC or even QLC NAND is probably fine, it'll be cheaper and likely run cooler, perhaps the Intel P4501.

 

[guiniol]

From my personal experience, intel drives run cooler than micron ones.
My p4610 1.6T run around 45-46°C idle and around 52-55° under heavy load.
Compared to my micron 9200 Pro 1.92TB which runs idle at 60°C with small fan attached to it.
Under heavy load it reaches around 70°C. Without small fan i even get SMART warnings about high temp.
So definitely would try to avoid micron models (at least 9200 series).
Don't know about samsung, never had U.2 samsung drive

So, avoid Micron.
Someone mentioned in a different thread that the PM983 don't run as hot as other drives. Not sure what "other drives" were though.

Jonsbo N5? definitely NO GO you need some airflow, but not 'some' like Jonsbo N5...

I mean, with the backplane being SATA, I am looking for a different case now. Not a racked one though...

I have had pretty good results with Intel drives (mostly p4800x) sitting in front of a low speed 140mm fan, any drive with openings in the end (like some Microns) will need air pushed/pulled through those openings (in addition to airflow over the case,) which means you need some serious static pressure.

Cabling 8 U.2 drives will be a nightmare, you really want a backplane for that many.

Some ebay sellers will list a health percentage or include screenshots of SMART data, if not you can contact the seller and ask, or go with a seller that accepts returns. Realistically most enterprise drives don't get used anywhere near their rating, and at 80-85% NAND life remaining they will last forever in home use.

If you're just planning to use these as bulk file storage then TLC or even QLC NAND is probably fine, it'll be cheaper and likely run cooler, perhaps the Intel P4501.

The main use case is going to be photo backups + Immich. I'll take a look at the P4501.

I'll see if I can find cases with a U.2 backplane, but I expect those are going to be all enterprise racked ones and I don't have a rack.

 

[nexox]

There's nothing stopping you from setting a rack mount server on a shelf, or propping it up on its side like a tower chassis, the real issue is how loud they tend to be.

 

[pimposh]

Bit pricey but always you can go with IcyDock's

 

[SlowmoDK]

Or If your use case as stated is running mainly Immich, then just use SATA SSDs, this will simplify your AM4 build a lot
and u will never feel the added speed from U2/U3 drives, but u will feel the heat and added wattage

 

[ca3y6]

I not sure how much of the nvme performance will be left after raidz2. Also I am sure you are aware you can't add a disk to a raidz2 vdev to expand it. You have to destroy the pool and recreate it. For mass storage array I tend to stick to sata/sas.

 

[zachj]

For your use case definitely sata. They run much cooler and they don’t use much power—consider the maximum power draw of 8x enterprise drives and you will have heartburn about using splitters.

The main issue I have with used sata ssds is they’re typically not used by enterprises and so it’s a bit harder to trust what you’re getting unless the seller provides smart diagnostics. They are likely to have had even less use than an enterprise drive but their endurance is so much less than enterprise drives that less use doesn’t mean much.

 

[ca3y6]

There are lots of used enterprise sata drives on ebay, with similar endurance than u.2. In fact if you are building a NAS, you should really use enterprise drives, would be very nervous without power loss protection. A power loss could kill your whole array of SSDs, RAID wouldn’t help.

 

[guiniol]

Or If your use case as stated is running mainly Immich, then just use SATA SSDs, this will simplify your AM4 build a lot
and u will never feel the added speed from U2/U3 drives, but u will feel the heat and added wattage

That's definitely been on my mind. That being said, finding a MB with 8 SATA ports has been challenging...

I not sure how much of the nvme performance will be left after raidz2. Also I am sure you are aware you can't add a disk to a raidz2 vdev to expand it. You have to destroy the pool and recreate it. For mass storage array I tend to stick to sata/sas.

Actually, it seems it is now possible to extend Raidz one drive at a time (eg. see RaidZ Expansion on ElectricEel Nightlies), which I was very happy to learn about. Yes, there are caveats (mostly that files don't get resharded), but it should be good enough for me.

For your use case definitely sata. They run much cooler and they don’t use much power—consider the maximum power draw of 8x enterprise drives and you will have heartburn about using splitters.

The main issue I have with used sata ssds is they’re typically not used by enterprises and so it’s a bit harder to trust what you’re getting unless the seller provides smart diagnostics. They are likely to have had even less use than an enterprise drive but their endurance is so much less than enterprise drives that less use doesn’t mean much.

That was another consideration. It seems that enterprise drives are really in a different class, so that's why I am looking at them. Still very much in the exploratory phase and I have about 8 different builds at this point.

There are lots of used enterprise sata drives on ebay, with similar endurance than u.2. In fact if you are building a NAS, you should really use enterprise drives, would be very nervous without power loss protection. A power loss could kill your whole array of SSDs, RAID wouldn’t help.

Are we talking WD Red and stuff or something else?

 

[zachj]

There’s definitely enterprise sas ssds that are similar in endurance compared to u.2.

I haven’t looked at enterprise sata ssds but I’m guessing they’re going to typically be rebranded sata drives—if power loss protection is important to you then make sure you check the specific model supports it.

one thing I would personally have as a requirement is support for tcg opal self encryption.

Enterprise drives usually ship in both opal and non opal variants so make sure you confirm there’s a PSID written on the label.

 

[SlowmoDK]

That's definitely been on my mind. That being said, finding a MB with 8 SATA ports has been challenging...

Are we talking WD Red and stuff or something else?

Since you planned on that PLX splitter card for U2 anyway, going with sata ssds will free up that slot, for an cheaper LSI card with 8 or 16 ports

Personally i run Intel drives got a stack of s4610 cheap, but something like Samsung PM863/PM883 is also a very valid choice

Even if you end up with enterprise drives with only 50% writes left, you will never wear them out as a home labber, but I've lost several consumer drives in less than a year, especially if running ZFS

 

[nexox]

That's definitely been on my mind. That being said, finding a MB with 8 SATA ports has been challenging...

Just get a basic cheap SAS HBA (eg LSI 3008,) with cables it will come out far less expensive than the 8xNVMe adapter and the cables those require.

Are we talking WD Red and stuff or something else?

I don't see power loss protection listed as a feature on the first WD Red SSD page I found, they look better in some ways than many consumer SSDs, but not enterprise grade. You might want to look at the Micron 5100, Intel S4510, various Samsungs that I can never keep track of, and some others - plus you could look at SAS SSDs once you have the SAS HBA.

 

[ca3y6]

And since you seem to be fairly new to all that, keep in mind there are two kind of SAS HBA, the "IT mode" ones, which are meant to be used with software RAID (like ZFS) and which expose the disk to the OS, and the hardware RAID ones, which typically do not expose the actual disks, only a virtual RAID disk (there is a way to flash those into IT mode but that's not the easy route).

So in your case you want to go for some IT mode SAS controller, for instance: Pardon our interruption...

For this model, each of the 4 ports can connect 4 SATA drives (or SAS3, you can mix and match), so 16 devices all in. If that's not enough you can add to that a SAS expander, which is kind of like a network switch for SAS. For instance this one Pardon our interruption... which take two SAS cables to the HBA (out of the four of the HBA above), and allows to connect 28 drives (7 cables).

 

[nabsltd]

The main use case is going to be photo backups + Immich.

This is a classic WORM (write once, read mostly). You'll write a picture file, maybe edit it and save a copy, but no real load on the disks, unless you have 20 users doing this at the same time. So, you don't care about flash endurance...anything that can support even a few hundred GB per day would probably be insane overkill.

And, you probably don't need an all-NVMe storage system. Spinning disk with some sort of NVMe cache/tiering would be much cheaper, generate far less heat, and be much easier to find hardware (chassis, backplane, etc.) to support it.

 

[guiniol]

So, cheap LSI HBA, in IT mode, sounds like a better option, and I have a few SSD models to look for, thanks!

one thing I would personally have as a requirement is support for tcg opal self encryption.

Enterprise drives usually ship in both opal and non opal variants so make sure you confirm there’s a PSID written on the label.

I imagine that's for disk encryption. What does that bring rather than the usual setup with non self-encrypting disks?

This is a classic WORM (write once, read mostly). You'll write a picture file, maybe edit it and save a copy, but no real load on the disks, unless you have 20 users doing this at the same time. So, you don't care about flash endurance...anything that can support even a few hundred GB per day would probably be insane overkill.

And, you probably don't need an all-NVMe storage system. Spinning disk with some sort of NVMe cache/tiering would be much cheaper, generate far less heat, and be much easier to find hardware (chassis, backplane, etc.) to support it.

I'm not necessarily looking at all NVMe, but I am looking at all SSD. No way I am running 5 to 8 drives on the desk I work at.