2025 suggestions for PCIe SSD storage

[oh2ftu]

Hi there,
I'm setting up a DL360 gen9 as shared storage with Truenas.
As speeds with SATA-SSD's are lower than expected we are looking into using PCIe AIC-SSD's.
We need 2TB net storage, and would like it to be mirrored (as usual).
The config would be something in the direction of;
- 2x 2TB PCIe SSD (which, is the question) in mirror
- 2x3.64TB SATA-SSD in mirror for replication and/or failover in case one PCIe ssd fails
- 2x cheap SSD for boot

What is the SSD to look for? Intel P3600, P3700? I would need one full and one half height for this.
Samsung PM1725? there are SOO many options.
One option would be SAS-SSD's as well, but these could be lacking in performance as well unless configured in RAIDz2 or similar.

The Proliant has a 20c E5-2640v4 and 256GB of ram - and a 10G nic.
Power usage does matter to some extent.

What would your recommendation be for reliability, cost effectiveness?
EU sellers preferred

 

[name stolen]

For PCIE, I would up the Intel generation you're looking at. P3700s should still last almost forever, but the performance is a bit yesteryear. P4610 is usually pretty affordable if you look around a bit and watch a few auctions, and starts at 1.6TB and goes up (I believe including the 2TB step). Even the 3.2TBs can be found at a decent value, which could give you some built-in overprovisioning, unnecessary as it may be. I would definitely take some lesser-used Kioxias here too; I just don't know their lineup as well. Samsung 1725 also seems a little lackluster on performance, but if it works for you, and they are new enough, sure.

For SATA, maybe Samsung 860 PRO, but more likely Intel S3710, even S3610 for (I believe) a little less write endurance (or if you can find them cheaply, which I cannot, S45x0/S46x0). S3710s should basically last forever. (ETA: just realized S3710 tops out at 1.2TB - will look further)

Cheap boot SSD - cheap is ok, but don't skimp hard on endurance here. Small SATA drives are cheap, so I stick with S3610, S3710, Toshiba Hawk HK4R (considered read intensive but rated for 1 DWPD plus PLP ECC etc) - if you have the spare SATA ports and bays.

2x PCIE, 2x SATA, and 2x boot will play quite nice electrically. I'd guess that adds <=10W most of the time, ~15W if its getting hammered.

I'm not sure if you're solely looking at add-in-card PCIE drives or if you're considering U.2 drives as well. Good values there, due to sheer volume, at the price of an adapter and/or cable(s).

 

[T_Minus]

Since it's an old build I tend to agree with @name stolen, lots of older cheap performant drives that will last nearly forever!

- P3700
- P4610 maybe even P4510
- Variety of Samsung models

For SATA I think you can find newer drives than the s3600\s3700 gen for cheap now too, I'd prefer those over any Samsung consumer drive for server use.


You could always grab those $300 1.5TB Optanes on ebay too if you want something snappy, but not as high of sequential.

 

[oh2ftu]

Thanks for the input. P4610 is U.2 right?
I'm a bit at loss would u.2 or pcie AIC be better?
The kit for DL360 gen9 is not cheap by any means, but the availability would be a bit better for drives. Also hotswap.
Don't know if the other servers that are possibly available (Dell R630 or R730) are easily retrofitted

 

[amalurk]

Two of these? Micron 9300 PRO 3.84TB SSD 2.5" U.2 PCIe 3.0x4 NVMe MTFDHAL3T8TDP-1AT1ZABYY | eBay

 

[name stolen]

I turned my U.2 Optane 905P into an AIC with this adapter, purchased July 2023 from Amazon, and still fine to this day after ~100TB of drive activity. It's in my main home Proxmox box.

 

[ca3y6]

Thanks for the input. P4610 is U.2 right?
I'm a bit at loss would u.2 or pcie AIC be better?

U.2 is 4 PCIe lanes, AIC usually 8. Then it is down to the specs of the card, but AIC in theory can get you potentially higher bandwidth. On the other hand you occupy a whole, precious, PCIe slot (a x8 PCIe slot can accomodate two U.2 drives with bifurcation, or even 4 drives with a PCIe switch). Used AIC tend to be a bit cheaper on ebay than U.2.

 

[name stolen]

If the AIC is x8, isn't it likely two drives on one card, like an Intel P3608? AICs I'm familiar with are really just U.2 drives on a card (or in the prosumer realm, m.2 on a card with a heatsink.)

 

[ca3y6]

P3608 is a bit of an exception, most AIC cards are a single drive. In the case of P3608, I am not sure of the internals, but I suspect it contains a PCIe switch, and that the two drives are connected either in x4 or x8 to that switch. Either way the card is connected with x8 to the motherboard, so you get full bandwidth, whether the bandwidth is half for each drive or full for each drive but shared.

A normal AIC card will be x8 lanes to the SSD directly.

 

[acquacow]

I turned my U.2 Optane 905P into an AIC with this adapter, purchased July 2023 from Amazon, and still fine to this day after ~100TB of drive activity. It's in my main home Proxmox box.

100TB is absolutely nothing for Optane =) I have 20-year-old Intel S3500 1.6TB drives with petabytes of writes on them.

Optane is vastly superior to that data-retention/cycle-wise, you don't need to fear at all =)

 

[MCmwi1sCEA]

I have 20-year-old Intel S3500 1.6TB

1.6TB was 11 years ago.

 

[nabsltd]

A normal AIC card will be x8 lanes to the SSD directly.

All the "normal" Intel AIC NVMe I have docs for are x4 cards. They also made some x4x4 and x2x2 AICs that require bifurcation. I haven't seen any that are true x8 cards.

 

[ca3y6]

All the "normal" Intel AIC NVMe I have docs for are x4 cards. They also made some x4x4 and x2x2 AICs that require bifurcation. I haven't seen any that are true x8 cards.

Maybe Intel, but I think all the Samsung and HGST AIC cards I have seen are x8 and their performance clearly exceed the bandwidth of x4.

For instance Samsung PM1725b, which is quite popular on ebay:

https://image-us.samsung.com/SamsungUS/PIM/Samsung_1725b_Product.pdf

U.2 read/write = 3.5GB/s / 3.1GB/s
AIC read/wite = 6.3GB/s / 3.3GB/s.

Clearly the AIC reads are in excess of a PCIe 3 x4 brandwidth which is 4GB/s (and I measure them at around 5.9GB/s in crystaldiskmark)

 

[nabsltd]

U.2 read/write = 3.5GB/s / 3.1GB/s
AIC read/wite = 6.3GB/s / 3.3GB/s

Interesting.

The AIC definitely isn't anything special (like a pair of drives in RAID0) for writes or for 4K performance. I can't imagine that it's a single drive with an x8 interface, but it doesn't seem to be a pair with a PCIe switch, either.

It also seems to use the extra area as SLC for ingest, as the notes say that if you remove the extra 20% overprovisioning, the random 4K write is cut in half.

 

[ca3y6]

AIC just means it's a card form factor, but it's otherwise the same controller and flash as the U.2 version. But it does have more PCIe lanes available. Doesn't mean that the flash and controller are capable of using them. In the case of PM1725b, it can for reads but writes are still slower than a x4 interface. Newer cards are faster.

 

[acquacow]

1.6TB was 11 years ago.

Aah, 2018 was the release date of the 1.6TB S3500s... the S3500 line started much earlier, neat. Figured all the S3500 lineup would have come at once.

 

[oh2ftu]

I opted for two 2TB Intel P4600 cards. We'll see how that turns out