storage SSD faster than cache SSD make sense? (vSAN)

[koifish59]

I'm setting up a cluster of 2 ESXi hosts, all flash vSAN. Would it make any sense if my datastore SSDs have higher IOPS and throughput than the cache SSD? These are what I plan on getting for each host:

• cache SSD: Intel 375gb P4800X

Spoiler: specs

seq read: 2400 MB/s
seq write: 2000 MB/s
random read: 550k IOPS
random write: 500k IOPS
latency read/write: 10 µs
edurance: 20.5 PBW

• storage SSD: 3x 2TB Samsung 980 PRO (not launched yet)

Spoiler: specs

seq read: 7000 MB/s
seq write: 5000 MB/s
random read: 1000k IOPS
random write: 1000k IOPS
edurance: 2.4 PBW

IOPS is what I'm interested in, and the new consumer Samsung 980 pro is twice as fast as the enterprise P4800X. So would this mean the cache SSD will be the bottleneck and the high speeds of the 980 pro will be a waste? Is this a wasteful setup for vSAN?

 

[i386]

The numbers for the P4800X are for steady state*, the numbers for 980 PRO are "up to" numbers**.
With previous models samsung ssds started to overheat and throttle.

A <5 minute search & reading on google lead me to guru3d where a 980 pro hit only high numbers in sysnthetic benchmarks and average numbers in normal use cases.

If I remeber it correctly vsans implementation first writes to the cache tier and then to the storage tier.
I think the optane will become the bottleneck: with the p4800x you will have max 2.5GByte/s and distribute this on the the 3 980 pros (~833 MBytes/s per device)

* Enterprise ssds are usually benchmarked with industry standards like spec (SPEC Benchmarks) and documented in datasheets/product documents
** Consumer ssds are often benchmarked with simple benchmark tools like crystaldiskmark where the tools use unrealistic settings to get the max/peak possible numbers in throughput/iops

 

[gb00s]

Other points are:

1. Samsung 980 Pro is Gen4 with NVMe1.3b or c (not sure) and only reaches the provided spec with Gen4 slots
2. Samsung guarantees just half of the endurance compared to 970 Pro ... TBW 150TB only
3. The speed of the 980 Pro is provided only if the TLC cells are available and used in the SLC modus > The older the drive the less TLC cells available ... means it will become slower and slower the more data are written to it. I think I saw a test somewhere where the speed came down to a 'slow' 2000MB/s from 5000MB/s at the beginning ...

If it has to be Gen4, maybe check a Samsung PM1733 which can hold it's speed specs ... I know ... price price

 

[amalurk]

Latency on Optane is lower and it is quite faster at lower queue depth than flash which is where you are probably going to be most of the time.

Look at Optane 905P here: The Samsung 980 PRO PCIe 4.0 SSD Review: A Spirit of Hope

For random reads It blows everything else out of the water by 4x at QD1 (first graph) and in the average of QD1, QD2 and QD4 (second graph) it is still 3x or better than everything else including 980.

Writing is a different story, they are very similar with the 980 pulling ahead in the mixed queue depth benchmark.

Obviously the Samsung sequential is faster on Gen4.

So it depends on your workload but Optane is going to be faster for many.

 

[Patrick]

I believe @WillTaillac already has his 980 Pro review done and in the queue for after the P5 today. As others have mentioned, the Optane drive is much faster.

Consumer drives tend to either have sequential or short burst workloads these days. Even OS loading, which was a big deal when SSDs first came out, is now a 5-10s affair. The Optane is designed to be hammered 24x7. Also, Optane tends to perform much better at lower queue depths.

 

[koifish59]

Thank you all! Based on THIS thread, I've decided to stick with optane for cache drive, but go with intel 905p instead of P4800X. Slightly higher sequential speeds and IOPS at the trade off for endurance. I've also read that the 905p does have PLP but intel does not state this in documentation because it's marketed as a consumer drive.